From nobody Sat Aug 26 05:05:01 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RXlBC3FWLz4rHgx for ; Sat, 26 Aug 2023 05:05:07 +0000 (UTC) (envelope-from diizzy@FreeBSD.org) Received: from relay6-d.mail.gandi.net (relay6-d.mail.gandi.net [IPv6:2001:4b98:dc4:8::226]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RXlBB5SWQz3M8S; Sat, 26 Aug 2023 05:05:06 +0000 (UTC) (envelope-from diizzy@FreeBSD.org) Authentication-Results: mx1.freebsd.org; none Received: by mail.gandi.net (Postfix) with ESMTPA id 92C4FC0003; Sat, 26 Aug 2023 05:05:01 +0000 (UTC) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Date: Sat, 26 Aug 2023 07:05:01 +0200 From: Daniel Engberg To: Cy Schubert Cc: Craig Leres , freebsd-hackers@freebsd.org Subject: Re: How do I configure a port to get a specific sourceforge commit? In-Reply-To: <20230826031839.DD932387@slippy.cwsent.com> References: <56d5e798-9bb4-ca9a-5fbf-59a7b17a334b@freebsd.org> <20230826031839.DD932387@slippy.cwsent.com> Message-ID: <390991b39f9b7c8b568d1aa2de809bfa@FreeBSD.org> X-Sender: diizzy@FreeBSD.org Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-GND-Sasl: daniel.engberg@pyret.net X-Rspamd-Queue-Id: 4RXlBB5SWQz3M8S X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:29169, ipnet:2001:4b98::/32, country:FR] On 2023-08-26 05:18, Cy Schubert wrote: > In message <56d5e798-9bb4-ca9a-5fbf-59a7b17a334b@freebsd.org>, Craig > Leres > writ > es: >> I'm trying to update graphics/netpbm to an actively maintained repo >> (PR >> 262212). I'm told by some that this is the current best choice: >> >> https://sourceforge.net/projects/netpbm/ >> >> Unfortunately its idea of "release management" is less than ideal. >> >> Let's say I browse this commit log: >> >> https://sourceforge.net/p/netpbm/code/4616/log/?path=/advanced >> >> and decide that release 11.03.04/commit r4606 is interesting. I can >> download it if I click on "Tree" for that commit and then "Download >> Snapshot" it gives me this as the "direct link": >> >> >> https://sourceforge.net/code-snapshots/svn/n/ne/netpbm/code/netpbm-code-r4606 >> -advanced.zip >> >> This definitely works. But I don't find an existing port that uses >> sourceforge with a specific commit and no combination of >> "MASTERSITES=SF..." I've tried works. >> >> I have something that does work but it's a horrible hack, essentially: >> >> DISTVERSION= 11.03.04 >> MASTER_SITES= >> ${MASTER_SITE_SOURCEFORGE:S,/project/%SUBDIR%/,/code-snapshots/svn/n/ne/netpb >> m/code/,} >> DISTNAME= netpbm-code-r4606-advanced >> USES= zip >> >> Is there a less hacky way to do this? >> >> Craig >> > > I proxy my -devel ports (cde-devel and motif-devel) based on SF sources > through my GH account, and the wpa_supplicant-devel/hostapd-devel on a > git > repo at w1.fi using that method as well. This allows me to use GH > plumbing. > > I probably wouldn't suggest this for a non -devel port but if you're > ever > thinking of a netpbm-devel, this may be easier, allowing the use of the > GH > plumbing in ports. > > What I do is clone the remote repo and change the remote name from > origin > to upstream, setting the GH remote to origin and pushing to GH. To > update I > git fetch upstream, git rebase upstream/main, and finally git push > origin. > > This too is a hack, probably just as hacky as your hack. You don't, that's why we've linked to the repos in the PR. https://github.com/ceamac/netpbm-make-dist is used by Alpine and the other with by Arch. https://git.alpinelinux.org/aports/tree/community/netpbm/APKBUILD Best regards, Daniel From nobody Mon Aug 28 07:32:35 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZ2Mc5YM4z4r01V; Mon, 28 Aug 2023 07:32:44 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from APC01-SG2-obe.outbound.protection.outlook.com (mail-sgaapc01on2105.outbound.protection.outlook.com [40.107.215.105]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZ2Mb35C3z3CWY; Mon, 28 Aug 2023 07:32:43 +0000 (UTC) (envelope-from weh@microsoft.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=microsoft.com header.s=selector2 header.b=PLxewIZe; arc=pass ("microsoft.com:s=arcselector9901:i=1"); spf=pass (mx1.freebsd.org: domain of weh@microsoft.com designates 40.107.215.105 as permitted sender) smtp.mailfrom=weh@microsoft.com; dmarc=pass (policy=reject) header.from=microsoft.com ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=SJ9KOURO2/nY1FBfPCOV3FtpHkDkAjbehnOxQMgyK/6VWVPrHigeY0kDDtwak89x6zT5STBRyUApwKvQnjDtLRVAT1RJspBPU4ROdkrEfKwLtG1SfrVQombxa/5F1qRpEh1DtNux7PjmtAHnrBM6seOmewD5IylOtxp3mx4uRr15+Fm9jjUZgJ73QLjEpduVA81PCS1muUaBeJLW/O13JlJ/t/E/mdH4wFVOgsIPbb6EVMMHxXG+z72OQeOndlYioRpRNQUzKnYceNDPW14HIlwyWOD0Hbb+eWOVSlVrT1u580HsXB4IR717dNpVzPvvVGkSoM+n+IwyJQWCRtHTjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=P2U6L2iDPUzMhus1Cndi4Bym8K6J4WEs8tq9xnXEo80=; b=Nr8R5e0vx60QxJuLYGQlgM6mZxFF+AF5KR+2n1x/S3/96e1S+QRqK1zEbuJv6cAlikHY48K1gVeJdzxqjNGF1QQXp9ky1M4lDyrsczFlhzBCY2+oVdEP8cAvHSvjdRJV93tXXmvZqdSRG8kj5wjRfayyIRBvQDTk8wPJinNePSRjVp/6QFsE2sbfgTU1ata1NNx5sbqsAAyjDQu/1Gn3oOvAukwZ9RiB00M9W2EyOgiCGVpBMgHAUY5yC/jxiedVh1f+MYkELhWKwGFfjWYMOXtrNSguHGYLaTPHEwAOGwsuki0kVyWurke/rOHrci/jA1mwNUDC+1/ebHz0Z4yKgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P2U6L2iDPUzMhus1Cndi4Bym8K6J4WEs8tq9xnXEo80=; b=PLxewIZeRWegUK7zyfhm12yOJ8i6i+C6GMUfC96JZwhHTAr6lwI18Yi3shqLwrzS9VNuCGP1ABFpzgg5tjr7NPXJVOG8h4PO7BTOAmTdGtcgmppWNaSQegApO79hunnPdqDMERVio3270F78+zRqy53No/F2Ohgcq2Ketxo9f3c= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by SEZP153MB0791.APCP153.PROD.OUTLOOK.COM (2603:1096:101:a6::13) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.6; Mon, 28 Aug 2023 07:32:36 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83%7]) with mapi id 15.20.6768.005; Mon, 28 Aug 2023 07:32:36 +0000 From: Wei Hu To: "freebsd-hackers@FreeBSD.org" CC: "freebsd-net@FreeBSD.org" Subject: Very slow scp performance comparing to Linux Thread-Topic: Very slow scp performance comparing to Linux Thread-Index: AdnZfvJbaS4mlPy2QcSWspi9tSFg8Q== Date: Mon, 28 Aug 2023 07:32:35 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=c0bd6cc2-296c-4e45-ab32-f5c2dce362b4;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-08-28T07:11:28Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2P153MB0441:EE_|SEZP153MB0791:EE_ x-ms-office365-filtering-correlation-id: f3c5db47-01c9-41fa-5405-08dba798f2df x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: e+NCVxVbVI5oa5VonIW3B1F3HF47QEtE7ZsxQlo+j6CN5Et2d/nn41wWUA0qBYF0a1A5WNMWch+XID8VtDAIexn2zTG7gMyg84JxYqDFYyrhrgQD4IEFGYeVBfZF9D7M0QrwIVCS59Djs/QyNtX1mmaVrOhqflclyWIGgWW00hkyA3poQKSQA0m5gy2pppieCdw1TBwSjawXapTfMoQIlQoCx3YZ4m8HqqIj9/ln5fQO/e+VOJi08ps0txwWz85Mf65/OB8UMxCpt71znOxtrQHjCVCVFI2yRZ0jKab4fbenFOiIq2U01BC2iaFXqqPiyJEh+76r9P2G/TXcqEjc0sYOFl0vwT3j8AVrdjmju6YpJjROXQchVitdQW0C9vriHPI0i4p7gbrA3s9E5FiJugBCy8MYxpts40vuwKspCJTyZm2LCUCROUEkZC6ghSF7QggM0aFbNKfKUjYCErLCqW7kT3m7rP78Bnu5DqhkNwJbbB4VyWwUu1nztGP4d/ArMD0CjunEVA26zk4yMrV1gh/nJypgCLxRWyuctl4yIs6Nl/U9nXC1m7oRqh4nPohXw9RbP3eid9igSsTCrvCCWH6wgyIlk0rj0lTV6QHKqDKAKGMFAxACmR6vOC1YelAIRFndZBiKw7nrqNLiYj6vhYgoyX5eEjTzgXcd8FfngjhCmNqdQaexMzTtFp7EUSAbuKs5uvwH2g0ACg1S8bhtuQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(376002)(396003)(366004)(39860400002)(136003)(346002)(186009)(451199024)(1800799009)(4326008)(8676002)(8936002)(5660300002)(316002)(6916009)(2906002)(66556008)(66446008)(64756008)(33656002)(66476007)(66946007)(76116006)(450100002)(52536014)(12101799020)(41300700001)(6506007)(7696005)(26005)(9686003)(55016003)(38100700002)(38070700005)(122000001)(82950400001)(478600001)(82960400001)(71200400001)(8990500004)(86362001)(10290500003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?I5/G9n3AA2iOceDiVGp6Ue2awkO55hUw5RJUoY50bWLif9W8git5f3JuIgsx?= =?us-ascii?Q?BHkFh9nxFcqNBaxDPprUPLfYcuhG6ZUJtmbItR6ITsIIaK4j09ZY26C2HTRj?= =?us-ascii?Q?Kh2ijgsNSuHsEsmLuSF5VX3EbC/ss7JMd4LCzolTf8Xy9z109Sueb6B7Ff1k?= =?us-ascii?Q?ijpb3e4Hx6H+paD5pRX1UKD/z3370Nxu8fj8X3yy1DZWGdPIEG25OoXpLd6U?= =?us-ascii?Q?bNdillJCy9VYlx891yCeWFuu3p1H68xcBX465jlH48rdr8bn1eGjU/pq1h/c?= =?us-ascii?Q?jpm7oYwgRBvYbFuUDmfY6G151jwrXhct3RFOjPHckr94vikwT6uhglbLp5eM?= =?us-ascii?Q?OiNCiWYZbS7hVUtHzPrByer/0r1fre/tc1CX2MgbnCb6MEvGQ0oYUhkf31Kd?= =?us-ascii?Q?Fj/+9PLZ8RMK67UsSe9Ges1mHMbuSriQb9hSMCHLwhrS0xnuznaIdXQ2GWGA?= =?us-ascii?Q?GvrihEviwvOPiOvVa+nQInpQhGHwMZpDFLEjpGq+zwkGmG8LnL2Vb+3kXs4n?= =?us-ascii?Q?mkI1pp7q10zNTlhO76XWNvRf8G2s9ahdbpdXZjPfJxxWo8Weux6vAyvBkSkd?= =?us-ascii?Q?7c/wmkBHY7wMAljTynq0BUiBdZGM5Il1rGeX0DP5x+wPdCYV+Q6rpAmbOHOR?= =?us-ascii?Q?uHSq5J+ZugEii6Rza7H8mP2hm6HwciBVTG2BFxriBZ90Pn1T4axohC/zbCG/?= =?us-ascii?Q?vsPk7pXGLeXHf0aQqvabTubvLCWpaA8Os94Z3Y8E8MzMA+wIlqYQG9q+bDTf?= =?us-ascii?Q?uhqhLjwKk24/Da07BClgbP1gNENbFAQLHjNo2ndHSwNrsk2u90STlB0fWlOL?= =?us-ascii?Q?MG/qK5+xccTwRBWnTjGJWbcReYHkuZBes5QARfgAHXctoVAS1WYXzDZWxdPv?= =?us-ascii?Q?9oKHpi1PQWvXn6C7fZx6IQBen0Ls1yd868Csmnzr+aEldqRCeuI3cLQNNgpC?= =?us-ascii?Q?PaFVCnZX7UWm/NkGa7p0bTSs7NBYA1tSbo0QCZnUojXbE/sOwaFJ3ff9G5+L?= =?us-ascii?Q?dOG4D7pq5f/bPAQ++zHZwxgLoQrbtKg7BwV90wAyBkCnjTZE4NbJHpf1tLwL?= =?us-ascii?Q?7vj7RvKH0xvHZ7r02NCtZiirHvOx8qCbH2Tkmh12yuDrxravS1bTNYWMKsNs?= =?us-ascii?Q?BjZqL24yv5thefvTTHfq2s6ZRfC0OEEOrp4/OelFB/zAvFOPJ9l4A+oQrU2i?= =?us-ascii?Q?EptOcU1J9rrrkywf3/mBpf+PhH4NeDYfSInfQSsHSYuAarJlpouifjQcl61e?= =?us-ascii?Q?TVY6LnFGWk2YPICfI5GSBZoCzommke8kW2/vHC3+Sn9uKWzhtIld8iLbKV8N?= =?us-ascii?Q?Eg/AMMqtJl1o3SEZy5IN43ciYHD751898Nq9nSl4R829IG1TqmdjOMh7/BY3?= =?us-ascii?Q?R3xshKZu+lrUsNI7WUKQButO4AyUUlO+QS0HDx53SJcO0nYlfDspQXgc8GYo?= =?us-ascii?Q?/WNIUcR4faMqQ+1gBx0q4SuZgyTQye9C2RGUkL5sc9/p/29yjx1FnqvYNzuo?= =?us-ascii?Q?fZAs6Kf2E2L1k0glReDgwb/IRgkChh8pWx4Q5PgPthz4jncosg++t9sG5aew?= =?us-ascii?Q?TRJFd+Je6ybZ+vbTYGs=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: f3c5db47-01c9-41fa-5405-08dba798f2df X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Aug 2023 07:32:35.4400 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 4yHkZoylDoawJoZRxZqVC+dRmVDZNOIuwoyCBrtkvXO29a39erZY8TEy1VHPU6/JZZQU7tiqm2IqooTj9GJ4cg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SEZP153MB0791 X-Spamd-Result: default: False [-9.00 / 15.00]; WHITELIST_SPF_DKIM(-3.00)[microsoft.com:d:+,microsoft.com:s:+]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; DWL_DNSWL_LOW(-1.00)[microsoft.com:dkim]; ARC_ALLOW(-1.00)[microsoft.com:s=arcselector9901:i=1]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[microsoft.com,reject]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; R_DKIM_ALLOW(-0.20)[microsoft.com:s=selector2]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.104.0.0/14, country:US]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org,freebsd-net@freebsd.org]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[40.107.215.105:from]; FROM_HAS_DN(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; DKIM_TRACE(0.00)[microsoft.com:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[40.107.215.105:from] X-Spamd-Bar: -------- X-Rspamd-Queue-Id: 4RZ2Mb35C3z3CWY Hi, When I was testing a new NIC, I found the single stream scp performance was= almost 8 time slower than Linux on the RX side. Initially I thought it mig= ht be something with the NIC. But when I switched to sending the file on lo= calhost, the numbers stay the same.=20 Here I was sending a 2GB file from sender to receiver using scp. FreeBSD i= s a recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. = Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has 16 = vcpus, while Ubuntu VM has 4 vcpu. Sender Receiver throughput Linux FreeBSD 70 MB/s Linux Linux 550 MB/s FreeBSD FreeBSD 70 MB/s FreeBSD Linux 350 MB/s FreeBSD localhost 70 MB/s Linux localhost 550 MB/s >From theses test, it seems I can rule out the issue on NIC and its driver. = Looks the FreeBSD kernel network stack is much slower than Linux on single = stream TCP, or there are some problem with scp? I also tried turning on following kernel parameters on FreeBSD kernel. But = it makes no difference, neither do the other tcp cc algorithms such as htcp= and newreno. net.inet.tcp.soreceive_stream=3D"1" net.isr.maxthreads=3D"-1" net.isr.bindthreads=3D"1" net.inet.ip.intr_queue_maxlen=3D2048 net.inet.tcp.recvbuf_max=3D16777216 net.inet.tcp.recvspace=3D419430 net.inet.tcp.sendbuf_max=3D16777216 net.inet.tcp.sendspace=3D209715 kern.ipc.maxsockbuf=3D16777216 Any ideas? Thanks, Wei From nobody Mon Aug 28 08:28:28 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZ3c70XQ0z4rHhL for ; Mon, 28 Aug 2023 08:28:39 +0000 (UTC) (envelope-from zmey20000@yahoo.com) Received: from sonic315-8.consmr.mail.gq1.yahoo.com (sonic315-8.consmr.mail.gq1.yahoo.com [98.137.65.32]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZ3c61y1vz3J0S for ; Mon, 28 Aug 2023 08:28:37 +0000 (UTC) (envelope-from zmey20000@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=c7zLn5eW; spf=pass (mx1.freebsd.org: domain of zmey20000@yahoo.com designates 98.137.65.32 as permitted sender) smtp.mailfrom=zmey20000@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693211315; bh=AbQFcCMxzQSw7cXhRZ2KcF1w17ak4FhfhPKmzpeIoOU=; h=Date:Subject:To:References:From:In-Reply-To:From:Subject:Reply-To; b=c7zLn5eW1OQq6gi5o/gZx1NWQatwctT4Nq+63nhdKf0ILkEE3pxXf4OWT4H1Cpjej+nuc7XYys5q4cpGgSQCS4cLwRiwrYIKjYT6c8ZnZJlHOFtK97LX9j8E/lwwP1+NNTOdMwggOAq2OHpaevPG2SVFo2aJ4p7fzXyVFyMKkGEHdi+H6hq45kaxoKOaJjlnZoQfdWLhjDAknzaASEQ2vUy8UTBsVHnCeYEhuMWX97tvBpoB9ASJhKODG1NKUUaP5M3LdNQhwq6VHixnSFbYEB12mNBdjfK3MTptJG5PrWybE2qLoSyulc5W7m9t2KypPuHk1Kh3Kk2YHS1N24gUGg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693211315; bh=VZinGwKkTLpehSnPxP1tKJdf+Qdp4VAXnUvccx6h/Kf=; h=X-Sonic-MF:Date:Subject:To:From:From:Subject; b=ZqkE2hgvae2dhdHHsSjWyjKF2khucKYOz11anpAPRf/zP3ZON6GxC31RI+C+kfFS/ZRtr9NAU2gSvFHuqcejBI9hmJ5/HxFk7i9LlncSLO0PWEoqE4gpsC1BNO9MoDsNeYKHSZb/4GyE5qYsNgnPGMdgEbqpXU3cvnvND958cmNsFfrNHIrEDASlljGxqJ/E//0V0y8idPO/kERAiIEv07jB7RErrhGk1D5epQSIhi+Q4HG5dR/yxt7boM2Icwdhd5nsi3+z47GRzuSixR5eB4zttaz3OPjg0gHAv+GbA2wbv63Aj+5WSe2pPjJLlWyP7UfkQQj0rO4PTC2quRfY1w== X-YMail-OSG: 0gXzmzEVM1kJ3rIyehMLwdDNbI.hEdnLHfOJmUf6LX0SGdZbI3VfBSQrh55pw23 fzWxGClsSxvBUvFlY0g5J.4SOIuYU2hKHA7e0CgOA2SXfPyboPJBtqY.3WlCUSIc7eeR3uIRnTRh VFL._2I8IMnc1P1BwB6d6P4jk.6rcJ3QJB9o1ziQnUMQUg8pM.jgm_S9FXKngDShrUfSR2ChVTNm fe_Tw_rX.k5kENxjNUSRv4ClJBUzVhxnIBKBleAei8HkWbxGzgGF7elZJSkf9XWs8lAoVMxKehkJ E9bS9VUbgAooBVCg5OWeoY_pzSJSJkEiLkzw7J31YRlU9czhUn0O7k3qBT9xIxi1QD04gKukGVDc HA7NTDWCDHfMk5hFVMEn2u8ksMWtcDhq5xuypP2fdG3hdpEmjvU_19DNhPL6kYzWOjNV1FrH.CHx MGlrjBk4AyywO47B5RzBkrvRoxlIGdS9noRJyOy37hyJBEBDAjBlyW91zWA3zwLzK9d5fpvGoWZE uYRNLaFZ7A22w07UUeExMolTO7XcsKD_KgOPidn5hRcAI9_1M3pk8UL77QJAiBl0Z4nCZDtpXdtL lXhEG2AYgUsRmkwS.dSgb5Qu7EwLSRzTFEy1cwI8BfK0jTbyubKf2Lsh7Pn8sS_3R554hnjuH6BT a7HbJJKBsQY0QXeXPO9fUp6425vqTr2k24sEHm9ik8dZkBtcMdPS4Z5705AUWbbMKAKTUCNnvZOb LbEsb0yQqFSOtZHFf08iia72PVVvxAVEbGbPFPIzK6ceCgwnAu3gJcLzHKFUcoZ_4REP8402UNrB fsqwxsDJT6kXSGk4X4CNeWCnsLhSyfKyHz9TCeJYQRXnIoFpWjAr3NBlJp2Ew9ywhdw5W_nHmv31 9tgahS0cqgzBhSxAWJ.AEWWG_INXbt6Yw7qeliHxVd7PhhN9pqkLfPSpNUpywxrSRR7aCACoutVa 0zn5d_77VOv0BMLueP.Kw6Zry5uQ8omz86sjGlvo55uNWRDlLXAS4.9_cvyjiutMNJjiN4qCCm96 5Tua8JIG2qLKSIQuJWwDLJsU2ABOuD97835JyRz9WeR7N1u8eC.U5rovn0eksZIscZMrXVoyat1c cT3CYveOrpFBp6NX3.yXnlFQiWuyP0gvwGI0lXXnrBt0zvkTMSm.lUVvRSj06uIn0CUsOVQz2yX3 ZygS_0Bq56XPGuhBJnHWyLNKtzfr9bMsMkvq13mi0YSGjjhFW62YHxqu77dv2qR1NtAb8J4dkngw 02X68Oe6UpzaGfHliuVwId9gVoyqMeAmyv4Z9sq7bNOCxaMcZcEH8jj60MGRmR3Z8JFB9cSGA4m3 yWshniwk_zgohaj2fgqYc.xXe.lZNzcAuFxHkQo_L0BXS58OIwYKHoME24F.Vq7jtw75RTeMGSbm HHLL7JAYtScfaPsSNeMQNui6UUllkvSE3x1r4Ml5P_sxkavHK27CAJq8WFTSOO495m5VWOLf6LXl RcFYu.lny1iTRgCqsOO_ulZbgseBj.brrucjoo7ojDq_qgzBlhWlY8YCcnRMRg_rQdgP5YZevfe2 05vrEQUCahTDqgPydBI6iISPidD5if3yYNWxScpc1_3WDq3Pxpmxqx0ymDJI0hgi3.D838wYEcP6 AILFsHZP2Ei2cUMaRFMssNCUAmmNVDwSVh3ZRR09cFL27sR.pJgQvvCwlW1HgScN0JYYmXSlCRoW SdMcZGYqIilHkS9VQK27HWlkDp7SNlfJmvddmS7v69vtEDl6OuqWC3O1pFf2jw_9IRTvsyqdS_Bz vMfFThoEfWO5cu4gSZJvGvYiyLvtzHbcBbCwpoC0b8xrNGXz92kUPrip0OiKFZJIB1HE121_I95w ut01pXkp64DoHenwdhEsPDzzHxVxkVhQVIHK4..aUbO5DuhXJ_MCDgDnapnYoXYynnBT13g2.nNw U9HgLXhu5K.Tbx6N0rldS5UmCzm8.l0d7icYoZmYD02EY36bDvsl4j7n6UDpK.rwUtBvf34HGzde 5DM6ZVx_8X7KwMLs09CIhpuP6g5GGJkPYwiR1QNc73jMqccjGStgzPEOa3gKmiYVa0U8n6YRRhdy J0uxi2hQUjr1tLIIMmHh64eLXp8PGU0Rj7T1ILwS52O3UBBSvfEiuWws6aSGriTd_IOv5HKm7Fg1 Lhm5t7jgegHeqgLyVZjYhSC0mHNEEFsn1GKVCqaPoUttpH0vXBf_s_3nOGPM8jVEqyDfow95Nv2T 6WaGS7fPewxf4Bw-- X-Sonic-MF: X-Sonic-ID: ba7ebb67-5ef3-4d5f-a3b3-37e3e9de0e41 Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Mon, 28 Aug 2023 08:28:35 +0000 Received: by hermes--production-ir2-5cc57b9c45-mv62g (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID b233e9053ea0e299500d7dbedd0a9223; Mon, 28 Aug 2023 08:28:29 +0000 (UTC) Message-ID: <6952b7fd-f90f-2677-1997-d0c708cb559e@yahoo.com> Date: Mon, 28 Aug 2023 10:28:28 +0200 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Re: Very slow scp performance comparing to Linux To: freebsd-hackers@freebsd.org References: Content-Language: en-US From: Mikhail Zakharov In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Mailer: WebService/1.1.21763 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; NEURAL_HAM_SHORT(-1.00)[-0.996]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.32:from]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.32:from]; DKIM_TRACE(0.00)[yahoo.com:+]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[yahoo.com]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RZ3c61y1vz3J0S Hi, What is the drive back-end of your machines? I mean, what is seem to be fast copied could still be cached in filesystem/storage buffers but not actually written on the disk. Best, Mike On 8/28/2023 9:32 AM, Wei Hu wrote: > Hi, > > When I was testing a new NIC, I found the single stream scp performance was almost 8 time slower than Linux on the RX side. Initially I thought it might be something with the NIC. But when I switched to sending the file on localhost, the numbers stay the same. > > Here I was sending a 2GB file from sender to receiver using scp. FreeBSD is a recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has 16 vcpus, while Ubuntu VM has 4 vcpu. > > Sender Receiver throughput > Linux FreeBSD 70 MB/s > Linux Linux 550 MB/s > FreeBSD FreeBSD 70 MB/s > FreeBSD Linux 350 MB/s > FreeBSD localhost 70 MB/s > Linux localhost 550 MB/s > > From theses test, it seems I can rule out the issue on NIC and its driver. Looks the FreeBSD kernel network stack is much slower than Linux on single stream TCP, or there are some problem with scp? > > I also tried turning on following kernel parameters on FreeBSD kernel. But it makes no difference, neither do the other tcp cc algorithms such as htcp and newreno. > > net.inet.tcp.soreceive_stream="1" > net.isr.maxthreads="-1" > net.isr.bindthreads="1" > > net.inet.ip.intr_queue_maxlen=2048 > net.inet.tcp.recvbuf_max=16777216 > net.inet.tcp.recvspace=419430 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.sendspace=209715 > kern.ipc.maxsockbuf=16777216 > > Any ideas? > > Thanks, > Wei > From nobody Mon Aug 28 10:03:02 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZ5jH2GCMz4rMmq for ; Mon, 28 Aug 2023 10:03:15 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from HK2P15301CU002.outbound.protection.outlook.com (mail-eastasiaazlp170100001.outbound.protection.outlook.com [IPv6:2a01:111:f403:c400::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZ5jG0Xsgz3Rpw for ; Mon, 28 Aug 2023 10:03:13 +0000 (UTC) (envelope-from weh@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hymMV0fKPINgcnYrfWw5FDcw9FwYn1F+RGk3Isu5X5/ydBhRGVOLW/Iu05miiwgbvuhzffqrrrxjlxNnQL7/tgnKgr+YTmy2nAimQ/uxkxMChV9EbmQEBrKBuKJX6ye56XJm0HkHZUj1xVt368y7/oHmQLMrXyoKOA3nKJCQKZVfO7vATy+5YAV7QTVV6p2QRIrIymODlJtMcTGlInPoUeGvpY450+yJ0d7aq3U4hwmylOIQUOPxkfJF6pwc2DBkLuuiki9XXz2/qPpDb55yP7MBkS0xWvdUcAcVS8XxgeX72NhR4Y1YsjIILV8VFvDuP1sUaNivfLeEe5JlLYol6g== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=WvSV6+ijK4D/EnQ5ISVvVJbcM9DOnnbCQ2YrGpJ9WIc=; b=R4IqoUWRHwrUAUWPIdyIlZL1NxtoPwXPYxtdfwIOTyqC9+eRcr2qoTczHzOH3Hm71VOLKHVuqar39i6JjPzDJVdI4nDwx0hM8sRKU9FBUlwGu/TizvuYZ/HC/tp5gvK00K9hduoE74E62ZkkvT/3/8/m5XHxh5q08Xxp8TnfSF/eRSKiV8oqk9rcXnOd3lV71vKo7YIjjTysDEf4xmkCYvHbYikpQS+PR2eupKGqbYVpi/ISbofn4gt5QbqUVyAVQv1Frmfelwep0Zfj/ZRifgQzHD5N0ixo/Vrh4qxoK4W4sEAjfBItjkb3O4ztch6+dLDaSS+JkAmK5iIOEzC+/g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WvSV6+ijK4D/EnQ5ISVvVJbcM9DOnnbCQ2YrGpJ9WIc=; b=ClEsQIVj+CznPVSgca/ntgMk/DpTKSZ076riLjOjuHPtB6lG1MLu6L7x/VgX+QqZvIo4izzj603cSdSJuv851Jlbp8/yQxCkary/ETuyQEhmQ+XPz6JyDiuSVirQdliBB4j/LQe8aKT/X9UmvaCKB0A89E8RR+hv2sIEhdYB3Q4= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by PSAP153MB0501.APCP153.PROD.OUTLOOK.COM (2603:1096:301:63::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.6; Mon, 28 Aug 2023 10:03:03 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83%7]) with mapi id 15.20.6768.005; Mon, 28 Aug 2023 10:03:02 +0000 From: Wei Hu To: Mikhail Zakharov , "freebsd-hackers@freebsd.org" Subject: RE: Very slow scp performance comparing to Linux Thread-Topic: Very slow scp performance comparing to Linux Thread-Index: AdnZfvJbaS4mlPy2QcSWspi9tSFg8QACqxQAAAM0yTA= Date: Mon, 28 Aug 2023 10:03:02 +0000 Message-ID: References: <6952b7fd-f90f-2677-1997-d0c708cb559e@yahoo.com> In-Reply-To: <6952b7fd-f90f-2677-1997-d0c708cb559e@yahoo.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=121293ab-a406-4fd4-b2ba-d3f7878a6681;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-08-28T10:00:16Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2P153MB0441:EE_|PSAP153MB0501:EE_ x-ms-office365-filtering-correlation-id: 11b27bc6-eed1-4af8-d8c7-08dba7adf796 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 3FbPRiTjXAUmQTm9+IJsTF0Dno1v30BMiaqPhMg4THcMNDSIat1VyodSC3jN9KjGXe6zl5MM7ysCIAjftnpkFxDKfvhoUNwfLSKnvLl2RvmsVGLoweb3hqP7uJgxVIxt6+snPfopDQozta1QpifdOImA+6vsJepwkBqgJt7ZG5lXVddJN5UWatKoP/DBO4fvmNaH2muQkBoDMyp3wL2+oD0+HhvZNGZHDdkBlIwk9xrlbtzNqsM966HG+DHBkgk3KTOvxSVJeOQ0xYV2kuGXY40U3ck2YOKpDYzW0UgpDkhRWmueThNLxcMVRy0e1GcocyUm6xly076GhuAo2Y/mHpK7kgkRU/ENcxy+L0+izwIlYjiopyVjP72UM/7T8dSxmy8rMTktZ8QYDD66pa+WCvWGkxcOkkN+D3p3OCo3yd5B3eDmgtTNYgNtWOEOWqaJkw94pA77U13KXICtTPcJgt8OOGAaEr48A681ee+ICFCJzzrpNmCBjdlD+EFMEj7uugaml1S+y5J3NJVFPYaso/C99YAKtzeZLWBaX765Dgsjfz/0IkjRlgcUmkBtpLCWxBjjr6HYEqqDqjMSnkYMMGG1k4D8Ln7KX8b8bvME0nyK8TNJeDd3Y54wLvD/DD+j9yQ0cSLmINx7iw85WD3ow5zaPDKbXVnT/iNRtS4ivKXtjM0OVSgZ60R0N6rBbs2g x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(366004)(39860400002)(346002)(376002)(396003)(136003)(1800799009)(186009)(451199024)(41300700001)(82950400001)(122000001)(38100700002)(71200400001)(38070700005)(86362001)(10290500003)(82960400001)(966005)(478600001)(83380400001)(8990500004)(26005)(9686003)(53546011)(7696005)(6506007)(55016003)(66946007)(76116006)(66556008)(110136005)(64756008)(66446008)(316002)(2906002)(33656002)(66476007)(8676002)(8936002)(5660300002)(52536014)(12101799020);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?e8xI0jQaNXQVqKt7S/0UHQwo+K5ke5wGuAh4xCONQPytUTECyhBV00cFUVyU?= =?us-ascii?Q?nzNnpOJmZ4HuaMI08AgkAySg/mLJeZXaKgdxU1oxkXVA42aUFIU6hFRcCKYT?= =?us-ascii?Q?rnD6PVBQSkovkcttxmwEkehG/c8DjZ2amhlKKMTBYpjyvYfX6wjRT7J3ZMVM?= =?us-ascii?Q?WW8y8WMDLE1ARMMKe3w9wsbuXAepeRKq9Mt+BfBdIoZcCvLe8pstpC9f6XQL?= =?us-ascii?Q?DOFAj3SOuDPTEmUZ1Zjuovi3SIovwZu3uG0rB89yh+fmqf9hahHMz0vQG/9w?= =?us-ascii?Q?k7pgYxjSSxdd4hXGV+pMBNKzqbRyh+NSgwdpCsHysq5+MPT9JCM1pQWvcVt9?= =?us-ascii?Q?jMXVuf63MINiolNtft82KqbKDqJ7YGUdtq5zBMEI8CdbNKoJcwIIWNnasfzA?= =?us-ascii?Q?GChBx4vhGSRVjSHnX9K6HQIuYyfyyXX9XGDfGFI18K/po3o2npy+Yxos9JQt?= =?us-ascii?Q?sf8rrOA2dl4+IfoZvNoDJCYAyt7NvV3W94X7q2LYTLe4K6mKlVrObmfYTfG1?= =?us-ascii?Q?ae64hCcPrxs122OOTf5hursDEFbq66zMzO7D3hfHHPhkThPINExa6qo3855v?= =?us-ascii?Q?Ri9eMQaW7QG5uoj/g15DPMDY5yC6UAx74hmYwZKEv2Yfp3SnUJt5SJToon9U?= =?us-ascii?Q?AsV2AlPey4D26GEN9TPA44n1joQIvjqfYAW/TCYYvSRsIu7/NUAz1WXIp5Z9?= =?us-ascii?Q?Iupwz02MmXBK3Beatl5j8OZz1pMVlzVEuQz0Hihu0TzTtXo4oOqIBEa/LAMA?= =?us-ascii?Q?YgTVL3YMJeEtMTE1GDAX4E1Cc22eB1244L/dpJfT9/9g5Q+r975NCf/Ar8QJ?= =?us-ascii?Q?DKj9Z4sGWVAYvJoDT/5FIwYaPes4qGhQs6Q2ows4udFEYHQFjpH2m3wBPGFK?= =?us-ascii?Q?lbXedzCnXeiG8ZNz3nLm66i6REaj7XrdJ8GQ3DrNAT7muGBJ976vaMN2Vp1o?= =?us-ascii?Q?6ZO7gHIra/sQ47kPL54lYu8HEsNB3V00M191wWBNJw93JGXh3caBf9MXk8+5?= =?us-ascii?Q?o4A/9rkwKFbs7u/iCRliOntbuPpekzFB0VR40D+Mydy8cTRvU9xPDCTe7OVc?= =?us-ascii?Q?sjaOtQ97tkTc4I3QOisnrH1cIzVkVnR9ymr6FkEBvO7xjg7glBhtgy+kpWfR?= =?us-ascii?Q?7UYEmx3HiC7M6YtQseZpH5ykLTYWCpLw5c/3oAb4RfpmvDhO/rXCw5u6khQp?= =?us-ascii?Q?N9S9L/z+cDi9eIMOtig2ak/ZmGcr/1D4le5USGyPAjDwIkXadC9o9Mn47SKc?= =?us-ascii?Q?/o8JlV/K3Aajc+6SQV+6y7ANUNuwXbpY2L7LCgR4maMEwausnT76HkSrcsWd?= =?us-ascii?Q?S/wqhs5Wwhp4M2T5mrWfZ5Kh/Y9ixogUUu9Hzm7qxdRETkek8LqrQOTM3tY0?= =?us-ascii?Q?VV4ogbeJLSUDEGm4rMycvDbJ47LqC3y9cyjZpRmbvQztcf7CqznBUfN32mPt?= =?us-ascii?Q?3R1aaCl38lzWX67rXnu0pHOZzfQNS0UwUEOH0Ir7s1t9v087J98ehzVzZ2+g?= =?us-ascii?Q?m1FjGLWrZWn9O/bLN3toFRSbT8btSDgAJMAu2NwPncCsRCXQGGySej0i/Xmb?= =?us-ascii?Q?19n8LMtfJrawgF/ic3g=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 11b27bc6-eed1-4af8-d8c7-08dba7adf796 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 Aug 2023 10:03:02.8070 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: ob4tlHxI9193V7YF1NlbwZAS3yUEGcmg7h/x8dRzo8O3A+qx2gcDnOVT8hYT3Z4/F9nJFShtmzCGxCKtp6/9Ig== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PSAP153MB0501 X-Rspamd-Queue-Id: 4RZ5jG0Xsgz3Rpw X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US] Hi Mike, All VMs are with same type and they are in Azure. If the copy on Linux is b= eing cached on the RX side, so is FreeBSD? Thanks, Wei > -----Original Message----- > From: owner-freebsd-hackers@freebsd.org hackers@freebsd.org> On Behalf Of Mikhail Zakharov > Sent: Monday, August 28, 2023 4:28 PM > To: freebsd-hackers@freebsd.org > Subject: Re: Very slow scp performance comparing to Linux >=20 > [You don't often get email from zmey20000@yahoo.com. Learn why this is > important at https://aka.ms/LearnAboutSenderIdentification ] >=20 > Hi, >=20 > What is the drive back-end of your machines? I mean, what is seem to be f= ast > copied could still be cached in filesystem/storage buffers but not actual= ly > written on the disk. >=20 > Best, > Mike >=20 >=20 > On 8/28/2023 9:32 AM, Wei Hu wrote: > > Hi, > > > > When I was testing a new NIC, I found the single stream scp performance > was almost 8 time slower than Linux on the RX side. Initially I thought i= t might > be something with the NIC. But when I switched to sending the file on > localhost, the numbers stay the same. > > > > Here I was sending a 2GB file from sender to receiver using scp. FreeB= SD is a > recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. > Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has > 16 vcpus, while Ubuntu VM has 4 vcpu. > > > > Sender Receiver throughput > > Linux FreeBSD 70 MB/s > > Linux Linux 550 MB/s > > FreeBSD FreeBSD 70 MB/s > > FreeBSD Linux 350 MB/s > > FreeBSD localhost 70 MB/s > > Linux localhost 550 MB/s > > > > From theses test, it seems I can rule out the issue on NIC and its dri= ver. > Looks the FreeBSD kernel network stack is much slower than Linux on singl= e > stream TCP, or there are some problem with scp? > > > > I also tried turning on following kernel parameters on FreeBSD kernel. = But it > makes no difference, neither do the other tcp cc algorithms such as htcp = and > newreno. > > > > net.inet.tcp.soreceive_stream=3D"1" > > net.isr.maxthreads=3D"-1" > > net.isr.bindthreads=3D"1" > > > > net.inet.ip.intr_queue_maxlen=3D2048 > > net.inet.tcp.recvbuf_max=3D16777216 > > net.inet.tcp.recvspace=3D419430 > > net.inet.tcp.sendbuf_max=3D16777216 > > net.inet.tcp.sendspace=3D209715 > > kern.ipc.maxsockbuf=3D16777216 > > > > Any ideas? > > > > Thanks, > > Wei > > From nobody Mon Aug 28 10:48:42 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZ6kc2j4Vz4rPxx for ; Mon, 28 Aug 2023 10:49:28 +0000 (UTC) (envelope-from mad@madpilot.net) Received: from mail.madpilot.net (vogon.madpilot.net [159.69.1.99]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZ6kb6kDWz3WV3 for ; Mon, 28 Aug 2023 10:49:27 +0000 (UTC) (envelope-from mad@madpilot.net) Authentication-Results: mx1.freebsd.org; none Received: from mail (mail [IPv6:fd5c:5351:d272::3]) by mail.madpilot.net (Postfix) with ESMTP id 4RZ6jr0Y3gz6db4; Mon, 28 Aug 2023 12:48:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=madpilot.net; h= content-transfer-encoding:content-type:content-type:in-reply-to :from:from:content-language:references:subject:subject:date:date :message-id:received; s=bjowvop61wgh; t=1693219725; x= 1695034126; bh=HAIbijX05yNTfUIGnieYtzOFnr+EbcuheQZJb7Q+2yk=; b=k SE/Acxx9YECBvru/pf2KjGWy8tzkG/+5R7F+e+1/d81tbg1KoUIw5E/WqMHUQYrW 8D1Te+w1++G1idAf0Mc6iGeBcWfn+pW0UxBBVXBO7peob4+w6VPHNWqlLy7dTt9i lNr0925g6PbTeJquN+LlapmDt5pk5IdaP/uTHMSP9wCc4OW9rtwKhey7tmXoEZ1s Kt/+y+IzMdX8B0fAbppxDtTyMuSw43uNkO8qZYaBHFDL6dcGkYT0ZPL0FOrsJuM+ O+XmxYIZ461kNwZRMWxI3fN4u3+lmqb4xrlrZ0QrjOYNQLONoO1FE6/U5LgH9dXL VNDLhRv9O86ysVlja1rIw== Received: from mail.madpilot.net ([IPv6:fd5c:5351:d272::3]) by mail (mail.madpilot.net [IPv6:fd5c:5351:d272::3]) (amavisd-new, port 10026) with ESMTP id AFYdilZvaw4b; Mon, 28 Aug 2023 12:48:45 +0200 (CEST) Message-ID: <455dc69a-dd15-18d9-8e93-91bbff3997e4@madpilot.net> Date: Mon, 28 Aug 2023 12:48:42 +0200 Subject: Re: Very slow scp performance comparing to Linux To: Wei Hu , Mikhail Zakharov , "freebsd-hackers@freebsd.org" References: <6952b7fd-f90f-2677-1997-d0c708cb559e@yahoo.com> Content-Language: en-US From: Guido Falsi In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4RZ6kb6kDWz3WV3 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:159.69.0.0/16, country:DE] List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org On 28/08/23 12:03, Wei Hu wrote: > Hi Mike, > > All VMs are with same type and they are in Azure. If the copy on Linux is being cached on the RX side, so is FreeBSD? > You could perform the test using files in memory filesystems (tmpfs or the like). This would factor out disk performance whatever the backend. -- Guido Falsi From nobody Mon Aug 28 14:01:56 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZC0r2GWwz4rZnq; Mon, 28 Aug 2023 14:02:04 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [IPv6:2607:f3e0:0:1::12]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smarthost1.sentex.ca", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZC0r0Cq3z4MqG; Mon, 28 Aug 2023 14:02:03 +0000 (UTC) (envelope-from mike@sentex.net) Authentication-Results: mx1.freebsd.org; none Received: from pyroxene2a.sentex.ca (pyroxene19.sentex.ca [199.212.134.19]) by smarthost1.sentex.ca (8.17.1/8.16.1) with ESMTPS id 37SE1udR081946 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Mon, 28 Aug 2023 10:01:56 -0400 (EDT) (envelope-from mike@sentex.net) Received: from [IPV6:2607:f3e0:0:4:5558:f64a:f719:8ead] ([IPv6:2607:f3e0:0:4:5558:f64a:f719:8ead]) by pyroxene2a.sentex.ca (8.16.1/8.15.2) with ESMTPS id 37SE1uP0028691 (version=TLSv1.3 cipher=TLS_AES_128_GCM_SHA256 bits=128 verify=NO); Mon, 28 Aug 2023 10:01:56 -0400 (EDT) (envelope-from mike@sentex.net) Message-ID: <73af5253-55ba-13d1-6d31-9bc96233e7d5@sentex.net> Date: Mon, 28 Aug 2023 10:01:56 -0400 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.14.0 Subject: Re: Very slow scp performance comparing to Linux Content-Language: en-US To: Wei Hu , "freebsd-hackers@FreeBSD.org" Cc: "freebsd-net@FreeBSD.org" References: From: mike tancsa In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 2.84 on 64.7.153.18 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:11647, ipnet:2607:f3e0::/32, country:CA] X-Rspamd-Queue-Id: 4RZC0r0Cq3z4MqG On 8/28/2023 3:32 AM, Wei Hu wrote: > Hi, > > When I was testing a new NIC, I found the single stream scp performance was almost 8 time slower than Linux on the RX side. Initially I thought it might be something with the NIC. But when I switched to sending the file on localhost, the numbers stay the same. > Just curious, how does iperf3 perform in comparison ?     ---Mike From nobody Mon Aug 28 15:43:03 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZFFl6gGsz4rgxf for ; Mon, 28 Aug 2023 15:43:23 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic315-54.consmr.mail.gq1.yahoo.com (sonic315-54.consmr.mail.gq1.yahoo.com [98.137.65.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZFFk4mQzz4fd0 for ; Mon, 28 Aug 2023 15:43:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=Sgt+2E6+; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693237400; bh=pjs9FuzOOW2K3pdqNIJ4qcYFn+CsjGZZNcX+z2WJ2+U=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=Sgt+2E6+KMRVSdAnx4zJl9KRJUultmFrLJ1tQWxia8ikv3LXwUctKPJsJrHCFrF+Eg/HUr9yuL+yQxWSI93xIhyo+6WHRxJtUUV5ECtQb3fDZEH2jW+ARG/+bWP+QQDKOY33M8xvYlY971/W7xPkGsnbZTcHFwIQOPnkboENpn3k2ScnbzrSfeWzN6PnRkQ4RN+1gE6HWey3LQ+w0OA+XvB+m4JelXXGuwGijc1KlBWJbLAJ3das1XU6uv9LlN7te/ik8wIaJD+xgCydVFK3wjkN6kU80tELX4868KXWdMrobEZFRc9ISXM2r58ST/El8/9cT9jQjqZTqbWJHd9K/w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693237400; bh=Hm10xH0HnaZw2zzbA5Y3WsxWDLz0O591OPpcUo2Mv9l=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=qkAh5R+62jIld3YbLIFUPWAift1qTiukdwHwo6RFsqBzsltKXNUx44HSonDNWY9ZDoaQqWYArGnfAqc4xRjclEG6l9BKtJNwEa0G3As1+s+r6QrGirhIhjhG65SBR1zmrY/Ji19Tz8YYqlt0dAX2yk0UKAZjLgwdYfx+1JWn4KnJHwOqEvEtZdPnfrjefRq/+mzT1PZ3rg0A8vKdldTqamj1W5j9J6CKBwgS8SGN8tJK4Eo6JG+JKhVdnb9/Zqml6WxL1eVACYMlZANpaBz09Yk9pyxhbPpp2V+8w8rg8ruvGRrkkvYNqpvTs53GtvDoxGPTgW5eqp1SqYwOTqyUJg== X-YMail-OSG: ruXq62wVM1lBUQ6aiK5jB31YPJA_0q.ZAQkbfa7nE9UJYdHKLBqF.vAlS63bVgM ThrBUuxzY5kVDaenb9SgCsotHp14lwfGbgwOVOgytjh6PRYoAoeMj7jJoAV9o40otJmpuO4dhvui 1xVIWQitS6S6SFlxJFSNGyUsYg6K0.40afgghDgf0sGyTLOTeWckSaINztfU4l58toB8HOiWG3q3 T7PKpjJljFTHmf0uaOZkxUGrWPTT8pD15deDz2kXdYHbD87i5ApMwaqFQmvzOTkLBRXiBqZg20jh A8UOxGEfYso1tzyXPT9OhTvytSnb6PWz2Fe99HiR_hKNqknwMlahCrgI80zqaQBnlLO5t3KE7BZ4 _IoK5d9PJUuB8aJOX6zv.Mfen.BXU3zRyv468aolbhb_1grXzfv0BSSPpcKBFznrQdKWEkSWCl57 FbzNUhzFTKkIC4Qnz9w5e4KD_ziZSWecggGVzn1fs3DE0d_DpCc_Ov2U44QPOR8jLm1jX_tKIF0A h3B9VC._B0Iac2ZNVYFC_JYUYejawrviX8XAsNY_0PvyCenJyYpot9NnqQ4vlZB7cymAeXc0vtHK 9htzALZdtz_wFCKRmKNihWM.ZCGuMxoYnfVJlTNwRr20d5oTLgP9mZYwtVzS84nLCOvZ5CXPa3vS U4dA8sBE_zEoJj5wMV3Pyt2u6du0p3paf.QXRf.dGrht7K8RkUG3J1fBIUDeNE.XCMKTcmIWoRWj 2MfzZmQfNPWAx037zn7YCL3AC85Kq6NFWqgZO_qQ83f_by5vioOo.gsHY_QQby6V1Nk4VtPLMlw6 PUVgRnFQCjIrNugAZOldSsa4B1uMOss7gLQDyTUKT1nKyuwQagcqR02aNM7K5H4IOa0KGpF5GQS7 J8AtEB39Tjt90RoZ0MCZPksdozoB1skm2wcQix5aUmOLVNOD.ffRd2sm8mzMo9ARzYqizTHPHuNd PPMNsK4Uhn7b0mpuvCZFzjOT8sJwi5JaZR0eqkfvKR_YppPI89_N3XQroY00NlKzXbGVUCLMtOkz XYCcOoXCMCkdJFUpKc8DbX2M1MvHR4mPKlBlq9bac_zQbAjVhD8qZj0VrxVxpSu25xOez7.vKKH8 93aEbP2R77O7uNBXJfl8DZ_OQqdb6gYgsrzXFr7os8.c4TMFgAnj.QrH5BtsBqhqf7eXgEcG7GGW 4IYZ5sWYDxh3zrdy4A__FPdhes1gzuXTwVq3MZVAQMetDt3trTcFhNs9rI95rK.CF6RyLA0ukB7t kzERrpvi8LjMr3zx8_xoSzk3Yp.Fp2_SNldpX24Sm1OpAQaxybuAkdgw8ZB2Bzq7w1h13_1HgmXH 1ofST7e0SzviXPl5KohojEvHyUm8yfVYHyZqKbaLgk1Ewf.vDScAYi_0vtN3sMrvMvwZqG1rDUP4 P3cuG87RwpixDLOhbsfLmy23USn3u.U.r3CA3kEHzEvJfBNjOkOkeG2_J70k9eHYj9cvYrVOkRG6 exm3qvoWuK5bTn.P21Yfos6PXlkQyCCUXVnxmvEK0MqZTvu8GcbOaGsEHvUKWQu.KVCrgIjU3gpx 0fuDI537hbATRS23e1ghZY3gXPFuCZbTs7laUzi6xzt32h7DU2D_Nt245VwyMNmIk5bBbJ9P_a4C bw3ym4eAFgmXH_kOiG5y4C7fD6ZGiNJ42XYCbptjYvB6sipoidAI2GsFKlT67mZS_2uJNC9Xq7HO nEiooygvncgRoU6IxcYJ8Fi.6eIEd9Mo30TZN82XtvPDQy6XNc3dKylmfPoa9Wj4q7wbHKjqWABb AVtnVv5D97WQ_UVfVs2fM90_mMvJTTtr4ioUzPYaMj86q66ekzeBWuHZ2mztkmH2P75Y82mQ.Frh XtXbvJghiYnRtSvdD9GEC.eIzbJIKE7nGhWPEn.WQFvQ3FxM9nLWLEphxC.6YFb7eHEFwwxYbQE2 huj76vQOp2i3k0y2dlvWbQly.YkP6bxkJUoYRFnolt7U3_Om9j7EWDP6gps3hv5ke_BE29ush9AI r8YNnlRh7l4uDKvsPDSbQ5FngvOvCHr8q.JVZucWN5dUoYWalP2t05yA4FLye_Xg5txfajLie3Mh NlqEFtA2epS6qzxgaoMEPXUbS1Ed_3MgdQampwPZr48Um5Cu9I1SpFRzn4Mo188WdhhwI4tk_LOw uFOBWpcHhHq8TUKAQO03blmVXPQqxOQ5yw4EyG_6nzXiLIdaBALkyncn6rVkG7N74Vrdhqx0vRa3 ZXcyXPlX6.LBUy7dOP2AgrSGMm5BAHu9qenn1drYgfDCM.74MfHhj2haqjYo3nCQoy.q_aq2p2Y0 Prw-- X-Sonic-MF: X-Sonic-ID: 0e65ab72-84eb-43d8-99b1-f5e55360bcac Received: from sonic.gate.mail.ne1.yahoo.com by sonic315.consmr.mail.gq1.yahoo.com with HTTP; Mon, 28 Aug 2023 15:43:20 +0000 Received: by hermes--production-ne1-7b767b77cc-msspp (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID df76bb844275db3a0b32046750ce2ae3; Mon, 28 Aug 2023 15:43:14 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux Message-Id: <948CAEBD-EB60-46B9-96EE-FE41CA6C64A1@yahoo.com> Date: Mon, 28 Aug 2023 08:43:03 -0700 To: weh@microsoft.com, FreeBSD Hackers X-Mailer: Apple Mail (2.3731.700.6) References: <948CAEBD-EB60-46B9-96EE-FE41CA6C64A1.ref@yahoo.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.991]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.30:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.65.30:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RZFFk4mQzz4fd0 Wei Hu wrote on Date: Mon, 28 Aug 2023 07:32:35 UTC : > When I was testing a new NIC, I found the single stream scp = performance was almost 8 time slower than Linux on the RX side. = Initially I thought it might be something with the NIC. But when I = switched to sending the file on localhost, the numbers stay the same.=20 >=20 > Here I was sending a 2GB file from sender to receiver using scp. = FreeBSD is a recent NON-DEBUG build from CURRENT. The Ubuntu Linux = kernel is 6.2.0. Both run in HyperV VMs on the same type of hardware. = The FreeBSD VM has 16 vcpus, while Ubuntu VM has 4 vcpu. >=20 > Sender Receiver throughput > Linux FreeBSD 70 MB/s > Linux Linux 550 MB/s > FreeBSD FreeBSD 70 MB/s > FreeBSD Linux 350 MB/s > FreeBSD localhost 70 MB/s > Linux localhost 550 MB/s >=20 > =46rom theses test, it seems I can rule out the issue on NIC and its = driver. Looks the FreeBSD kernel network stack is much slower than Linux = on single stream TCP, or there are some problem with scp? >=20 > I also tried turning on following kernel parameters on FreeBSD kernel. = But it makes no difference, neither do the other tcp cc algorithms such = as htcp and newreno. >=20 > net.inet.tcp.soreceive_stream=3D"1" > net.isr.maxthreads=3D"-1" > net.isr.bindthreads=3D"1" >=20 > net.inet.ip.intr_queue_maxlen=3D2048 > net.inet.tcp.recvbuf_max=3D16777216 > net.inet.tcp.recvspace=3D419430 > net.inet.tcp.sendbuf_max=3D16777216 > net.inet.tcp.sendspace=3D209715 > kern.ipc.maxsockbuf=3D16777216 >=20 > Any ideas? You do not give explicit commands to try. Nor do you specify your hardware context that is involved, just that HyperV is involved. So, on a HoneyComb (16 cortex-A72's) with Optane boot media in its PCIe slot I, no HyperV or VM involved, tried: # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 120.2MB/s 00:42 It is not a high performance system. 64 GiBytes of RAM. So instead trying a ThreadRipper 1950X that also has Optane in a CPIe slot for its boot media, no HyperV or VM involved, # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 299.7MB/s 00:17 (These systems do not run with any tmpfs areas, not even /tmp . So I'm not providing that kind of example, at least for now.) 128 GiBytes of RAM. Both systems are ZFS based but with a simple single partition. (Used for bectl BE not for other types of reasons to use ZFS. I could boot UFS variants of the boot media and test that kind of context.) So both show between your FreeBSD figure and the Linux figure. I've no means of checking how reasonable the figures are relative to your test context. I just know the results are better than you report for localhost use. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Mon Aug 28 16:15:59 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZFzl43C6z4rj6w for ; Mon, 28 Aug 2023 16:16:19 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-21.consmr.mail.gq1.yahoo.com (sonic310-21.consmr.mail.gq1.yahoo.com [98.137.69.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZFzk08Ysz3Fl6 for ; Mon, 28 Aug 2023 16:16:17 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="FbdJUr/9"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693239376; bh=FoGrBo8oyvguUafMwQkKKBOmSfxpL5XpstnN0H5sYeo=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=FbdJUr/9K1vFP+hwVbQnohKY2/LTtSbZqtzXxiijXAQekEPlRDk/ajW9Fu3SBBitSJ9w3PCsuCiKD9OMhQoC098jVRIR0cgJrd6BJD7ER14PnDndZQf/wP5XHHwhdczWccSTQZ8BBAYKJdklYSD70+sYWYm2dY8LMZlPjzcJ+ck/uhd+Tj3vjmu25P1s6RaW0N0IluZXUIMQ4TXflJKZaFWxa4WzfrFPNOHMShTzZivwjAAEJBA4e/9/xKlDYskbSM42BKrtHRDMZCbQiVLidc7yJpIvxwVSVHpsHXAxSPkOS37j4oMoLEEut1QodzST86HwLrKhEQtzqlsS1LQG5Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693239376; bh=P+csdyG70ToJfSH1CdCz3GCRgcXotIEU2nY0BgD9R6M=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=nWICjwev8rbKRaO0Ac+6V1jpgx2NYY9YEGWfTwaFmuFRNEsJesSzb0AcPUl9FGYJPig0f2lbjvQHUNGLv9UAfJKmJhmpggMmeYrug38b7CyhmOu0LkQlbnRXY6XURB/A4BcDb+bjNdQs+IuUomsAFjjzlNeiKbKKbG+vMgbuUOT1TXVyQgOSEa9wgg5ZOvqkBiYRXWUbkX5EYB32RscLZHkvwZ2TxWfMHiGasKVXZNI8wPhdzNgDnbVggcMD8z/xWhpFnirsm6oaw/QoDJykTyc5jfC/3jabCkL54UbH7YrCncxXDi2WLM1diY6e9kwwKbEN7rXKuVsdpYw9HRIHRA== X-YMail-OSG: _NowpoMVM1lPxWDPvZaEky_VGkwK3M3eSUJzEiV3mtnj7BrM.a4tpqip2rBjuOU 1JxbEFc3ztb6eTdFiUrvkkc2._k8CmkGiaKgQs4NNAwd4pmZ1Wt7.GKRyLvJKqRZJmmHxkkaG.2A n53fVctIXyLNLjk_NVMKlna.zMKT6XoDYtT6yUNl6AvawjCQpjgq4AEVC09U03Uu2GY06h6W6Fub 7Jz9y2FMV63gjPiyfwzHuaobMx80QCsTFB.RkPbSrcZ64bxARiR9SiX0.ddZv9Q.o6ohnOxQo8H. INv7FtVwhX9DArxyYgr0ilT43ISv8Zfrl_tQemyKHTeOm9wh2NQ1LK7z4iQ9B7nCyFzH9fAcjSrY 22I0BxJXsyt3R_XuTkK6Y5fjD5_Vgas3.UREqboTVCB_.HpQkgl2XdXZBjrFw6hFhxVlLWnjAyYw gbHYHl3CMR.dM9DxYvK278QFuY5dSzrll8p9pUSjSJ0uHLEq.yz9qwHG3N_KllxxAxUk5vEweKl4 IKFioWS2tTqjrwsF0MAIhY5nMtZvgD49CvcEXDtkriKR.UE5KSK3URt8ia1luuD0D.hWHjPhHxVJ S3NOMqW9zaFeER14xIqs64Km5uDHb5qpc490T6Nm3BFEVe_nbKe2b6.M7YFFPtpUeXnSjWte5DhM ojOTZduPzY1KtFnliCZsBgN8mfUn.x7EHCvNposgeydo.xn_loYVmjX0XZXdNe1ijJJBakvSEG7w lljk3P4T58h.jwAbGIPwi9T246UXF9nQzEByu.gxHzk6mrA9jQ77eJWeATtRpTs3W2l12e15Rt4p 42aA9Bb51e84E2zObjbWMvb7NSNfFzg.fWTsXsNo1_vj18O3WoiG78XPdc2dKLwB.UyCq0rd0Ps5 HEFZiZSfCLUuxUaIi9Q0iBz0Ma3rlrWjs3rfe4lmFPjvO.w43cd6lr.pyYPuNzdObIfywXQlG8fq XOiEbaJVETNyG10BfTpcEGVcf3.7udWzbAV6uLOHfD2Qj4.krJ9.7uOk7EEqSM9enY6.xroBGJbO c5_NnJCYjpucZu9bKrRDC8GrivttLJRE0EfNbeYX1WTWJnkI201sSiHa4qSFAVh_WYRp.VdEbGNr FXspYB1bTBX3OORqLSxlRLin_WpiA8YYTESzBP99aD5JdnIhzGW1azhpkFXJcH9biml_7Z1TxIce PKmF302Vh2hIyuEg0uLmj1GV3R56SKw6W56FG7ayoj6WBQtTlJOa5f8LjDoQ2DXYwo58KCnduR90 rAYinThmp8wbqGSAkIsi2Z.eueT8LTYQbte0BpvO9XJXP.mAR2aBpAsVrrzvCOyjFWJWv.5Hoagx QVS.LGXfM8qWqI.fcMhGqxgl2OHo63dT5xQoCuOLZO0S4ZvYdtwsrN7U5M8e2ZfOZ4pO93gYDSaS MSQ66ynDrnNLav7jFrzVRIFD8XJyuW_IpWh_DyFwjP19zJrfFUxIeX3oRDiYtNTaQQ.vklxj7vJr 10_xajJN_A5uiZ15xEn5nt30H4RfJTJRpsMBhPdb4rc0PYkGYqElyFoj98UnSwOOvK1mw4ED67Vi 0Hg6P1tDEq69DDa7uhqfsqoqZpJugMX4AEt0HQ1Nnhp6c6tU7VIoiFkogZMTHEk4_SyHx_QmIyzc 1Fukdfy3asLtu2njiHH7_s0zcX5WwRCMJmedwuWw7MrtbEoeCynwh1U3Vk7zaT.itHjJyc3z4DUP ZQ_rjAsLRlYqrJDiDttP1CYsQDziJy_NGmq3c3atU5E4oQENSYIp4wdt7XmY1CuR72yYN6b4.kSh fZs59ujRh6qml.0sMwMRVj2XLJ.vJbS.XnPvv0OueO.TVUCXg526pkxx3WTQ56e9a8bW40sLxF4R DAkVMcP.FJ_vvkDdpRqED9E7jwLgcYtIvS.VlpI6Naap_dxuGqpGwdzyEbD4pM7uo2Pf0XRrBlQQ 26uwHxYid78sj8JCE.8DlI40BKXEwmQ.auUQJZdZcP6xtFeHofwyA8SGkyJzEUo2FQOfeu8ZABg_ Xfas_tBc4G_vwLsdleoKLJmoVidlwaRq68yWTIcEZdF5caX8TKPO_YvCKdjqssUMZAqTJF0KDvTj Yg1tRJTehnDF7huGuLKAJdH7iDcxEdDJ3lKRIokrC4yDHeHfM.WmP9zGw.c1yC8lqWXgogikQFIz IoJ1QoysaAptUEJQg2_7tsiQ6lGJd8JhJk6hsS7Z11_WKyyttArR_mrpKgFqkM2JB6Pm_PCJqwJm 5NYNlbh9G.Yh1BnvpdEIsRHnC9xouGERQOQk2jmAfRal2hV3OJWK1E6Qjdv4GE6pUmXnz819HvlH OsQ-- X-Sonic-MF: X-Sonic-ID: ab00da86-4758-4ee9-a994-18a53a7cb8ff Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Mon, 28 Aug 2023 16:16:16 +0000 Received: by hermes--production-ne1-7b767b77cc-27nt8 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 1338cfc330c34f563f69ce6a8193088b; Mon, 28 Aug 2023 16:16:11 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux Date: Mon, 28 Aug 2023 09:15:59 -0700 References: <948CAEBD-EB60-46B9-96EE-FE41CA6C64A1@yahoo.com> To: weh@microsoft.com, FreeBSD Hackers In-Reply-To: <948CAEBD-EB60-46B9-96EE-FE41CA6C64A1@yahoo.com> Message-Id: <07C2C9E3-7317-43AF-A60C-393ADF90079D@yahoo.com> X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.992]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.147:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.147:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RZFzk08Ysz3Fl6 On Aug 28, 2023, at 08:43, Mark Millard wrote: > Wei Hu wrote on > Date: Mon, 28 Aug 2023 07:32:35 UTC : >=20 >> When I was testing a new NIC, I found the single stream scp = performance was almost 8 time slower than Linux on the RX side. = Initially I thought it might be something with the NIC. But when I = switched to sending the file on localhost, the numbers stay the same.=20 >>=20 >> Here I was sending a 2GB file from sender to receiver using scp. = FreeBSD is a recent NON-DEBUG build from CURRENT. The Ubuntu Linux = kernel is 6.2.0. Both run in HyperV VMs on the same type of hardware. = The FreeBSD VM has 16 vcpus, while Ubuntu VM has 4 vcpu. >>=20 >> Sender Receiver throughput >> Linux FreeBSD 70 MB/s >> Linux Linux 550 MB/s >> FreeBSD FreeBSD 70 MB/s >> FreeBSD Linux 350 MB/s >> FreeBSD localhost 70 MB/s >> Linux localhost 550 MB/s >>=20 >> =46rom theses test, it seems I can rule out the issue on NIC and its = driver. Looks the FreeBSD kernel network stack is much slower than Linux = on single stream TCP, or there are some problem with scp? >>=20 >> I also tried turning on following kernel parameters on FreeBSD = kernel. But it makes no difference, neither do the other tcp cc = algorithms such as htcp and newreno. >>=20 >> net.inet.tcp.soreceive_stream=3D"1" >> net.isr.maxthreads=3D"-1" >> net.isr.bindthreads=3D"1" >>=20 >> net.inet.ip.intr_queue_maxlen=3D2048 >> net.inet.tcp.recvbuf_max=3D16777216 >> net.inet.tcp.recvspace=3D419430 >> net.inet.tcp.sendbuf_max=3D16777216 >> net.inet.tcp.sendspace=3D209715 >> kern.ipc.maxsockbuf=3D16777216 >>=20 >> Any ideas? >=20 >=20 > You do not give explicit commands to try. Nor do you specify your > hardware context that is involved, just that HyperV is involved. >=20 > So, on a HoneyComb (16 cortex-A72's) with Optane boot media in > its PCIe slot I, no HyperV or VM involved, tried: I should have listed the non-debug build in use: # uname -apKU FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 = #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT 2023 = root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 120.2MB/s 00:42 >=20 > It is not a high performance system. 64 GiBytes of RAM. >=20 > So instead trying a ThreadRipper 1950X that also has Optane in a > CPIe slot for its boot media, no HyperV or VM involved, I should have listed the non-debug build in use: # uname -apKU FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 #116 = main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:20 PDT 2023 = root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main-src/amd64.a= md64/sys/GENERIC-NODBG amd64 amd64 1500000 1500000 (Same source tree content.) > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 299.7MB/s 00:17 >=20 > (These systems do not run with any tmpfs areas, not even /tmp . So > I'm not providing that kind of example, at least for now.) >=20 > 128 GiBytes of RAM. >=20 > Both systems are ZFS based but with a simple single partition. > (Used for bectl BE not for other types of reasons to use ZFS. > I could boot UFS variants of the boot media and test that > kind of context.) >=20 > So both show between your FreeBSD figure and the Linux figure. > I've no means of checking how reasonable the figures are relative > to your test context. I just know the results are better than > you report for localhost use. Adding a Windows Dev Kit 2023 booting via USB3 (but via a U.2 adapter to Optane media), again ZFS, again no VM involved: # uname -apKU FreeBSD CA78C-WDK23-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 = 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT = 2023 = root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/a= rm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000 # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 168.7MB/s 00:30 Note: the cortex-a72 and cortex-a78c/x1c builds were optimized via -mcpu=3D use. The ThreadRipper build was not. Note: I've not controlled for if the reads of the input *.img data were gotten from memory caching of prior activity or not. I could do so if you want: reboot before scp command. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Tue Aug 29 06:06:29 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZcPt4tSgz4rPt4 for ; Tue, 29 Aug 2023 06:06:42 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from HK2P15301CU002.outbound.protection.outlook.com (mail-eastasiaazlp170100001.outbound.protection.outlook.com [IPv6:2a01:111:f403:c400::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZcPt1Ncyz3fFL for ; Tue, 29 Aug 2023 06:06:42 +0000 (UTC) (envelope-from weh@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GC8FnR/6AQoV93PXSSyVWJK6Z6oUKJTO75rBGM2x3vVFtXnXX7Nu2ayXGBGf4qFur2/DjJJEVib1hXdS1If3fhPKeOUgr60sX81z6OdQ8KuCdz8sft67ZP19Ar4738vUk7ZOLJT+r0+Wxce9E9q7Ky8rUmB7uK/GEeBGRow/DkXNSfAVxn3eGUHiiyj0iXqmTQQJIGMm3/UgJ/dWj3oTUFXkYrJbAggKbOX4YILX+TlJP4Ol3W0PV87b/thROrxjWTCQ1eKHLg+wSUCJ0+uUy6PXLyqcdpTE5LhYUKLCdOLDX1O++NdCxg+7gu9tvp5Mald7o6oPrJvBvqdWsxypug== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=Xl++lITd6kIxqzcbZuT1M1u7XZzkS0tISBkYVd0uXNw=; b=A1am2o5gkEEF5LAN4o21lJDuuLaZjSyIR2Y8vXysbXm3IzUin1YHbtx7aGiNriF6XzRX5MGSwDcVZ6kYK7F0nt5qPY3oEfEHWgAy1guO/M91c3XZnBskoz1ELv+bXDAScsnhe3fo7GK7NVHrHhH2FN7ln9wo+gdwqhrzm4u9+n1UxelUl9MPHZ6iJIPP9akXfLhPBJnXfReHvXeIiFDdQ37mO23HtxtGZpr6saKzlxlvhqDapk6cH0VbhjkrVkw/xFLM9jUHHrUoIjYGxnKzoRJDv8bU5u7UUgHgy/BHS2woLCePHYu+dcOZwCPDzJcSgCnpz4sBNwhMzNHARFxJkQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Xl++lITd6kIxqzcbZuT1M1u7XZzkS0tISBkYVd0uXNw=; b=MxErS127syrE0w7SQ3DvHeigSmiH+4FJv99LSuXnlr9L/FMwEwOEQUfycBoAAt6PqL4fwbZbcGW5cSzzZ7yjyUJ7EDMvv4+LmkyJK2dQBmbTihWlCz52B25U5hOIT9/x3paE0tFF1hWo9bjgXeDptug8DjQGISxNa2bKwqusBVo= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by SI2P153MB0460.APCP153.PROD.OUTLOOK.COM (2603:1096:4:128::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.8; Tue, 29 Aug 2023 06:06:33 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83%7]) with mapi id 15.20.6768.007; Tue, 29 Aug 2023 06:06:30 +0000 From: Wei Hu To: Guido Falsi , Mikhail Zakharov , "freebsd-hackers@freebsd.org" Subject: RE: Very slow scp performance comparing to Linux Thread-Topic: Very slow scp performance comparing to Linux Thread-Index: AdnZfvJbaS4mlPy2QcSWspi9tSFg8QACqxQAAAM0yTAAAbEAAAAmeeng Date: Tue, 29 Aug 2023 06:06:29 +0000 Message-ID: References: <6952b7fd-f90f-2677-1997-d0c708cb559e@yahoo.com> <455dc69a-dd15-18d9-8e93-91bbff3997e4@madpilot.net> In-Reply-To: <455dc69a-dd15-18d9-8e93-91bbff3997e4@madpilot.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=1a1f8564-35d3-4bc3-82c2-e2bfe40321c7;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-08-29T05:10:23Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2P153MB0441:EE_|SI2P153MB0460:EE_ x-ms-office365-filtering-correlation-id: 6d99479d-fa4e-4cc2-8133-08dba8561620 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: R+XGzwrn9d37lGMkyTsE3o7h4/hckvVTjRODXSoX36LpzpejnuxnA58srKWrEquLDpueTfrYfhCMvXYRBCMDDtl3NiG+NpTTT+Jehyc3T9+VHAH60KuDlRJOHwSjXFILkXb7AGL2+ty2g1usI9I76kkeT2BYTNX+yPXlQvpSqOMFEVAXNF+r2wNsozF7D1SpRv0wGk2w6t2U2UM8mGoQraHKARPE1VJ9v6V4CJaiEO657cBB7u744LVtViXaSzb4REaFcBH10I5mV31EyJn8M0o9R6f+AaZ19NEq2WbgtAW3U9tvD1XSPTrQA/mjNtyOGXPikQ2TuJDj1Zhg5oi5Eky0/yTMfRDxWh51rzNYtAOk89PNSuBnynbQvtDsItXkTV8H1llAELuimde6JnbGci0tKDDb5IQcRdcwCdtFqkpsYAZnHIcuW6ViSXmdJzMPl6FdXVyRhGTDvcTrffokW3tEzIqj3RvZ9q/NgYkbNFi7RCVajspl9r2p9Vl8XTIxZSIvTdyqe8AVVAhwd/BC06JRWjbqQGKb4sZJfe57QL5HL9tpXW9UZGxxKqP5saNDvCUWWuw728epxFBd1cN64c5HSLC+xUuNgLnmTgo6nVjigNDTdQU1Ky/Grv8PTO07I/0u2zOWNpQWtybIPhOb0l4hVSJvIPghjggHgGUxrl1iia6cUWWqCuJVQBLfD0y6 x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(376002)(396003)(136003)(366004)(346002)(451199024)(186009)(1800799009)(9686003)(7696005)(6506007)(71200400001)(53546011)(10290500003)(83380400001)(76116006)(4744005)(26005)(2906002)(478600001)(64756008)(316002)(8990500004)(66446008)(66476007)(8936002)(66946007)(41300700001)(66556008)(110136005)(52536014)(5660300002)(8676002)(33656002)(82950400001)(122000001)(55016003)(82960400001)(38100700002)(38070700005)(86362001)(12101799020);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?RUwzYVgrOXAwdTJxdTJoNGxldGdPeE5pQWNqekJWdStaR1lxd2dRWHhnbWFC?= =?utf-8?B?YU8xL0dTYU1tVEJ0cnE4RlIycmpDYTkxWHJIZzZUZWdiYUNJWWhyeE9Wb2FM?= =?utf-8?B?bG5qTU5zOHdTbXczQk1mVGpIaHFBN0paNHVoUXNTZVk2TGhsMUYxaDVTbFlI?= =?utf-8?B?dDlNMURJTzdsa01QZ3ZvdFkwVTdwWHo1dnIzclZrVitubS9qbjZObDVBa3Vp?= =?utf-8?B?eEk4Nit6N3h0bWhJMThCZlRvS2V4cnFxZGx5aGNKRXhCNy93VmduMEJzOXZW?= =?utf-8?B?NStMbEV6WE45UEo4TlJTUit3Ty9yMWlkRkI0aHhITGpvajFHN0xHa1FRNHNy?= =?utf-8?B?ZXNzVzl4WEltZFh1NndueXl1UjBMTFZLdDF1MHNhVk0yalBzLzRYY29NWEUr?= =?utf-8?B?dFAzNjA4cFpBS015QWlwS0h1VFZHU2hseG1BeWdLT2crYkxKYk12UjIxa05p?= =?utf-8?B?M21sMW83aDI5dDJhTEc2RWt4VTRvNlZHZ1ZRK1luY1JBcExEMjhxdzE5SUhW?= =?utf-8?B?NHBnTUQ3MG5sUUtiRDJUYWtOVmVOWWJ3ZTBPUEo5eFJPNWtqSjFUcmE3c09V?= =?utf-8?B?Y1Irb1JLTmpJSjFBS2gycldUL3p3MEFNdkh2ZnV5NXdXU0E1ajk0L2UrcC9a?= =?utf-8?B?RkhsT1NRcGdabCtmZHU0aFNzT29ELzhib0FEVFE0K0phemlUVUJaalFqektM?= =?utf-8?B?N1huTGo0L0ZFMStpdW5pb0NSbnFCTWsxaG1tM2lHUWFYT1JMNk9yc1dDQVFw?= =?utf-8?B?UkFqN3pPMm1JQW5yVno1THphcGhrbklDeWdhK3dnWjNsUGFTVkNGWlk1N2FI?= =?utf-8?B?Z3k0S3IwNmtSQjYwcVpZVlF0M3JOWUs3M0VzenBxQWREY1RmOVpCb3BHOEN1?= =?utf-8?B?aTdTbkZGMHZXa0p1RGRwTC9HelN6enVwOXFwT0dZRmJIVEpLSjFuTWRUemdS?= =?utf-8?B?WGpmRXNIUVYvSkkzN0tjZzFvbFg0aE1jb2dlTlZpbmJIejFVK0NHTXNTdm4y?= =?utf-8?B?YXROTXA1MC9wRlJrQndlK1QyNTRGTlRLMU1mMmQ2VnBxOVluNnMxa2N4RGZL?= =?utf-8?B?OXN6N3E0ancvNG9NL3R3MVRtbWxDd1FLVnVlNkxhQTRjWXRKL3M4eEJ4bldG?= =?utf-8?B?TWdjR0xENHUvRVBlejRBZ2hQb0h2NTdoejdybzd3anZPV25ObjFJM2M4cHYy?= =?utf-8?B?c3AwUS91U3dEaXdnOUJNRmtmZ0lhNUNLaFdRY1M3cUZob2k2TlE2L09DTTJF?= =?utf-8?B?VlczakdzZFo1ZDJlZVFDQzFZN3h3Q2NOMXFiT2o4cERkN0NpbFR1dXY5Uyt4?= =?utf-8?B?c1hUVWxreUpoK0F5YldUYXNoSWRTRmVDZFBqcUk5ekJJRlN1UnZYMUk4U0J6?= =?utf-8?B?bll4R3hvZkszdHN0Z1I4VDB1YkVUSzhHNVZvQkJKMXlTbUx3ZkV3NzM1VGUy?= =?utf-8?B?ODVzMW1zK0ZNQ3dZNVllb1UxQWMwL3JBbjlzT1o0Vk8rOGhRVkpzSWdKT0tP?= =?utf-8?B?OEQyZEJHQjdrVjUvUWlyMGVtK1ZkZk1XeHdmSHlXaWEwZ1dIUTdIdFd6cXFU?= =?utf-8?B?UnJvR0svMW5pYVVaWnlDUHdZWnNsUUhEUzcrT0g1R09qdkZhcmwzRkt2RWht?= =?utf-8?B?SGZHRnRmNjVadzhqeGthV2FhT3htOXFnZExVQ3orcnMyd0JaZ3ZjZkpjK0F2?= =?utf-8?B?NGZDTnU4dFR2SzV5V3l4REJUM3c1UXZVUEIvNG9HUHRIUlY0S2p0ai9DL052?= =?utf-8?B?TFNlQWtBNEdQbzBkNURoc1VHMXFOMmJNejFMU3RUYUNXOEZKUURmYVgwMWZm?= =?utf-8?B?STRlaEhwQ21qR1BleU9KS1VmS0hiSzZ6ckt5Nk1RSHBNZlFLS1p2NkxldklG?= =?utf-8?B?UzdjM01xSWhoQmZHYTBNY3BIMklsK1FScUhtSzZ3YzMyYVRRU1dWQ0ZJaUJn?= =?utf-8?B?bHdEVW9ZcjluT3pLbHk1WC93N040TUJVVVlTNVhiL1doVy9WR3k5ejRSNm15?= =?utf-8?B?OGlRbitBUFJGbWh5MUZISmI2Vi8xODdiUUgvWnBGZ3VCZkpVOGdyemllWEFs?= =?utf-8?B?eTBRSkxlZGFQYmlRTTdDdEpmMHJWTEp3UDNSdHJqK3lia2hCbFV2ejczUkRG?= =?utf-8?Q?7WPE=3D?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 6d99479d-fa4e-4cc2-8133-08dba8561620 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Aug 2023 06:06:29.4672 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: A0IbptRrCXO7n8FgS8EQwDfm5+oSq/sWonaVvLMxWdgiHSXeCZydcyc1x2a1lS4iVs4l3hm3JZJlY4HIixbfEg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: SI2P153MB0460 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US] X-Rspamd-Queue-Id: 4RZcPt1Ncyz3fFL DQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogR3VpZG8gRmFsc2kgPG1h ZEBtYWRwaWxvdC5uZXQ+DQo+IFNlbnQ6IE1vbmRheSwgQXVndXN0IDI4LCAyMDIzIDY6NDkgUE0N Cj4gVG86IFdlaSBIdSA8d2VoQG1pY3Jvc29mdC5jb20+OyBNaWtoYWlsIFpha2hhcm92DQo+IDx6 bWV5MjAwMDBAeWFob28uY29tPjsgZnJlZWJzZC1oYWNrZXJzQGZyZWVic2Qub3JnDQo+IFN1Ympl Y3Q6IFJlOiBWZXJ5IHNsb3cgc2NwIHBlcmZvcm1hbmNlIGNvbXBhcmluZyB0byBMaW51eA0KPiAN Cj4gT24gMjgvMDgvMjMgMTI6MDMsIFdlaSBIdSB3cm90ZToNCj4gPiBIaSBNaWtlLA0KPiA+DQo+ ID4gQWxsIFZNcyBhcmUgd2l0aCBzYW1lIHR5cGUgYW5kIHRoZXkgYXJlIGluIEF6dXJlLiBJZiB0 aGUgY29weSBvbiBMaW51eCBpcw0KPiBiZWluZyBjYWNoZWQgb24gdGhlIFJYIHNpZGUsIHNvIGlz IEZyZWVCU0Q/DQo+ID4NCj4gDQo+IFlvdSBjb3VsZCBwZXJmb3JtIHRoZSB0ZXN0IHVzaW5nIGZp bGVzIGluIG1lbW9yeSBmaWxlc3lzdGVtcyAodG1wZnMgb3IgdGhlDQo+IGxpa2UpLiBUaGlzIHdv dWxkIGZhY3RvciBvdXQgZGlzayBwZXJmb3JtYW5jZSB3aGF0ZXZlciB0aGUgYmFja2VuZC4NCj4g DQpUaGFua3MsIEd1aWRvLiBJIHRyaWVkIHRtcGZzIG9uIHRoZSBSWCBzaWRlIHdpdGggYm90aCBG cmVlQlNEIGFuZCBMaW51eCBWTXMuIA0KVGhlIHRocm91Z2hwdXQgb24gRnJlZUJTRCB3ZW50IHVw IHNpZ25pZmljYW50bHkgZnJvbSA1MCBNQi9zIHRvIDYzMCBNQi9zIHdpdGggDQpOSUMgaW50ZXJm YWNlLiBMaW51eCB3ZW50IHVwIG1vZGVzdGx5IGZvcm0gNTUwIE1CL3MgdG8gNjYwIE1CL3MuIA0K DQpTbywgbG9va3MgaW4gdGhlIG5vbi10bXBmcyBjYXNlLCB0aGUgTGludXggZXh0NCBmaWxlc3lz dGVtIGRvZXMgY2FjaGUgbGFyZ2UgYW1vdW50cw0Kb2YgZGF0YSBpbiBtZW1vcnksIG11Y2ggbGFy Z2VyIHRoYW4gRnJlZUJTRCB1ZnMuDQoNCk1hbnkgdGhhbmtzLCBNaWtlIGZvciBicmluZ2luZyB0 aGlzIHVwIGFuZCBHdWlkbywgZm9yIHRoZSBzdWdnZXN0aW9uIHRvIHRyeSB0bXBmcy4NCg0KV2Vp DQoNCg0K From nobody Tue Aug 29 07:07:39 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZdmN4CQ8z4rS2S for ; Tue, 29 Aug 2023 07:07:48 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from APC01-TYZ-obe.outbound.protection.outlook.com (mail-tyzapc01on20712.outbound.protection.outlook.com [IPv6:2a01:111:f403:704b::712]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZdmN1QnTz4GXN for ; Tue, 29 Aug 2023 07:07:48 +0000 (UTC) (envelope-from weh@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IJaSkIEE/hh2Ad/kv638kUpKam2D1wwKl6GX66T+dhMg+jOtpNNlfh0dUsAGP81UTRBfThHAtvL9PWAaUmfD1Sz6Sl2ulEgvoygQRm5Nu/d74IAdxLyw9NxFduBL83Vgy9kgAZk93mF1Wo4u7m5fX+pU6PtFb+J9P34m6tToTxwhlPlADeEEd+MWyzx8ND1Yy1thgT2bA5vcrS7y56EQx8zeCxOdvZUpDWu0FuUmtqbmqxs6Z/vpGew6OQXJP7fTlgeufrPa7wp7d3EGl0rbCKiiDRXisScUoeeAr1kdZ25moA+AluMGjr0qn0nc0NbE22R0VlPXwf0+PYFwoPJq4w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=H4dFVorlrFwJVqn1AfVgWtwlXrgjA+ezcz+OASsuVis=; b=HkEtsgKNDxGpxl4itJWqIOud0HWXjThba+DFWzqmOd7bPRnkBJkkgeY1Bornc5cjP5l3y0ejw80Zl35Z/0MkgaiPeuv/22OtKYhj3knW6CdwHjqt7PI+ba1a9kx0qrAGZMCBcru3P1fGfVuWURp9mUjnPzUFN8lbiJxbkFHbEkTzVZeNtYWnZwMpGfJjOAs945BNSxaYKcN+/qyaqMRI0hWjKHH5r/p0XgiHv5bFF+dGiZb2s6YkbCvfJ9bXDwXCi4FRuCLe1xvc9pIk3dpy7Ke7Z66N9UfnGS8ZGbrgSetqgbT0CZJvUXTQJrP/bi1Z47BWIvfhW+RiTtXglYFX0A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=H4dFVorlrFwJVqn1AfVgWtwlXrgjA+ezcz+OASsuVis=; b=Kn4+sCziyZWWP9k87F0va7HiBWpnDTMhLCgS41fFke5ji79N0lrtY2uGal4cw/QsjWpB3NfWfiKZfVfMgSXoPyJ6FIw9egp43kfa2QRjZNhzHAHqKXiXapiZI+b9u3zueql03+zj/fE0c2Ix/OoKSz6YqV+tG+jzgvr9yK7wghA= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by TYZP153MB0416.APCP153.PROD.OUTLOOK.COM (2603:1096:400:25::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.8; Tue, 29 Aug 2023 07:07:40 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83%7]) with mapi id 15.20.6768.007; Tue, 29 Aug 2023 07:07:40 +0000 From: Wei Hu To: Mark Millard , FreeBSD Hackers Subject: RE: Very slow scp performance comparing to Linux Thread-Topic: Very slow scp performance comparing to Linux Thread-Index: AQHZ2cZijEFfDgyXnUKWUbsPmO6Saq//4k+AgADxB4A= Date: Tue, 29 Aug 2023 07:07:39 +0000 Message-ID: References: <948CAEBD-EB60-46B9-96EE-FE41CA6C64A1@yahoo.com> <07C2C9E3-7317-43AF-A60C-393ADF90079D@yahoo.com> In-Reply-To: <07C2C9E3-7317-43AF-A60C-393ADF90079D@yahoo.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=4386a806-7a1f-497d-b24c-85ca4c88d627;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-08-29T06:38:38Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2P153MB0441:EE_|TYZP153MB0416:EE_ x-ms-office365-filtering-correlation-id: ae46fe46-d4e6-42ec-448b-08dba85ea1e4 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 1z2EXRkbxCjzJafiFDclWLLaTG+Q8strjcLkseJi0FxUyRdg1nBkdMfgvNfWtFczFxSXJsdDeu+V9Cqvg+3n9X+drAHSe8/1jEzMTvhFClD/7rv8INvOEojM2crg6PJW8u0eHUdWTp6+kTp4hN/Gfw53QWvW1KA0lxgxbH6imj0BVy/kS8npZe6TVR1XUd1mm71LmqEVANIhdUsfFWPHYt9dm9CJp9RgNau1JTvQOvDIGh9k2Gbxi/VsYN01eKcatvk8lVQteUNTUNcjsN1fIyiiiqy9Dk1xk7gC3Me9YNAwWSOgnT1sTuIm9y5XmW/XZPoLibbTWQ/i0r4MD3Yt8Qoyg9KvcVt8oUTFpvod14qGabYV8GCeiJ6x1sxVWdp2kL9DtZn2sH00EpD5iOogmY3A9a2bmO+ccJ+ruLmC6Mhly7nzcEuC66Tdql/oMw6GtRzfJ38L4hgztcIbjX4fbmLDXU/fqLS4kEnBTtNwOpb4ocMx6CTFFi+TRGWGiJjWB/Zaq2j/HI//t0I1kPeXUb8JDXyI0ddTDfy5wGs+RfhEaUjXF1q32gX684yw8I2WaCaDtgV8QcSmFE7IhB9PSr4WhbVdafg9blbRwYDg+miGaXtwyCDKgnuC9OnrYKpykqiY+FirO6pworylBgebAmuFn+TfiB/OEV+XDs8U9v7WPik20AfZUdPpiM1sUxk85ULjQm7k0vjyH1t3M0LujQ== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(376002)(396003)(136003)(366004)(346002)(451199024)(186009)(1800799009)(9686003)(7696005)(6506007)(71200400001)(53546011)(19627235002)(10290500003)(83380400001)(76116006)(26005)(2906002)(478600001)(64756008)(316002)(8990500004)(66446008)(66476007)(8936002)(66946007)(41300700001)(66556008)(110136005)(52536014)(5660300002)(8676002)(33656002)(82950400001)(122000001)(55016003)(82960400001)(38100700002)(38070700005)(86362001)(12101799020);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?RyW2qB9giIXRPEi1UDzm9H/h10jdbYKKTa0ns8eUzF2Ni812yKjT5gGa8Kj5?= =?us-ascii?Q?3T4hVgRj67xZXvGCCQ5YlbEFaho5Kq1XWpsSKk+hCgIcNV+aKIG1gY+UgrZK?= =?us-ascii?Q?CAkCZjYl82GlvnvVCnM+1mDUy7jrMh4X1a/qxta/q+eCPT/ZEIDqhjf7zC3m?= =?us-ascii?Q?VdnzizI/EW0wLakINfELM5d6dTF7NoxVJ9NqHg2i3YmMqqYGR+eymhijptOa?= =?us-ascii?Q?vx8HATUFXj4vlfeHoUJ9QK+IDg4TpOQXbiIn8BNoQys+Fl1lPrMVKuKDNhg1?= =?us-ascii?Q?VNZBQxyDvia2EtAnVVnBBtl8N2F45YhG2jZUYupOSsafmfpO71JPaXHfW7hn?= =?us-ascii?Q?2/mOe9BI7fSrgdOq0//FumadcqzivZRs3jcY0+0cOZvpf0XFq+QgGu6Uqvem?= =?us-ascii?Q?8bbnJ4ZL35SFfZMcI4G1kJRb6LKir/BWf/NSkCxla+FyAOibLPy3oN4rQpKm?= =?us-ascii?Q?MwQxhMKhIVMgRWUtYCx1G6EmvrSVNtKaP2NWtC+IhPpc3e1Lhe7zMl3jFjpw?= =?us-ascii?Q?PIrX1jK9Veekek6PMHyD/9LCjS5bsAqOZVbNo97eDznOwVRD9vmsAweT8N6U?= =?us-ascii?Q?72ZHLSmtz6VfLz+nrSauxIQ7G9XCb5p5xJmEp9+eQW91CsGcMaoKXDo1udPJ?= =?us-ascii?Q?4wdjcp1jkUoR/QpkjYtZw4mJbKW3tQDa/s3h85n30PWifFnFWi1wWS+TCO7N?= =?us-ascii?Q?2dn+pB54acdxXuRppTT2E8d2Vhj8jqwf9kT6sijEVfhMiyxvYtt4igSN7Yx7?= =?us-ascii?Q?UU9SG9PTiOCULr3fJ1K3Kn4dNd7y0y9pIf2+TpwborBP3v4tTE/YCOsxj9Nu?= =?us-ascii?Q?/nNrsZ4WRL3FhTNBFsk9gtRbRZxsO43+HtWSljG4RAJjH+1ab60gNbM+t9bd?= =?us-ascii?Q?omZYFQ430DFFmfVff63oxQQBHji0/u97VR57jhppfEuvtrQ+Q65U1dwg5R8V?= =?us-ascii?Q?VRytf0yQy/QRO5vwq/UtqkprISyiPZhr1dnWd2tPhaWjT5HKqpK4sPwzV/TI?= =?us-ascii?Q?Uncl/H6xvGdGUvjHfusXWfTyPAe8++jKRsOjZEQN3nI4cpxL8Gs8ByBSeQoG?= =?us-ascii?Q?oeOAryQzlMU7JIPqrqKJruitTrCccHHC36pBDES7340mO8FpH+jHaxbF8be8?= =?us-ascii?Q?y7SQksExN/XULypqxT/Hd//DVotIkbHxoehIKL6OcwndCdWu5MrRi3EbQ89x?= =?us-ascii?Q?6+0PQakgH7sbWr4oMESgC101rAcltXwavw2ELhH4Ra41N5oLa7f7E/63jGuJ?= =?us-ascii?Q?78APtsM9yOghwllvmembrnq7RCcV3jyUNUH47UpIZfpRlg8FycfGaRQKmdGJ?= =?us-ascii?Q?z49USHgfEG5fDC7NNBjeKARsH2DA1YEzAUmcKbhMkZh6NyF5NXK666RXK6Og?= =?us-ascii?Q?Z+tqDLilD2XI9IYqfcarU2KtOJ6/7ECMU4EN3YNv02xVdFLo2jgKLQmZ+zJM?= =?us-ascii?Q?Q2NrOzCz4f2w2OCh8rwXPeo2Sh8A47m9CPbnacVvqK3S7Cg7NJGGacZWkq8N?= =?us-ascii?Q?Xncaq4iG9EOXUnuPeITsqh4yWE6WE0qtO+PzyPKLyJ+4se5tuxjrLuU4IzHR?= =?us-ascii?Q?iL4M6EGDSDz7cwalceY=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: ae46fe46-d4e6-42ec-448b-08dba85ea1e4 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Aug 2023 07:07:39.9099 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: mOkx/uHPIwdjFOP0gBUgAlKtPqNgnloxwJzuBiWRx01OhMIIBQ+f4leSaLBs1UMLSSr2NDEzLD/q0MgkMbSXsQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYZP153MB0416 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US] X-Rspamd-Queue-Id: 4RZdmN1QnTz4GXN Hi Mark, Sorry for the top posting. But I don't want to make it look too messy. Here= is the Information that I have missed in my original email. All VMs are running on Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-clas= s CPU). FreeBSD VMs are 16 vcpu with 128 GB memory, in non-debug build: 14.0-ALPHA1 FreeBSD 14.0-ALPHA1 amd64 1400094 #7 nodbg-n264692-59e706ffee52= -dirty... /usr/obj/usr/src/main/amd64.amd64/sys/GENERIC-NODEBUG amd64 Ubuntu VMs are 4 vcpu with 32 GB memory, kernel version: 6.2.0-1009-azure #9~22.04.3-Ubuntu SMP Tue Aug 1 20:51:07 UTC 2023 x86_64 = x86_64 x86_64 GNU/Linux I did a couple more tests as suggested by others in this thread. In recap: Scp to localhost, FreeBSD (ufs) vs Ubuntu (ext4): 70 MB/s vs 550 MB/s Scp to localhost, FreeBSD (tmpfs) vs Ubuntu (tmpfs): 630 MB/s vs 660 MB/s Iperf3 single stream to localhost: FreeBSD vs Ubuntu: 30.9 Gb/s vs 48.8 Gb/= s Would these numbers suggest that 1. ext4 caches a lot more than ufs? 2. there is a tcp performance gap in the network stack between FreeBSD and = Ubuntu? Would you also try run scp on ufs on your bare metal arm host? I am curious= to now how different between ufs and zfs. Thanks, Wei > -----Original Message----- > From: Mark Millard > Sent: Tuesday, August 29, 2023 12:16 AM > To: Wei Hu ; FreeBSD Hackers hackers@freebsd.org> > Subject: Re: Very slow scp performance comparing to Linux >=20 > On Aug 28, 2023, at 08:43, Mark Millard wrote: >=20 > > Wei Hu wrote on > > Date: Mon, 28 Aug 2023 07:32:35 UTC : > > > >> When I was testing a new NIC, I found the single stream scp performanc= e > was almost 8 time slower than Linux on the RX side. Initially I thought i= t might > be something with the NIC. But when I switched to sending the file on > localhost, the numbers stay the same. > >> > >> Here I was sending a 2GB file from sender to receiver using scp. FreeB= SD is a > recent NON-DEBUG build from CURRENT. The Ubuntu Linux kernel is 6.2.0. > Both run in HyperV VMs on the same type of hardware. The FreeBSD VM has > 16 vcpus, while Ubuntu VM has 4 vcpu. > >> > >> Sender Receiver throughput > >> Linux FreeBSD 70 MB/s > >> Linux Linux 550 MB/s > >> FreeBSD FreeBSD 70 MB/s > >> FreeBSD Linux 350 MB/s > >> FreeBSD localhost 70 MB/s > >> Linux localhost 550 MB/s > >> > >> From theses test, it seems I can rule out the issue on NIC and its dri= ver. > Looks the FreeBSD kernel network stack is much slower than Linux on singl= e > stream TCP, or there are some problem with scp? > >> > >> I also tried turning on following kernel parameters on FreeBSD kernel.= But it > makes no difference, neither do the other tcp cc algorithms such as htcp = and > newreno. > >> > >> net.inet.tcp.soreceive_stream=3D"1" > >> net.isr.maxthreads=3D"-1" > >> net.isr.bindthreads=3D"1" > >> > >> net.inet.ip.intr_queue_maxlen=3D2048 > >> net.inet.tcp.recvbuf_max=3D16777216 > >> net.inet.tcp.recvspace=3D419430 > >> net.inet.tcp.sendbuf_max=3D16777216 > >> net.inet.tcp.sendspace=3D209715 > >> kern.ipc.maxsockbuf=3D16777216 > >> > >> Any ideas? > > > > > > You do not give explicit commands to try. Nor do you specify your > > hardware context that is involved, just that HyperV is involved. > > > > So, on a HoneyComb (16 cortex-A72's) with Optane boot media in its > > PCIe slot I, no HyperV or VM involved, tried: >=20 > I should have listed the non-debug build in use: >=20 > # uname -apKU > FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 > 1500000 #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT > 2023 root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg- > clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72 arm64 > aarch64 1500000 1500000 >=20 > > # scp > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.i > > mg root@localhost:FreeBSD-14-TEST.img > > . . . > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 120.2MB/s 00:42 > > > > It is not a high performance system. 64 GiBytes of RAM. > > > > So instead trying a ThreadRipper 1950X that also has Optane in a CPIe > > slot for its boot media, no HyperV or VM involved, >=20 > I should have listed the non-debug build in use: >=20 > # uname -apKU > FreeBSD amd64-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT amd64 1500000 > #116 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:20 PDT 2023 > root@amd64-ZFS:/usr/obj/BUILDs/main-amd64-nodbg-clang/usr/main- > src/amd64.amd64/sys/GENERIC-NODBG amd64 amd64 1500000 1500000 >=20 > (Same source tree content.) >=20 > > # scp > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.i > > mg root@localhost:FreeBSD-14-TEST.img > > . . . > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 299.7MB/s 00:17 > > > > (These systems do not run with any tmpfs areas, not even /tmp . So I'm > > not providing that kind of example, at least for now.) > > > > 128 GiBytes of RAM. > > > > Both systems are ZFS based but with a simple single partition. > > (Used for bectl BE not for other types of reasons to use ZFS. > > I could boot UFS variants of the boot media and test that kind of > > context.) > > > > So both show between your FreeBSD figure and the Linux figure. > > I've no means of checking how reasonable the figures are relative to > > your test context. I just know the results are better than you report > > for localhost use. >=20 > Adding a Windows Dev Kit 2023 booting via USB3 (but via a > U.2 adapter to Optane media), again ZFS, again no VM involved: >=20 > # uname -apKU > FreeBSD CA78C-WDK23-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 > 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT > 2023 root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg- > clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 > aarch64 1500000 1500000 >=20 > # scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818- > 77013f29d048-264841.img root@localhost:FreeBSD-14-TEST.img > . . . > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 168.7MB/s 00:30 >=20 >=20 > Note: the cortex-a72 and cortex-a78c/x1c builds were optimized via -mcpu= =3D > use. The ThreadRipper build was not. >=20 >=20 > Note: I've not controlled for if the reads of the input *.img data were g= otten > from memory caching of prior activity or not. I could do so if you want: = reboot > before scp command. >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com From nobody Tue Aug 29 08:28:57 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZgZ42SZsz4rX9L for ; Tue, 29 Aug 2023 08:29:00 +0000 (UTC) (envelope-from mjguzik@gmail.com) Received: from mail-oo1-xc30.google.com (mail-oo1-xc30.google.com [IPv6:2607:f8b0:4864:20::c30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZgZ35lZtz4Shm for ; Tue, 29 Aug 2023 08:28:59 +0000 (UTC) (envelope-from mjguzik@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-oo1-xc30.google.com with SMTP id 006d021491bc7-5738949f62cso1094723eaf.0 for ; Tue, 29 Aug 2023 01:28:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693297738; x=1693902538; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=QOmSrM029mBkkII1KnqOlbK+0gZWcp05A0JMXroBRaE=; b=iNQCCb3MqStptGPVOTqOk6nu+wN3RxBbLT2Sd8rneWYVkm79S1piQ/uTFI5WfBAwoA Nh6Z4sdcmCdcha35wdMs5latp/ZSF8nia2dy6CyJ2OuPp2IL2hbvoDcQmHV/C5a4m2Nk it2Q7O2O+EcHqFBX51Uw7AUJo7SyNV34HGQtFuaf4arclFVWjMvLjpJXDxWujXPklE2u /P4vL4tKZTSt/lb4vf5lZC7B+l5MUBXJYwapZZceHSihCVOqfvdQ9FLKiXO2bBgir9S3 9XVEFCRLm9acA/J47AnH8B4lUtE9GOTao2bR+YY0XJMO123n2b09rjDrk50MGSLoqXRk xzfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693297738; x=1693902538; h=cc:to:subject:message-id:date:from:references:in-reply-to :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=QOmSrM029mBkkII1KnqOlbK+0gZWcp05A0JMXroBRaE=; b=hG3D2HUJxnkTXuH5xYXjHa9ZnkPuAfnUe89cifq5S0voJb1fhsrv4gB1f9PVrsQaVP OJNHl9Ws9sAtbqHaS9jmqclcAcyGnoLXitYH6Y1jetMQmwVB4BZurxfy+spvz11QXLP9 5c6mCgy5gcr4IX8oxcFR3jdYbKxG7V3eiJz997mmX++7dULh/Oo5vnqK6/C2OkEPEYOJ EH27+TVEHeoo0o4G9Vo8eNWrwqkZ++aRk3FAaqPFf+lIze2LpeqcjxbJhgNO4k6/AgMB v7/Qt6KReo9ZHztFti/Jguac9OGTfGHiIX4SmhZCeENU/jfe4sVhmOCY3hdLfDdiC6PU k28g== X-Gm-Message-State: AOJu0YzZ5faNvCjuZqPdyC+pVpMSFzCFBi2Zjom0mUwIKYbQd+X3sklo nYKKOceqA4yaH/dDX9uQt+i3Ei6TzP9GN352exk= X-Google-Smtp-Source: AGHT+IGMo+327KGp+WFGsOVqizeQHf4RoDPuBO+S6Z8yafjPH081lTyuyPhSOzmU3LOrdzJUSqTyEE/XFrWCTsxth3E= X-Received: by 2002:a4a:3014:0:b0:573:55f4:5175 with SMTP id q20-20020a4a3014000000b0057355f45175mr7932883oof.0.1693297738125; Tue, 29 Aug 2023 01:28:58 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Received: by 2002:a8a:1141:0:b0:4f0:1250:dd51 with HTTP; Tue, 29 Aug 2023 01:28:57 -0700 (PDT) In-Reply-To: References: <6952b7fd-f90f-2677-1997-d0c708cb559e@yahoo.com> <455dc69a-dd15-18d9-8e93-91bbff3997e4@madpilot.net> From: Mateusz Guzik Date: Tue, 29 Aug 2023 10:28:57 +0200 Message-ID: Subject: Re: Very slow scp performance comparing to Linux To: Wei Hu Cc: Guido Falsi , Mikhail Zakharov , "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4RZgZ35lZtz4Shm On 8/29/23, Wei Hu wrote: > > >> -----Original Message----- >> From: Guido Falsi >> Sent: Monday, August 28, 2023 6:49 PM >> To: Wei Hu ; Mikhail Zakharov >> ; freebsd-hackers@freebsd.org >> Subject: Re: Very slow scp performance comparing to Linux >> >> On 28/08/23 12:03, Wei Hu wrote: >> > Hi Mike, >> > >> > All VMs are with same type and they are in Azure. If the copy on Linux >> > is >> being cached on the RX side, so is FreeBSD? >> > >> >> You could perform the test using files in memory filesystems (tmpfs or >> the >> like). This would factor out disk performance whatever the backend. >> > Thanks, Guido. I tried tmpfs on the RX side with both FreeBSD and Linux VMs. > > The throughput on FreeBSD went up significantly from 50 MB/s to 630 MB/s > with > NIC interface. Linux went up modestly form 550 MB/s to 660 MB/s. > > So, looks in the non-tmpfs case, the Linux ext4 filesystem does cache large > amounts > of data in memory, much larger than FreeBSD ufs. > > Many thanks, Mike for bringing this up and Guido, for the suggestion to try > tmpfs. > There is probably something funky going on here. Here is a hack which will collect basic profiling info, works in a vm: dtrace -w -n 'profile:::profile-4999 { @[sym(arg0)] = count(); } tick-10s { system("clear"); trunc(@, 40); printa("%40a %@16d\n", @); clear(@); }' so start scp, start dtrace and see what happens -- Mateusz Guzik From nobody Tue Aug 29 10:27:00 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZkBf2jhvz4rdCj for ; Tue, 29 Aug 2023 10:27:22 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-21.consmr.mail.gq1.yahoo.com (sonic301-21.consmr.mail.gq1.yahoo.com [98.137.64.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZkBb0LkHz4d8j for ; Tue, 29 Aug 2023 10:27:18 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=ak5BAemP; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693304837; bh=u8WeiJ96HfIVHyVGHSkXkMvM0LWn18B9TY0Ku5O2lNI=; h=From:Subject:Date:Cc:To:References:From:Subject:Reply-To; b=ak5BAemPPfkmEYcY7sLh8NbqKMx9mMbYq4kVxVgrFzv7RTyt+1+zQ+uap9a1T+Q8k42no0oj+TIcH/3P/HKv7HGb6RSQfPJ5fz/q2lYFVIIftUiGsESfDY7/1/o40ZfANOhNBlFRxjpogjHN+VRuFQ8hwtrFbw+qolWdv7EU2hMAjLL7PB40GVE97vx7e/+Ty3O7SXJm5CMAvc7IvIkEfrjRaUHF/1F9RvU905DOF7N5Me+cJhgCJRWFfYwqnLRv7gfVZP/zqEYyZdbEcuGqpHP26ok1oSJz/UOvisCAPGXjYlwJmXDmTTWNiw6/ZoQKf0rBVahuQTxKBWXtW7krlw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693304837; bh=U8rJLQk6dTToYGeVbTSuDSbanCbh8M4pIpeTvwerElI=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=JsL4amujgrlqPXfBL8gCGD/A+eJ3Haa4Q53LyBoz61s0LY4n7GARdhqv7U5ED1OZwLI/f/+qpG2fu2t7LKBxSbuIl9jf92hnHZae0eR+n5JFAISqL4zeu6+QnWfczToiM3DHcWi4jJgGr6OY4bhXAEBVwXJQrIf6fKO8y61DGBe9JyLFu1YzhKQkXeWbYVS5jvXZa5B5TOJeYGEAwRSr5BO+xQ6yBIUAMYrAc/0PMwOtL9O2X/3/5BAP2pfow98GiH8zOPsuK5uJB7r+lB8TH/IOPilGoH4dvdEU1lhszrKNAxc9ywM0eTkCOgeqMfeyTNYB1aUGydeNjaUxtaZGZg== X-YMail-OSG: QkgXs.0VM1mtyGJjo2dZr5qzjMv0PXv2GspoUMji8bEXJljb22uA.g6PofTMo_8 PXH9otrYtbHTjMIRRHbP8iXLv9HafpT0UPvxWclR3e.ZsBqRlh.qrE8PC_T0Hf8Vifh_1s13uyK9 I5w0UdRZtBOoqG5XnkXNvhjYNpfEzo2T8ARsQrqD4Nvc1y4PPWrzUabkJNFPY0bjc_JzmHRRNVEv GgHLlt60KETGNvzGr2GxewIaLbBYnK0u8H.d2AKXxrzbaitzS3LHsizTdyhjzYiuyNZ9LX9lB.pu ZHj3KhRWFuZf1uiwX_BNWqFg5cEGEUNJhs__D5AJSQhPGGJ83lgIC3CkmSfmFfBGeE2.Ec._z62f ZrK6DDlrEs0Na8azAMg_ZQYTjW_cd7dlmfj4myhWn2.R5BukGhdFqmkl4LLczxfFnaAZMYYgD5Df m8F4hA8y6H5T_zrALvGC8AH0dOpjjA2z1WBpO_K.2Jyg5_EHl9nebgTwdRQtldnlKdWRitT32emC VtavYumInA91UaLpkhMESx3Eu.DTprsvjd8s.xlOUvyVrSNKyN9I50rSDJXfcSvcQYa4J.AjmcQn lw3bhM1FpQGxtJpfPPV5vKToFDaj4WVI.pH9ReHQtLLJ8Vad5vJQl543awyGKehENmRAlepoLw88 XnxJHfwFM7HVutHM5uAMn1k6WPqTREjy80iUUMoOOkz_XYj4Uk19O8qQMT9RuzevZCbaCudqi8dz KVkft1c_0e4S.JfkBQoscLdVbIl18LClfHlFrX.m_6nUZCY4cS7xnXy6Uw6F3DAtGjBvTr_RNxPF FoMV8BBIvQDBg_.jaZlwPONfIPj23dzFL6.a7_eNUHAauryIcJagpApZBbcsLq.CCNK6qqNLgxft NrNrvRmlDjs6tj5WCmZPMqVKTgfSrBh7lfwqSWXdyTCwesfargZPUp3mpeCn4hrIZPEtiDDQkCtG ebM.Bx.5S1k4eMmYK_5qsBO1BBgl5Nd_5anJV8wDGt5XQO2aiQoMiJt4i5mlpBM.4ZWC1CWtA0Ie DHer3T1kYlo2h0pIZuXEzep3f67PA04CBGGDBcL3qi5gUf8.4gtxbISqtbWHzrUb78AZWf0NdPO9 E3SRNIsP1aEmrYk8dI1m3UGj2Ap1gibTODn1TvTKo8Rr3UdzzT1dMnna6HGqzYx4NOretASKnlUN oLhLPc4h4R61EfjetaTEq8mteEvrVXOKLGWA3yVaG3CvnAq5PWNRMwedckWx46UdEzHVTLJtrWmE AEW1F.Ea_KrKX9u2dVGUJtMP4PNJ0OCGhVwo4DzFRd.xEjf3lg0CHTEYqZ3SexiiE4mwOxxZwIdM dr6H9UP2gqPBVB9wKrjU0EGb5FP.CLqOtFTby066FHluh20d40ayXomj82d2lDgo979hRNrOGk4h Yl8bcvD1UG_Db_q0.ZP0CaMyiJv_kY8NT4AUfhmph7XPncSiL0deqzDW1biVo_x_HXA59hyBYwVQ xVaKmqD_B6d_De2uyh2OoqyrO9mXLCjyDmzP6d2pc6QDcj4ANBF9O.Jw1HWLb0iuhgbv.uo.UEmF kvfYNQywi1lA.mRhQROPP7hKJf3dU7SBtMDEPshyNCpBdhihTOvZe_iu4e.sYrveiOkXQO1Imtm5 DgOZGrQjFPB4M218Tl8QYakGfPeWPkOGmBHNibX4vxfXA6J75s7efWRal89PS44YjbNvCZSjc8WO 01Y2Y3OaYXDgSiVzUFFdGicaHrZ9VoWOosGglAuNVO0kiheoBiu1LKFpvO0WgjuOjQGHKTmIzxJY la.UEYBNJslAYDB80kD8AtHPkiijButFbwToDHnUhwQobsPTVsmoDl_bi5E4NXDzv6gNr9VHeu2t hCa1N.0hdorkRrQXfJbIgL_33wPmqOVjgPfo6J5SXFVZ04Dyd_bk8DESWMxejHtvLeLY1jtLfDRm .HWzlwrMTIvugEWM.3BTPw_JYNjTGhouogvwrPfKW_TDQ_77KIdV3Whi0p6.gVZSI.hvpLQddK6t vFqPjm27lsXCvpRa5q2HzV39hdQprirO.aslUBLb7UjAuu30.DKT3DA.g3pgs1XlPQESckQ56nyq NSx_jnU.P6HXHCHAFyaigFuDDllAThTggw0ELiLxCgMOlX9_DdBRDgHgUwmBXTzTRcaDTziYtiNq TY7USjXV66oU1p4Hp9.xQTO_qucE3geIrNE77uMg4OqdIEpQmjV7UtH1Y5zl.mBvYIhbKo2vEWnM VbIrr_mE4cpPDNvTFjuCv4lwuEeg2zh93RY4_hlcRhC0HhwJ8WGtmSlviD3rY841PDuuS_lB47a9 RhUMb X-Sonic-MF: X-Sonic-ID: 7c059123-b4c3-41c5-83fe-157d27477191 Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Tue, 29 Aug 2023 10:27:17 +0000 Received: by hermes--production-ne1-7b767b77cc-ht9fv (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 529693c29ed31783430621bcd3b8e4af; Tue, 29 Aug 2023 10:27:11 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: RE: Very slow scp performance comparing to Linux Message-Id: <98C8E07C-2247-4439-8836-ED350CC83F16@yahoo.com> Date: Tue, 29 Aug 2023 03:27:00 -0700 Cc: FreeBSD Hackers To: weh@microsoft.com X-Mailer: Apple Mail (2.3731.700.6) References: <98C8E07C-2247-4439-8836-ED350CC83F16.ref@yahoo.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.147:from]; BLOCKLISTDE_FAIL(0.00)[98.137.64.147:server fail]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.147:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RZkBb0LkHz4d8j Wei Hu wrote on Date: Tue, 29 Aug 2023 07:07:39 UTC : > Sorry for the top posting. But I don't want to make it look too messy. = Here is the > Information that I have missed in my original email. >=20 > All VMs are running on Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz = K8-class CPU). >=20 > FreeBSD VMs are 16 vcpu with 128 GB memory, in non-debug build: > 14.0-ALPHA1 FreeBSD 14.0-ALPHA1 amd64 1400094 #7 = nodbg-n264692-59e706ffee52-dirty... = /usr/obj/usr/src/main/amd64.amd64/sys/GENERIC-NODEBUG amd64 >=20 > Ubuntu VMs are 4 vcpu with 32 GB memory, kernel version: > 6.2.0-1009-azure #9~22.04.3-Ubuntu SMP Tue Aug 1 20:51:07 UTC 2023 = x86_64 x86_64 x86_64 GNU/Linux >=20 > I did a couple more tests as suggested by others in this thread. In = recap: >=20 > Scp to localhost, FreeBSD (ufs) vs Ubuntu (ext4): 70 MB/s vs 550 MB/s > Scp to localhost, FreeBSD (tmpfs) vs Ubuntu (tmpfs): 630 MB/s vs 660 = MB/s >=20 > Iperf3 single stream to localhost: FreeBSD vs Ubuntu: 30.9 Gb/s vs = 48.8 Gb/s >=20 > Would these numbers suggest that > 1. ext4 caches a lot more than ufs? > 2. there is a tcp performance gap in the network stack between FreeBSD = and Ubuntu? >=20 > Would you also try run scp on ufs on your bare metal arm host? I am = curious to now how different between ufs and zfs. For this round I'm rebooting between the unxz and the 1st scp. So I'll also have zfs results again. I'll also do a 2nd scp (no reboot) to see if it gets notably different results. . . . Well, I just got FreeBSD main [so: 15] running under HyperV on the Windows Dev Kit 2023. So reporting for there first. This was via an ssh session. The context is ZFS. The VM file size is fixed, as is the RAM size. 6 cores (of 8) and 24576 MiBytes assigned (of 32 GiBytes) to the one FreeBSD instance. The VM file is on the internal NVMe drive in the Windows 11 Pro file system in the default place. (I was having it copy the hardrive media to the VM file when I started this process. Modern HyperV no longer seems to support direct use of USB3 physical media. I first had to produce a copy of the material on smaller media so that a fixed VM file size from a copy to create the VM file would fit in the NVMe's free space.) # uname -apKU FreeBSD CA78C-WDK23s-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 = 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT = 2023 = root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/a= rm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000 (The ZFS content is a copy of the USB3 interfaced ZFS Optane media's content previously reported on. So the installed system was built with -mcpu=3D based optimization, as noted before.) # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 193.6MB/s 00:26 # rm ~/FreeBSD-14-TEST.img # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 198.0MB/s 00:25 So, faster than what you are reporting for the Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) context. For reference: # gpart show -pl =3D> 40 468862055 da0 GPT (224G) 40 32728 - free - (16M) 32768 102400 da0p1 wdk23sCA78Cefi (50M) 135168 421703680 da0p2 wdk23sCA78Czfs (201G) 421838848 47022080 da0p3 wdk23sCA78Cswp22 (22G) 468860928 1167 - free - (584K) # zpool list NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP = HEALTH ALTROOT zwdk23s 200G 79.8G 120G - - 0% 39% 1.00x = ONLINE - (UFS would have notably more allocated and less free for the same size partition.) The below is be based on the HoneyComb (16 cortex-a72's) since I've got the HyperV context going on the Windows Dev Kit 2023 at the moment. UFS first: # uname -apKU FreeBSD HC-CA72-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 = #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT 2023 = root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000 # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 129.7MB/s 00:39 # rm ~/FreeBSD-14-TEST.img # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 130.9MB/s 00:39 So, faster than what you are reporting for the Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) context. Note: This is via a U.2 Optane 960 GB media and an M.2 adapter instead of being via a PCIe Optane 960 GB media in the PCIe slot. ZFS second: # uname -apKU FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 = #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT 2023 = root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000 # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 121.1MB/s 00:42 # rm ~/FreeBSD-14-TEST.img # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img (root@localhost) Password for root@CA72-16Gp-ZFS: FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 124.6MB/s 00:41 So, faster than what you are reporting for the Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) context. Note: This is via a PCIe Optane 960 GB media in the PCIe slot. UFS was slightly faster then ZFS for the HoneyComb context but there is the M.2 vs. PCIe difference as well. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Tue Aug 29 11:22:26 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZlQW6WSqz4rhH1 for ; Tue, 29 Aug 2023 11:22:43 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-54.consmr.mail.gq1.yahoo.com (sonic307-54.consmr.mail.gq1.yahoo.com [98.137.64.30]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZlQV50gNz4mSj for ; Tue, 29 Aug 2023 11:22:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=tIV2DMdQ; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.30 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693308160; bh=LCvqO5u/ULi/vmytuXwB0BMMVBkavW2kRHUkmRdsS0g=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=tIV2DMdQvPLdL4iqoy4SDnKLbYD07uTI57SbcloYEM5lemE0TjamcvjVp4MSPsC9ATjFiwgpYRcxNRg30kqDZrnKTulQGj9/4WMTAl0STALvZxJUKrJ0vXYPwZgUst9dW+6SF/3uwmJsRnNY4cu4cnP4cdx5VEUe1qgye0cZrrfD6iTuDd+m21y/Qmp147kDywmjCeuKA9iVRw9Wn4YUSiEjqaPmbmZRmfE2dxsxROCWH6r0Bwojdd5bsbAiCcHqVv+e2QPe2CMHHdCKaXnYNW4xZ2/dj3d446iD+4Ws8Rutjhv28G1VJM8hUkppYsjGONWJSq/X7cTweuhuIOkxHw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693308160; bh=RYeSsZzikPg/r5zu1d7qgQm4kAqYq5Vdy2KFxCAEXOl=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=IEkr/K7agbmv8ai3pEIJ8jSjjGVhj7QLk/mC44Y7exenfQeJNw8C5I9AxU4r2VpcafqUhgkB7/nRFkNacYhtCklDeGlxT3+RXjqWxysf/76unn3RuHLyIEje46WbAVAlKw5v5cfCoMsRtL/Hz9VE1jKA6MK2NbPAEFiZ1Xfm/8NMG9XWBU1vzjFr6lreWgm0SYYyfG7CGzGkdDE/Ouyj7zZq9SX6Bsp7lx8Ez/c81+02NZ/v7fIZW6e1vVjsrxVAUeuWa0uOEVtedN71Lo/ad05DtJVAl/Jn3DBstnlLKE+d1LNGuMenw//et0XES+8HrxLgMjRA5UIsETMNBbai2w== X-YMail-OSG: r0uPXokVM1m2Fpo5RB86BiG3FsWjEZOLy3Gl0MjrBrSpq7QOcrLQFZ2xtX9Y.OY HDJtL0p7quJwFQC0hIHuLvVBrkdU7slote9bLA4RplttkD.uhBZTP1XLEp0Qp.1nVr8QFqNRMFXJ uv0ZsIGxVTu6rLluFOhLu_QvAadrZnR3F2Rh9EhfUfPv2bwAn8UrqehxIkXQI7J2MGGEwARjz9Lk P.EgqMBsmHZ62bbelvCHDhDMv6TIZ_jnxnL72XbXmF3arzzbiBgyh5DPWmdwhkZJvTkxoMAvpfw5 z_8IdnDuR2tQ6ndRBmjr.EoJoiZVsx9meWdAz2KxBExBvC84PkHzC2vUyYkV0SbRBr2TroLBWtKE TkcT3oN5ULQvFPpg3M9G4VDrJ9e5JzrEfYM4FsI9z7nDq3hiL7jWvJLx.K0klinT24Btu1WgSABX CUnTPr3o1DiNu9FuuQsN2o9uwqnuOijOIv8JX816G1H5BAzQ8QEINXEXY6xZORUK4SStyQjB9SkF Hn3_M5scEYg4pQVOhTLeQ5aiPgXahsNdkqmYAox8NqFN61k0GTwl6mBpvZ0lqVkZ6Ut7fDlH6Vde 6LeRUsmBfvVtPDWouSRBNZlkqU6d0QhFDUpP3MsXakb6wCLPhU8Ic9zpoeas6VA4AEqmFiwVRE96 KRUzFWSJw0j9KShFVqnR43g.XRvm.LyIlg2MBnfITvEstazebhh9HXfpk43MHEvFKnbsOdRYN70f Lpo7xM6z9lktOu1JOim3KveQZJHloE5tg.3DFGNS39xbNUaSRU9PUSgRRuPzkoUs4B7mnCtn9g1I 2SiivmThTIsEZQ0QGydYpZ0FHuEnOF10gWmk6VddQPnrOyQTzzhekQUMUCmWhwHL541BPvkuXX7G jk2NY6JUwy4LLJ_c0Qf_ydZ4VmNib__V4AnYjJFaoa574Oq4qplrSWegb.P5A8GWgCz28f0cO9Lb lex76D8n6mP7xNUlKAmfw1w3KrnQOvSkXPU6HE_2lkgtfAGXbcwGsFzGB7aD5XV6duCma0tSXrG1 AE0EgMV3zrnWOG2BSOd7glw7MynSzkzMS93viNwcUcGicoIbxa6P8SsMCYau1qip5ZdmG81af8V. tyd0rUPxVTD85FCVm_hjssxYzyZCdwV7Ey1hNmZ0mHGcqG0wn6NUlSo2FpvczMXOdp7Np_6CuZFs omXmtjkBQhRoHNCEeoRCOcVil2fgIy_cSNdK.TIXyFz.KQcYWkixO8g7PXxCRWyzq8.ApGSay84e W9KwCfeuMbYLgwAy0igD8pJjiMFXO4t0xOkkGczH6sSoKaiuK3V2pKV6sWOqr0wTNxmBWD6G9I3Y 2h7aeVr645zFajxor_mGVsjoc6YB9jzdIOsQ1z.Qw_Kqs6lEXNJyA9YEuvWAuYUNTXyibOneq3jJ k7ral_trLAjBlm08sGLPDLIyvqTLxOpzdJ_MgV4OzAQ3diUwKw1lYaP1OaefEphY7Lr4oI045KR. 8DgbtQbDm7v6oaMPyt57JuaXRbiPQKwcJhlHn1kgSvnCYMNZvjmZOTPrQZfmkD9Jf4m0UFEOvP2A Qf_RH9fcyfSR5i3kaKfUqwEnNQkzj4re6w88NkC.L6EZqB5VrjXGPoIciEUvyVMjFqxxBQMIPbsR VbHarW9WX7koiJxgxeoQNg.ZDEk4PMVLzmNmBCiUcTItkEqXfwv5OpO3PXWTQut8cBvCXmxujVpF pqB4D_4OMTybhcT9Xe20jqoVkioTXHKyTwQ6vgivL3dglq3TlZbDyqDRShVrPPc7DY.l4SllMYVq bR9zna43OSfZKCiLucrwCGvlS3bzCrQ28yctbHHDqgUiIzc5GzTxJM21DF.09m6QycQo22RGzWPR VDi.DW7m0q49sVZcpIMcfFW1jmH_fRcshVwElMH8oPl7hOL4LV5NZzOC1dHZxnTR3rEvcu_Nh7oi Jqxjkw14S9aRid.wdL74RbUL6ldEvWME2Y.K0uko5cMy9ULd1i_R0rsyMZTxys4uhwqDlUWMbgPM 1vCCV2GcDsk8RMuisxhmEOmaC7vlR6B5HnPz3jcckwsLocJCEuzsnjYBk_NZyR2zd8sUchZ.7PDc BOLDL449dYtIGHNqfqKne2o.BmSH.2_wRD_0rwnES7t8DC53Bl6HAjrKLP_XPMh5A7Z2vYzEsC5q CAahDNCl3DpKUTz69BTmnji3RYrLeponiHYVjUa6A_unINFxRgsUKyM.tUTYC5nRJFPJD9Tw0xmC wNNVupLCVCQnMUBaxPAJOugXz1DnTi1FwePDtBLKNG6bU.HuGyddGdgcrCJI61EqthIaC9EvSLhI oN52Mpg-- X-Sonic-MF: X-Sonic-ID: ff211f11-9159-485f-ba8d-fa2f2dd35ec5 Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Tue, 29 Aug 2023 11:22:40 +0000 Received: by hermes--production-gq1-6b7c87dcf5-sv5pn (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 7e5d4e90102019137dffc6bf2e91f0ba; Tue, 29 Aug 2023 11:22:37 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux From: Mark Millard In-Reply-To: <98C8E07C-2247-4439-8836-ED350CC83F16@yahoo.com> Date: Tue, 29 Aug 2023 04:22:26 -0700 Cc: FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <6E433834-E192-44F1-9FF1-3814F13449FF@yahoo.com> References: <98C8E07C-2247-4439-8836-ED350CC83F16@yahoo.com> To: weh@microsoft.com X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.995]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.30:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.30:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RZlQV50gNz4mSj [Adding USB3/U.2 Optane UFS Windows Dev Kit 2023 scp examples, no VM's involved.] On Aug 29, 2023, at 03:27, Mark Millard wrote: > Wei Hu wrote on > Date: Tue, 29 Aug 2023 07:07:39 UTC : >=20 >> Sorry for the top posting. But I don't want to make it look too = messy. Here is the >> Information that I have missed in my original email. >>=20 >> All VMs are running on Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz = K8-class CPU). >>=20 >> FreeBSD VMs are 16 vcpu with 128 GB memory, in non-debug build: >> 14.0-ALPHA1 FreeBSD 14.0-ALPHA1 amd64 1400094 #7 = nodbg-n264692-59e706ffee52-dirty... = /usr/obj/usr/src/main/amd64.amd64/sys/GENERIC-NODEBUG amd64 >>=20 >> Ubuntu VMs are 4 vcpu with 32 GB memory, kernel version: >> 6.2.0-1009-azure #9~22.04.3-Ubuntu SMP Tue Aug 1 20:51:07 UTC 2023 = x86_64 x86_64 x86_64 GNU/Linux >>=20 >> I did a couple more tests as suggested by others in this thread. In = recap: >>=20 >> Scp to localhost, FreeBSD (ufs) vs Ubuntu (ext4): 70 MB/s vs 550 MB/s >> Scp to localhost, FreeBSD (tmpfs) vs Ubuntu (tmpfs): 630 MB/s vs 660 = MB/s >>=20 >> Iperf3 single stream to localhost: FreeBSD vs Ubuntu: 30.9 Gb/s vs = 48.8 Gb/s >>=20 >> Would these numbers suggest that >> 1. ext4 caches a lot more than ufs? >> 2. there is a tcp performance gap in the network stack between = FreeBSD and Ubuntu? >>=20 >> Would you also try run scp on ufs on your bare metal arm host? I am = curious to now how different between ufs and zfs. >=20 >=20 > For this round I'm rebooting between the unxz and the 1st scp. > So I'll also have zfs results again. I'll also do a 2nd scp > (no reboot) to see if it gets notably different results. >=20 > . . . >=20 > Well, I just got FreeBSD main [so: 15] running under > HyperV on the Windows Dev Kit 2023. So reporting for > there first. This was via an ssh session. The context > is ZFS. The VM file size is fixed, as is the RAM size. > 6 cores (of 8) and 24576 MiBytes assigned (of 32 > GiBytes) to the one FreeBSD instance. The VM file is > on the internal NVMe drive in the Windows 11 Pro file > system in the default place. >=20 > (I was having it copy the hardrive media to the VM file > when I started this process. Modern HyperV no longer > seems to support direct use of USB3 physical media. I > first had to produce a copy of the material on smaller > media so that a fixed VM file size from a copy to > create the VM file would fit in the NVMe's free space.) >=20 > # uname -apKU > FreeBSD CA78C-WDK23s-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 = 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT = 2023 = root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/a= rm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000 >=20 > (The ZFS content is a copy of the USB3 interfaced > ZFS Optane media's content previously reported on. > So the installed system was built with -mcpu=3D based > optimization, as noted before.) >=20 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 193.6MB/s 00:26 >=20 > # rm ~/FreeBSD-14-TEST.img > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 198.0MB/s 00:25 >=20 >=20 > So, faster than what you are reporting for the > Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) > context. >=20 > For reference: >=20 > # gpart show -pl > =3D> 40 468862055 da0 GPT (224G) > 40 32728 - free - (16M) > 32768 102400 da0p1 wdk23sCA78Cefi (50M) > 135168 421703680 da0p2 wdk23sCA78Czfs (201G) > 421838848 47022080 da0p3 wdk23sCA78Cswp22 (22G) > 468860928 1167 - free - (584K) >=20 > # zpool list > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP = HEALTH ALTROOT > zwdk23s 200G 79.8G 120G - - 0% 39% 1.00x = ONLINE - >=20 > (UFS would have notably more allocated and less free > for the same size partition.) >=20 >=20 >=20 > The below is be based on the HoneyComb (16 cortex-a72's) > since I've got the HyperV context going on the Windows > Dev Kit 2023 at the moment. >=20 >=20 > UFS first: >=20 > # uname -apKU > FreeBSD HC-CA72-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 1500000 = #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT 2023 = root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000 >=20 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 129.7MB/s 00:39 >=20 > # rm ~/FreeBSD-14-TEST.img > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 130.9MB/s 00:39 >=20 >=20 > So, faster than what you are reporting for the > Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) > context. >=20 > Note: This is via a U.2 Optane 960 GB media and an M.2 adapter > instead of being via a PCIe Optane 960 GB media in the PCIe > slot. >=20 >=20 > ZFS second: >=20 > # uname -apKU > FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 = 1500000 #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT = 2023 = root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg-clang/usr/main-src/arm6= 4.aarch64/sys/GENERIC-NODBG-CA72 arm64 aarch64 1500000 1500000 >=20 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 121.1MB/s 00:42 >=20 > # rm ~/FreeBSD-14-TEST.img > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img > (root@localhost) Password for root@CA72-16Gp-ZFS: > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 124.6MB/s 00:41 >=20 >=20 > So, faster than what you are reporting for the > Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) > context. >=20 > Note: This is via a PCIe Optane 960 GB media in the > PCIe slot. >=20 >=20 > UFS was slightly faster then ZFS for the HoneyComb > context but there is the M.2 vs. PCIe difference > as well. >=20 # uname -apKU FreeBSD CA78C-WDK23-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 = 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT = 2023 = root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg-clang/usr/main-src/a= rm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 aarch64 1500000 1500000 Again, a -mcpu=3D optimized build context for the FreeBSD in operation. (Still rebooting first. Then . . .) # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 199.3MB/s 00:25 # rm ~/FreeBSD-14-TEST.img # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 204.9MB/s 00:24 So, faster than what you are reporting for the Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) context. The Windows Dev Kit 2023 figures are generally faster than the HoneyComb figures. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Tue Aug 29 12:55:35 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZnTt1Lxlz4rmQn for ; Tue, 29 Aug 2023 12:55:46 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from HK2P15301CU002.outbound.protection.outlook.com (mail-eastasiaazlp170100001.outbound.protection.outlook.com [IPv6:2a01:111:f403:c400::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZnTs25k9z3G5y for ; Tue, 29 Aug 2023 12:55:45 +0000 (UTC) (envelope-from weh@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eQXKaxLhT+yQYDSdhDZOYmUI3EI8G29jycdi77ICCPgxI7XGgh2dXOhuSRb0Zd0+wWa4G+1yzTte4C9sSSkGDHnNZoF9fPItFQrlVSwgb1MPcD4T3amVqX/A7bdPGaunSWywSJS4RziM8ZV2fmuoFT49kwQMIQrVczkzYfgFT38tcRiN6TBoHk6HKtftiUGXYAXnAMFN9hDwZYbMxnfuYIUZ5Z+LTjKVR5tiQO9Lm7hsmN3/lizNxvlaTkrMHKvxpZYQwnQBOCavXdqTs70KZBQEppDdI3z6IR8VQ4NI3JiASDqWI4IMZYRJi/mR2mnN7FxK9SQOgwAVOIpbgXlHtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=AM91Rsn0QFQGPj5eBcbOIC6e7ASPYfGZwgT+PwFDW5Y=; b=CSDJB8S4dH/PIok5WFwhYX9ld5MiOBLmm1BXeR+Cl+sil3e4uK4LJY7Ch+cLZbubh/Ob4W7EKB8nOeNxQKAU/crwh6wN69zaZex81K/wB171ubgpnTOMf0CGtwR09EjCTU6xLsVmFJGZ9v1oDaHcUj4IuWdFi0G3g9285yoNmZY2HihoJR4aI+1gkEPcBnLsIZHG3/mqDaCI1R+dNx0/Xk7Btzk7RqAlGiCIYFXt/TBbrhr4FMoCMXZDh1ihBv2RPgBN4O9nqTlk1TTEsn2ZnyMeV0A5LHdRiVBjmd2qNRVt5SgUKXUZfSaKvhQTm1TXPAE54gknEtZ90ivN65ZlAA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AM91Rsn0QFQGPj5eBcbOIC6e7ASPYfGZwgT+PwFDW5Y=; b=FOwzFnN/szqL8dAIAV1muJvWc0Bd5FyXNGtiyq6TTS2G2buRyB7+daN+CZMSBzWe32J+254E+kxi4hCheV9YnIDg1BxE6H7lKd2jBinTCtgebwOTec+hi6IwcmEnD6lIm9h8Jcmus1dcaZBnuf7p04raJvTKQoJuM7Or87JXNtA= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by PSAP153MB0502.APCP153.PROD.OUTLOOK.COM (2603:1096:301:90::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.9; Tue, 29 Aug 2023 12:55:37 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83%7]) with mapi id 15.20.6768.007; Tue, 29 Aug 2023 12:55:36 +0000 From: Wei Hu To: Mark Millard CC: FreeBSD Hackers Subject: RE: Very slow scp performance comparing to Linux Thread-Topic: Very slow scp performance comparing to Linux Thread-Index: AQHZ2mNniu2mI7eQ9kazPr8glcZkyrABIWYAgAAYXWA= Date: Tue, 29 Aug 2023 12:55:35 +0000 Message-ID: References: <98C8E07C-2247-4439-8836-ED350CC83F16@yahoo.com> <6E433834-E192-44F1-9FF1-3814F13449FF@yahoo.com> In-Reply-To: <6E433834-E192-44F1-9FF1-3814F13449FF@yahoo.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=553017a6-a2be-4a47-b25c-7ef67c14489b;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-08-29T12:49:38Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2P153MB0441:EE_|PSAP153MB0502:EE_ x-ms-office365-filtering-correlation-id: e1b6f4a5-76d7-4d8a-8468-08dba88f3ce0 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 7w/RjSqaCN00rd5WqoBo5RQRfG8RCg05ZF+VIv8icyfg/VDxDgWZZQM9y50TrUalTRK6T4JLICkuve0MeQDvpBbqNVB2UNLBahoZZBam+YgOCdLtEECq+fZYx8/S22vo/pD4DouNCbOmEODORVS8VzwKAs67OQc5EFKYSf5IgoUurlqqP1ttSzTFnRdWQI79i1Bn//k/hd3Pgrzz8zCgj5/YbBcKT0p6iKN3xRdqvFwD/n6LMEWOeBoJspDrt4g7aIORHbhyQzkDzzdN8HgSZr6+FL0ZJpxQNL1xlOIyHA4ruOT13Ii6+QXwRu0qLhpBA86ta8Jom6K0U1kxDBtk22cxFw2P5XaUx0rVWvBY7u36lnFG/ZMJQKkyAyWZLnJqEgRqodt5KwybniZkOU7UUryQCMJOA/ygDYaM8awin4aSINin88W8FLrEdz1eIZpf4mgo/OTXSwNzbfnWcsdz4ZFkITjDJBoMzWzRGgTuyCRNRfUiWdO+PgwvbP89dvHokjlmQnmQbywt9ZKktew2TT1TA8gyUf69n3YfDleW18siOfu+XdeA3UAnwpTtwHwZbusaEXuHHRbGF2MHdEjgLgm0FkN88dv7j9tnfhTqVIxiyDmykBxXkWWnZvxFaPWs3XtwZ6zp6knctY3MMVIwqO3hkkwjRqxem+GughwD8gfWp89cdrlgN4uVFS20LG4kFpJwKs65Y5XGqFit67en/Q== x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(396003)(366004)(376002)(136003)(39860400002)(346002)(1800799009)(451199024)(186009)(478600001)(86362001)(55016003)(33656002)(2906002)(19627235002)(83380400001)(82960400001)(38070700005)(38100700002)(6916009)(82950400001)(41300700001)(316002)(122000001)(5660300002)(76116006)(52536014)(8676002)(4326008)(8936002)(66946007)(64756008)(66446008)(66476007)(66556008)(53546011)(7696005)(6506007)(12101799020)(71200400001)(8990500004)(9686003)(10290500003);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?PCA7r1bxG56jJJtvFO4VhqUiXPlWMT+gYOe8RwHeOHNFa8ekf/GLwPQ5t/2c?= =?us-ascii?Q?kV+g1uCFVj71g+D9/8QlAZ9MfimkOkupNP+CUhwI5C7oBHgXPZbVtCVmSRLb?= =?us-ascii?Q?dacbOIT5NsjBgdwWKtUUtRuvYNVamawkZMYOle1IXm5F8sJH2y8xQqlYl96v?= =?us-ascii?Q?h67+AvOCJQQ8IewsQEB5nlSpx1DAuuVz9JLWspGGRZfKD7rZi6dldVzCIZne?= =?us-ascii?Q?HLeHoEZKBaGpDL4Au1L/nEmES6Ae1HWVdoXK2YLLJVDo/vxr4hl1Unw86NTC?= =?us-ascii?Q?hS5R4D1tCtz8h6vIG4iaSTTmzx5Jfibx1MiOKS1u4Lj9ZFF31q+Io+XjX6h1?= =?us-ascii?Q?XYJkgPOjQeusduzZprmd2ujCT2+cjMZJ+Akf1pE+5iQpOO6r+FW3302ktcyF?= =?us-ascii?Q?RMToMBOtHXUckxKVyHYeF4XUJGG0el8egHOwwU7b5WsaJarVwelYaUYq2Pre?= =?us-ascii?Q?JbB4VEj6WFJDMyHM35LGMvmInNnH2Ux0uS19NvSa9Vj+W98KJk7VqwvFjJAW?= =?us-ascii?Q?PA/HpsM/JlRK+4qE0x57Fs5gNieSGscjuM2PzUqscx0R5V4Sq2hEYGlol47x?= =?us-ascii?Q?Yn2Xv+dY9eXmvg5cLjB5DiAEpawEykR2XVFlxS03xrOOnks+CcCPdg3o1bz4?= =?us-ascii?Q?OS37uDPI9TRA9cEd9UvWJdc0uadP3SVEIbvRDP1T+GZJmGHfRNCZCU6LHWhr?= =?us-ascii?Q?EaJPICUlI/CfznyggjlbWCL0baiV5YmF3sqBVvajsVHEjioJSF1zFQ52/dUz?= =?us-ascii?Q?DJra3nnqvhuo5SY6iYNm2hNpH6g4iazLQRZyNCth2AF2LyuUeM+t9nfVbDsj?= =?us-ascii?Q?fj/z4m7pfKq9iTuucMITk86iBtcS+nIBlg8RqqKA+QxwtHcHv6lRmIiUBIND?= =?us-ascii?Q?/p9CmTUPVSwXbJyE5lDZntIV79DsUAAXnvlK42TCrZAQ1dkjoSt5mo6zWOaQ?= =?us-ascii?Q?z3M/CBxfIOaxpvbgxCfI36+sqlcNOfpV+eR0ovaMS35D5/9VWKjBU+YhajTz?= =?us-ascii?Q?a/bArUkvyo2jeqviWaeeB4yrira2ttLQ+yEaAdvNriVQekPYbeosVpBrHECO?= =?us-ascii?Q?4AyiSWbATGMvAyshlADdnffStk2ZRSWYuQ5llskMXrWI17/OyavXPS+a/7+4?= =?us-ascii?Q?VmGFE7pa3ucFf8pDKHzOzfcQHmdIJmpyD9SICaQwS2Ov5dqDi8FWDr5NeJtC?= =?us-ascii?Q?JyQRkMd4eXsB1dPmmuEOSbGaK+Em8Qu/TO8hh5M8ovZklEPvXMNpNqaEMEwn?= =?us-ascii?Q?Q8o1q1ULaHWvmWy+Anpt8yXIQANMqHcFzBJBFkhkrLSWgd7/z7aM8OLKEP7g?= =?us-ascii?Q?JgGiXCdTEpMQSv+4HE43yP3djSo0BlFIG9rN3iEqGQBcqq4fl4Bw09zrpbMB?= =?us-ascii?Q?FK2/TRRVKRKoDCiRVj1L6JgZMbNv3HzokyL5/htMxVpsOaoTPvZBnt5NNIxu?= =?us-ascii?Q?P6s9L6Sl4L0goN4hD+PE8t2s5mRjlYeBX5Gq5fNeRQNZat+SbzPnUg0hM+5B?= =?us-ascii?Q?HJkCoZNdExt/+LmaAC2Co89SLSvElC34SqHiHB5NKtF8Nan02pUOPP3XbrVU?= =?us-ascii?Q?XnJmFL78igtcgxF2o5ENgtX23At7COkF5LwQrQN/lzZUra7z1/IZ5VuO8zl8?= =?us-ascii?Q?aw=3D=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: e1b6f4a5-76d7-4d8a-8468-08dba88f3ce0 X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Aug 2023 12:55:35.8077 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: TjrFpA45XmN0mUGVxRIN+CxT8DQG45JxhMRcf0XpX6tX6LmXUwFjs5sIXUMbK0CxoVmIIFeJDwVXd5Bn0qQPRw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PSAP153MB0502 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:2a01:111:f000::/36, country:US] X-Rspamd-Queue-Id: 4RZnTs25k9z3G5y Hi Mark, Thanks for the update. Seems the numbers are the same on zfs and ufs. That'= s=20 good to know.=20 Yes, your numbers on ARM64 are better than mine on Intel. However, my origi= nal intention was to find out why scp on Linux is performing much better than F= reeBSD under the same hardware env.=20 Is it possible to try Linux in your ARM64 setting? I am using Ubuntu 22.04 = on ext4=20 file system. Thanks, Wei=20 > -----Original Message----- > From: Mark Millard > Sent: Tuesday, August 29, 2023 7:22 PM > To: Wei Hu > Cc: FreeBSD Hackers > Subject: Re: Very slow scp performance comparing to Linux >=20 > [Adding USB3/U.2 Optane UFS Windows Dev Kit 2023 scp examples, no VM's > involved.] >=20 > On Aug 29, 2023, at 03:27, Mark Millard wrote: >=20 > > Wei Hu wrote on > > Date: Tue, 29 Aug 2023 07:07:39 UTC : > > > >> Sorry for the top posting. But I don't want to make it look too > >> messy. Here is the Information that I have missed in my original email= . > >> > >> All VMs are running on Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz > K8-class CPU). > >> > >> FreeBSD VMs are 16 vcpu with 128 GB memory, in non-debug build: > >> 14.0-ALPHA1 FreeBSD 14.0-ALPHA1 amd64 1400094 #7 > >> nodbg-n264692-59e706ffee52-dirty... > >> /usr/obj/usr/src/main/amd64.amd64/sys/GENERIC-NODEBUG amd64 > >> > >> Ubuntu VMs are 4 vcpu with 32 GB memory, kernel version: > >> 6.2.0-1009-azure #9~22.04.3-Ubuntu SMP Tue Aug 1 20:51:07 UTC 2023 > >> x86_64 x86_64 x86_64 GNU/Linux > >> > >> I did a couple more tests as suggested by others in this thread. In re= cap: > >> > >> Scp to localhost, FreeBSD (ufs) vs Ubuntu (ext4): 70 MB/s vs 550 MB/s > >> Scp to localhost, FreeBSD (tmpfs) vs Ubuntu (tmpfs): 630 MB/s vs 660 > >> MB/s > >> > >> Iperf3 single stream to localhost: FreeBSD vs Ubuntu: 30.9 Gb/s vs > >> 48.8 Gb/s > >> > >> Would these numbers suggest that > >> 1. ext4 caches a lot more than ufs? > >> 2. there is a tcp performance gap in the network stack between FreeBSD > and Ubuntu? > >> > >> Would you also try run scp on ufs on your bare metal arm host? I am > curious to now how different between ufs and zfs. > > > > > > For this round I'm rebooting between the unxz and the 1st scp. > > So I'll also have zfs results again. I'll also do a 2nd scp (no > > reboot) to see if it gets notably different results. > > > > . . . > > > > Well, I just got FreeBSD main [so: 15] running under HyperV on the > > Windows Dev Kit 2023. So reporting for there first. This was via an > > ssh session. The context is ZFS. The VM file size is fixed, as is the > > RAM size. > > 6 cores (of 8) and 24576 MiBytes assigned (of 32 > > GiBytes) to the one FreeBSD instance. The VM file is on the internal > > NVMe drive in the Windows 11 Pro file system in the default place. > > > > (I was having it copy the hardrive media to the VM file when I started > > this process. Modern HyperV no longer seems to support direct use of > > USB3 physical media. I first had to produce a copy of the material on > > smaller media so that a fixed VM file size from a copy to create the > > VM file would fit in the NVMe's free space.) > > > > # uname -apKU > > FreeBSD CA78C-WDK23s-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT > aarch64 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 > 09:20:31 PDT 2023 root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main- > CA78C-nodbg-clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG- > CA78C arm64 aarch64 1500000 1500000 > > > > (The ZFS content is a copy of the USB3 interfaced ZFS Optane media's > > content previously reported on. > > So the installed system was built with -mcpu=3D based optimization, as > > noted before.) > > > > # scp > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.i > > mg root@localhost:FreeBSD-14-TEST.img > > . . . > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 193.6MB/s 00:26 > > > > # rm ~/FreeBSD-14-TEST.img > > # scp > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.i > > mg root@localhost:FreeBSD-14-TEST.img > > . . . > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 198.0MB/s 00:25 > > > > > > So, faster than what you are reporting for the > > Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) context. > > > > For reference: > > > > # gpart show -pl > > =3D> 40 468862055 da0 GPT (224G) > > 40 32728 - free - (16M) > > 32768 102400 da0p1 wdk23sCA78Cefi (50M) > > 135168 421703680 da0p2 wdk23sCA78Czfs (201G) > > 421838848 47022080 da0p3 wdk23sCA78Cswp22 (22G) > > 468860928 1167 - free - (584K) > > > > # zpool list > > NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP > HEALTH ALTROOT > > zwdk23s 200G 79.8G 120G - - 0% 39% 1.00x = ONLINE - > > > > (UFS would have notably more allocated and less free for the same size > > partition.) > > > > > > > > The below is be based on the HoneyComb (16 cortex-a72's) since I've > > got the HyperV context going on the Windows Dev Kit 2023 at the > > moment. > > > > > > UFS first: > > > > # uname -apKU > > FreeBSD HC-CA72-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 > 1500000 #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT > 2023 root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg- > clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72 arm64 > aarch64 1500000 1500000 > > > > # scp > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.i > > mg root@localhost:FreeBSD-14-TEST.img > > . . . > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 129.7MB/s 00:39 > > > > # rm ~/FreeBSD-14-TEST.img > > # scp > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.i > > mg root@localhost:FreeBSD-14-TEST.img > > . . . > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 130.9MB/s 00:39 > > > > > > So, faster than what you are reporting for the > > Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) context. > > > > Note: This is via a U.2 Optane 960 GB media and an M.2 adapter instead > > of being via a PCIe Optane 960 GB media in the PCIe slot. > > > > > > ZFS second: > > > > # uname -apKU > > FreeBSD CA72-16Gp-ZFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 > 1500000 #110 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:19:53 PDT > 2023 root@CA72-16Gp-ZFS:/usr/obj/BUILDs/main-CA72-nodbg- > clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA72 arm64 > aarch64 1500000 1500000 > > > > # scp > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.i > > mg root@localhost:FreeBSD-14-TEST.img > > . . . > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 121.1MB/s 00:42 > > > > # rm ~/FreeBSD-14-TEST.img > > # scp > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.i > > mg root@localhost:FreeBSD-14-TEST.img > > (root@localhost) Password for root@CA72-16Gp-ZFS: > > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 124.6MB/s 00:41 > > > > > > So, faster than what you are reporting for the > > Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) context. > > > > Note: This is via a PCIe Optane 960 GB media in the PCIe slot. > > > > > > UFS was slightly faster then ZFS for the HoneyComb context but there > > is the M.2 vs. PCIe difference as well. > > >=20 > # uname -apKU > FreeBSD CA78C-WDK23-UFS 15.0-CURRENT FreeBSD 15.0-CURRENT aarch64 > 1500000 #13 main-n265027-2f06449d6429-dirty: Fri Aug 25 09:20:31 PDT > 2023 root@CA78C-WDK23-ZFS:/usr/obj/BUILDs/main-CA78C-nodbg- > clang/usr/main-src/arm64.aarch64/sys/GENERIC-NODBG-CA78C arm64 > aarch64 1500000 1500000 >=20 > Again, a -mcpu=3D optimized build context for the FreeBSD in > operation. >=20 > (Still rebooting first. Then . . .) >=20 > # scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818- > 77013f29d048-264841.img root@localhost:FreeBSD-14-TEST.img > . . . > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 199.3MB/s 00:25 >=20 > # rm ~/FreeBSD-14-TEST.img > # scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818- > 77013f29d048-264841.img root@localhost:FreeBSD-14-TEST.img > . . . > FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048- > 264841.img = 100% 5120MB > 204.9MB/s 00:24 >=20 >=20 > So, faster than what you are reporting for the > Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class CPU) > context. >=20 > The Windows Dev Kit 2023 figures are generally faster than the > HoneyComb figures. >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com From nobody Tue Aug 29 19:52:52 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RZylV4vKFz4s881 for ; Tue, 29 Aug 2023 19:53:10 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic307-55.consmr.mail.gq1.yahoo.com (sonic307-55.consmr.mail.gq1.yahoo.com [98.137.64.31]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RZylT46RMz4cSZ for ; Tue, 29 Aug 2023 19:53:09 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="h0lF/dlj"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.31 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693338788; bh=KfCO4PW7lW1bl0XfdeHUqQUpu/QMgK8z0kKZ6hxn3ZA=; h=From:Subject:Date:To:References:From:Subject:Reply-To; b=h0lF/dljtRFSpaiYyF07Ir/LS+/GITMy4agsQm/+4zLLkRfArSCQkXhfN9PGbxiVgk9jmhmo47mrUnWVwtQGR8ykfT3IyHx7UN/z3HowseZtK6E/Liw/0eDJqjrnB88QSR0Ys/jKtJeF+3y4fu3B7vKlYnP15svv2l4ZrhCFrsS/W7Sa+XdRlR5LdJbmj114HjISt3VRDpJPJPWxdbhAH7ycg4sKNTfujIAqtcsjumfzGM82F75qLibBcquBwAyWV8wBipEgyFn26ZKgT3Ia35iVNlfU266LYfdQTa3nUHmmufAeS1sq2ZF6B8odFwtcWvVWCrCdWD/eBWVISFV+vQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693338788; bh=hbep3Ez0ApyFvEPMLbp7HdTcInQXEtP7aelGKUHj/og=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=ODOrI3hhdtJb3Y5iMJJMQi97c/musH3H88JrFef937Zgv7c01bhRBCjWWPszan25tzmc5dnDD/29k15JHsVHZulxfbPF5MkcRTpN1Ry5iPegGbs0erkbkYYLhyIr0CCFvPx0V9uVo49T5TWz48V+2R5nNZsHBWZLUnDlzs4sJco+ofqnuWDElena2qFPjEGKH4B2BdNUH/p6svkPGAn2D5ydRVg72scQBGCc1VlgwzfsiQW7smavC5LIKZ//SmQRD4cr/8gdst9uAq9/Y57Jfv+JW1B5QCtOR6YF5LNVJWyehptPzVV1qBzbHDZlZx7fMIfZBb1Kvjg5jUx3j1fBqw== X-YMail-OSG: qJajcEEVM1mi4OcTXP7eO_CWJ2OmJBUaATF3i1lUrubORt.0l0.v3F7aJHPMzgq _RUIkXsdBluw0NpQKpNRVkoaYWlL_Z.VkUr5ybJxNRZ7itFvUU9FzoxclVq31zBEhwolDmV2V58N 6MJmLiC.es3MtWMqD9UqZXMR4ujFpZVjn_sI.n7CKyOmuwC_R9v6.RCa9gR0wbRHyjKxX.eQVM0j 555otrbm6JJb8fU0uExTsF8U_ZF4Q9FuH5II0x403m.Dfz2gZrWv4Fnz03pQZoWzf9o0tB8L10mb fx8zwTrf8e_OWdARFBU.jl0J7MffV51zHRxmHpJMozQ.ZE0Xeq5Lyxt4U6pJs3bPt.AskCXbTbCm sfSi8Xij.3KDsegZ7l5Ga0415DGVkZujZGHFXVvdeqxcC6UFcZrEb4tLw1o4rVioQXgqBnTQvVvr vbFiR5iGcPm395otYEnByBTPF9BCuqi1SxrztDkZ0qGjWXL1vYij97a_yHSKtbT3gSDPU4rEesSc 58eMNMWjkGthssKICpBZMthuqFZxUWwUDSyUQTFJ.EScdp2Qu2iAXQUPzEb7sfsw1kW2b2bX_IMu jYYFil1iqsfKw0YU6rs1ywDxf8gwwgRLuwDHgq4QUqcmn87M3hCtLcG13sgBIREhRDn3D5WheEXd Z5ElybYqq_DInF55s72keE5QEXbP6GE8XU42o3RPmxC4INIsBD5dUuMS8WPl446fw.IFOM1IVnKg saPQccDO54_N88Zot7XXsAC7RZZzjalM3XvquAKiwsMZkN9KTyIqabZCj5znS4IYJDRFudd8LeP6 A9PxzzUBujm9o1X8UWxztCFu09VN1k.2E07w39sSTJwNY9nO0NfzhlK2vn3x8KQ9oQXsOsV2J7aI v3UD1vXi6Joa.i7eNHtkXYRrXR2NH03u.42Qr6RDTXjMLAUwDGa8XW08751pbJfV4iQrGR54qJ5O Ux.5_O89e.JmiyGLxY9MXMODBu2o8VZGo_SVk1zwsGupbG3byWAoDZTKvzftK3dd8bzh8Lp_Qg_Y tMs21TWRzh4LnKTy1fB.MOH.d.J78dyLAd6gBWhtVa54iDpV2ZLeCS.JK4jVhbqhSLqRzo_OyLBi bdJWHxiM2TEGipbErRKEQzfobv3OJzhfCjnOPibKo54XRn6nUwI4B73.jGgFcxFIPSagjaHGEmk_ XdFKj6A7VpeOpXZOoMV8DxlFTh5kFFSSSXR39ib2KZ.W1nllDQjN40NAYBptnTQA._RY68gA5s7Q UC_nYxlrFDBIJUrFSQz3W5MqeBzBD._dQMbze.073JgKEsU5Ex8PkZ_rpHVaS3lHsrX1zOtbkfrH 11J8dA2jQBLQxsD87Wtfff_bnxvAhQPtmElGJLwgzph_9fkikRKMJXA8M3ldL_k4e96HiQq_q6J1 MwD0lJyc3o0l0.nB_f5sl0PCLbyVlSS8IWH22EBsyp_0TKNjRspXEFHfBDnVwlVFabrYPp1HDkPP OffGM5MaEadygBg46qJPNFMT75C2OX_NBLkW7QHH5DdSeXovHQUqj6qjhonGqRA.TGp05GBrwsrk EzIoKPQtEsai9UwI0r8fAeupfm9eMU4bp3_zOsyQZYfX.98cwKD72PIjBtCHnLciZYAYAozIDLjJ OFklrirHJCUTLubxu9KXJ1w.w8JZ4nCJGAtLqnU38d1byvfdAtl1WdC_g.ECuGIPjxkU40sEINS3 h2EFv3MSz1N0WaEQaRL5Ssjqbxd_Vmkh_CRqt54xohyMcdSJs8q52iymBg2jxUIIJkf.06Pi3_Kv bRBgjq34i7UG_7i4CD7IuOaqLvhEOzi7zi9G54AYCRSl.uGFttG.HEX.Dz7f1jzZRDAOmoJqthPA mAafwLTF_m9hRfJa7_cwK.l2yTnomyC8ff6MPSxAtUqUZ7Hb1nI.hOxP.5tPlfY855Mhq9TvT8u. .Cf0ga0dd..qpcrrn2lbPmF9ZIA4x4A_XmNTR1REUiSBhuhAC4Ib_9Rax4cQ4vYG.fgjPv3H5nfL nJPTUP1iAh_KPWGd0hIqV0W03etnE8yIz2D7ETLAfyG_BzeWH1j8KsDZiOqRrSCfwlEIgfw4am.j CZ9ADKr4ClIbWV3XSKR4i6vm9r.taqv1tBUVW8KUFb8M22LdjGdaz1D3HoU0V3YyGFdg3f4x4QFR 4lr3816osZyy1axt98WsyjQfa6R6WBFfNqC7PaB6tacH8ygPWwUGr4AxcFGtvYfhDd.k_NW0QCv3 HqJveW54O4j4YbsMD5dRsUuPmpAhUFL3TudvxjwzkWN7F8ba.6XFNAAeCvN77ll5eaLggrOTpIwx xs_A- X-Sonic-MF: X-Sonic-ID: d4a1d5a3-d98d-44bb-bb4f-42863b55405e Received: from sonic.gate.mail.ne1.yahoo.com by sonic307.consmr.mail.gq1.yahoo.com with HTTP; Tue, 29 Aug 2023 19:53:08 +0000 Received: by hermes--production-bf1-865889d799-x5klk (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 2a9c934adbeabfc3a13b1a25a07266bb; Tue, 29 Aug 2023 19:53:04 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: RE: Very slow scp performance comparing to Linux Message-Id: <87B95CDA-1812-44B8-9356-46631DEA9428@yahoo.com> Date: Tue, 29 Aug 2023 12:52:52 -0700 To: weh@microsoft.com, FreeBSD Hackers X-Mailer: Apple Mail (2.3731.700.6) References: <87B95CDA-1812-44B8-9356-46631DEA9428.ref@yahoo.com> X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.31:from]; BLOCKLISTDE_FAIL(0.00)[98.137.64.31:server fail]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.31:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RZylT46RMz4cSZ Wei Hu wrote on Date: Tue, 29 Aug 2023 12:55:35 UTC : > Thanks for the update. Seems the numbers are the same on zfs and ufs. = That's=20 > good to know.=20 >=20 > Yes, your numbers on ARM64 are better than mine on Intel. However, my = original > intention was to find out why scp on Linux is performing much better = than FreeBSD > under the same hardware env.=20 >=20 > Is it possible to try Linux in your ARM64 setting? I am using Ubuntu = 22.04 on ext4=20 > file system. I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023 to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the Hyper-V UEFI reports for the disk created: 1. SCSI Disk 0,0 The boot loader did not load an operating system. (It then reports the network adapter attempt found no boot image, but that is expected.) That leaves me wondering if Hyper-V Quick Create established a VM file holding Intel/AMD material despite the aarch64 context. Establishing a Ubuntu more directly is not familiar and will have to be a background activity and, so, likely will not be timely. If I did any experiments outside Hyper-V (native booting), they would be with slower USB3 SSD media than I use for FreeBSD. I did notice that Hyper-V Quick Create did not create a fixed sized disk but a dynamic sized one. That is different than what I did for FreeBSD. Also, it was not obvious if you were after aarch64 Hyper-V testing vs. native-boot testing vs. both. So I may have gone the wrong direction from the start. It is possible that I'd find establishing a native-boot easier and then be able to have a VM file created from the media, more like what I did with FreeBSD. The Ubuntu activity likely would not be analogous to the FreeBSD builds having -mcpu=3D optimization used. Back to $work. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Wed Aug 30 08:17:34 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbHGp0zTFz4rrvG for ; Wed, 30 Aug 2023 08:17:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-21.consmr.mail.gq1.yahoo.com (sonic301-21.consmr.mail.gq1.yahoo.com [98.137.64.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbHGm5hX1z3GMt for ; Wed, 30 Aug 2023 08:17:52 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=b57mZMNG; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693383469; bh=vmDbPCdJ5cL5EoNEgS19dOPb1QT1xl72mc7GW3ScXd8=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=b57mZMNGDDxjmwG1K0rIvpL3MQQFff4LxaDrPzqbQtHnrHE+KhNKaXc1kfKTtXqq1yZUq1WCG91GLXf4LTLI2xby3Wir9uV3NdX2Yt/JcD6w/rw1LinI6ppNcaniW1kMoqEqKP4M+tNUOQozJJBi4SktCsXHr59c0wYk14hXYa+wEMxff/dU/h525NLxk8bMxbAVowADAbX0jbA1qKf2zzfY7D1BU1Ojyw0PdQgWWaTfGCLsdFYwNOldES8RfnSLOVxHHq5+Z7bsBizXc8qkESYAaUS+9nQdjHbhyvPjQIunpWRkIpmpDPPHLlPjHfyQXZfi6nWHKeaUck5zx1gMnQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693383469; bh=Fz1RKUSps2/az0ca9rCcunKb8A7e4e/l8VkMiziFlaE=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=LlArUrp4kq2EjTQde8kAN3nrgsgN88++fVTWtoP0JwF7dVRAeRFg2iqyuumG73xeCIyH70yay45IuhGm3cPoYPAF3rxdri/1IhIOKBguCaBSYZji1o8r6RzBFsbTsLRi0+1cRKiB5Poojd0DUP3DB+2cI33NF2uAo2PQvFldZal+NPFXVxnWcCBlY1Je7uFtBm3ZdW1w0mNNyJoFwDnPH8gjEEwJ8iD/blW4srT8rvgoa900bQjmQyAXZMmGzYDkbA/OGkVeVs/4cfXsmotxq4qWx/bqsberdw84P+dZQcsdz2rFBLkV5mPPryTe/pW0O0iMIDW3NXLNRaAGmITruA== X-YMail-OSG: jz6v3kUVM1lLbNC7HFUjAlkzW3aRSW0PYQAukdQPjjZmTclzyuQRJNzgjcAnU9C oFdpFFFToCe8W3DhLco5sWBT0MnmGrS.kUwlguKMVyHxAfbbJgiF9fBtdh1M4mlxCGZ4WPVCRaj8 epgpQXFh8uv.c8EQLzjtlkhvfJvn_96lvLheRqnKqNv8L8TXcm5qcs9iS7niIVlC2rOQbHojK2Xd TQREqfohVsm2DxzeviJ_QXSnRCZ82zgSbqlS3bkGNBChtx.P1spt2g6J21UZOjNLbnnlscXzUwmE QJcESkr745B3_Mnm7Z5uExGEvvofszyxcyDiSs9AoRE7ondKreYx_0mdo2nm1n3hguZh_yQFeU2M usiRC1aW7Zyxo3oSkC_Rix454IvzcfjyGIITNXIsb69wE4QPx9vMslLj1Qmne4ZJJkOy0mXnqpWS xIwo6iB7DaH6JWqv.hfWkH2K6hjc6z4QxsWus1Ltd_Jc4sU8vkldt8yDOEHyoFvHB3C9sA6OkjiQ 3zc9tT.6vgeZrD5YnzxiK71Tvha9KjpiJfAfGnyJzA4KiMgkOYWfIu5K1gPEK8ZW2fynD_5BKpCa 0XYIGpiaEMVPPtn2ZXJI9SKyD.pW1HCD3LDAGy2pYDED5PzWhA4lMB3c7EMZQP2mTCTfDzEUZTgJ jcJhjQSE2Mtf1pusqPl3I5ePaYuTf8WFoZR80Oi0djQWhX9adZf4LvMOQO9paaQ.4BemlVqDikiJ d0L1REq0OA1RABNc1Y1UtjVxYArUIbyn4GGRNrPsYDmVXtTWI1iTcMX0hOGAIDK3pGo2PoAyfO.t psJdb_TElM7Sq_ceFOhCqgscG_AgMoT11UdmkS8YjQJzFo6hDXlJtTww7b5ZkesFgQ6OX05lBa.i R.ybjLQDDH5rTvHSRxevg8gwcRK4KtTlQc.NlK5AmWXHFKjbcyede5pCPlsRI6MjIoMExfCjhDM8 SGlQUQCcXwMm2l6dGTwRtWbzJBzMG7N9TSV_Z0YdkaR_BR79hvxXTOHoNXDbeonesOY1K0uushnx _y9y4MwNImbuJnwG8_0PjytAPEZRRg69pOdmdkCJK.2GZxqKr1QNIHgWx_rhDxNO_PP1pVZhAZN. ubD661ul3qN2Fl0zFzC7p8Q3ZlD8rcsgy0GERT37LOuvFK7BywRM1OfLjo6JMl77MKH2TOuSDrbr Hm1v6ovGUAc4fCDlgT5wY1PCx4TSd.ZqTc1jRsT5qqt4BM45wqXvrTp3sxHxxx.ve_F3LH2jAYVS j8zxNby7SBEk_gcJwosRS_MLvfhyZEex7W_WvkrstW3YQLVvo1w1_DElDmkC2yCIZD92ts2SbKrT gm1liNzOq.dOFeYF7BzG4LJ76hb0KcT0xfunCSLEBzd_skaDPOWO5UjurM6yB7yw7IDynRoVdn7X 3BQtJmTTcxbyEbYJ.DJcvDtQSJA32y5dwmF4Js1oqh3xc_q5_4CZ1hDXndEk3n1FnjiAt6QruEj_ jWb0Q.wnk._LeDEG1uWYsjYurIsLNh7qoxkBd1PNAxhNfwKDJq6XmPq6tbNjCTpaxEeIddN3T0oN JbsWs2JGNyUN6oYDmZavFN_LUeZOMnSswGZE8QQrequ_11ENbn4pNTOVL4Q9yuzRUspjIPn7ARwT m4BGRdUuWViCtwDyiCLyFVu5hjXvFqeG5nfjp5OAJ3tZH94se4JKRhb3cN4QDK2A44JP.b1mM9Fm GfOSqsjm7xyEbfNJyXgYQA1pWmki26WNZLzT6JwFWgog.RT8_BiIR1IlHMID25KXZcIhNMiQtvSO bGVFOW4kEqddBDUh..OgEoSkOaYDvSYiVIXb9IPgF36E6yJTujtHer585RmZqId2LS_pDosMEmni 7BqUajoFNAeahDxqjkMgRVYQ1uzo2M8J_IEcwsJF8.MQkfVbGb0cALOMiF64fodqVMZ0erHAO2iI XU.AVoYSofX.n5x6ofaOOiuNPXhO81M5IPuSfp7uFsn8VybizrjVTSrxsWzOAQaXe2doyhusb5lP LfoQNmjq3RCLifyskVyUecaOKUImrIXWMhIiaJLh9rn.kGJvW7PPjMRfNGp1qBDVjElrGMEq.6em szPUftnSo_bWL3yJcLZ6NrDvjCkb8c8kPEByOQ7PMqmrPDX4qAM5Wi8ow7korU0oA.pZ.nXT4wNm BdiEgsuwzlJkLgIjCNmhpdgxy9VflIOBKgPLRS8JhaUsQfFPEuvOK.UDV5xypXsSZW5buqEmTf0J rH1ttRbZkI40Bgq_RTYgvAOXmVDCycrU7ddgkFEoSkG8_LEVJR.lVW10RaQIdQYPMBBn9HfEr2QA - X-Sonic-MF: X-Sonic-ID: c1e29675-80ae-4261-877a-01be25f38a9d Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Wed, 30 Aug 2023 08:17:49 +0000 Received: by hermes--production-gq1-6b7c87dcf5-9dzwt (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 002dc3c216f619f1fefa6b8a120b1603; Wed, 30 Aug 2023 08:17:44 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux Date: Wed, 30 Aug 2023 01:17:34 -0700 References: <87B95CDA-1812-44B8-9356-46631DEA9428@yahoo.com> To: weh@microsoft.com, FreeBSD Hackers In-Reply-To: <87B95CDA-1812-44B8-9356-46631DEA9428@yahoo.com> Message-Id: X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.987]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.147:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.147:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RbHGm5hX1z3GMt On Aug 29, 2023, at 12:52, Mark Millard wrote: > Wei Hu wrote on > Date: Tue, 29 Aug 2023 12:55:35 UTC : >=20 >> Thanks for the update. Seems the numbers are the same on zfs and ufs. = That's=20 >> good to know.=20 >>=20 >> Yes, your numbers on ARM64 are better than mine on Intel. However, my = original >> intention was to find out why scp on Linux is performing much better = than FreeBSD >> under the same hardware env.=20 >>=20 >> Is it possible to try Linux in your ARM64 setting? I am using Ubuntu = 22.04 on ext4=20 >> file system. >=20 >=20 > I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023 > to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the > Hyper-V UEFI reports for the disk created: >=20 > 1. SCSI Disk 0,0 > The boot loader did not load an operating system. >=20 > (It then reports the network adapter attempt found no > boot image, but that is expected.) >=20 > That leaves me wondering if Hyper-V Quick Create > established a VM file holding Intel/AMD material > despite the aarch64 context. >=20 > Establishing a Ubuntu more directly is not familiar and > will have to be a background activity and, so, likely > will not be timely. If I did any experiments outside > Hyper-V (native booting), they would be with slower > USB3 SSD media than I use for FreeBSD. >=20 > I did notice that Hyper-V Quick Create did not create > a fixed sized disk but a dynamic sized one. That is > different than what I did for FreeBSD. >=20 > Also, it was not obvious if you were after aarch64 > Hyper-V testing vs. native-boot testing vs. both. So > I may have gone the wrong direction from the start. > It is possible that I'd find establishing a native-boot > easier and then be able to have a VM file created from > the media, more like what I did with FreeBSD. >=20 > The Ubuntu activity likely would not be analogous to > the FreeBSD builds having -mcpu=3D optimization used. >=20 > Back to $work. >=20 I found a sequence of UI operations that worked for installing Ubuntu server 22.04.3 into Hyper-V in Windows 11 Pro on the Windows Dev Kit 2023 via use of a downloaded *.iso . The kernel that results predates 6.0: $ uname -ap Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 UTC = 2023 aarch64 aarch64 aarch64 GNU/Linux Using my usual rule of rebooting before the first scp: $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 431.3MB/s 00:11=20 $ rm FreeBSD-14-TEST.img $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 482.2MB/s 00:10 Definitely faster than the FreeBSD results that I reported earlier, including faster than the ThreadRipper 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec). I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual hard disk under Hyper-V. For reference: $ lsblk -f NAME FSTYPE FSVER LABEL UUID FSAVAIL = FSUSE% MOUNTPOINTS loop0 squashfs 4.0 0 = 100% /snap/core20/1977 loop1 squashfs 4.0 0 = 100% /snap/lxd/24326 loop2 squashfs 4.0 0 = 100% /snap/snapd/19459 sda = =E2=94=9C=E2=94=80sda1 vfat FAT32 F7E9-1344 = 1G 1% /boot/efi =E2=94=94=E2=94=80sda2 ext4 1.0 = 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% / An experiment would be to have a small amount if RAM relative the file size. That would force it to actually write to media for some part of the file copy. So using 1024 MiByte of RAM assigned in Hyper-V: $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 407.5MB/s 00:12 $ rm FreeBSD-14-TEST.img $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 404.7MB/s 00:12 Still definitely faster than the FreeBSD results that I reported earlier, including faster than the ThreadRipper 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec). =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Wed Aug 30 08:22:36 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbHNW0zq9z4rsNm for ; Wed, 30 Aug 2023 08:22:51 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-21.consmr.mail.gq1.yahoo.com (sonic301-21.consmr.mail.gq1.yahoo.com [98.137.64.147]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbHNV3R5dz3Hwy for ; Wed, 30 Aug 2023 08:22:50 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="Jc1UY/AX"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.64.147 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693383769; bh=6LMK4eUS+A2WGKdWjW8NJrj3gjabpFeKme8yCYAxOU4=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=Jc1UY/AXN5ZrI5Dxw7JNdXDkQUouLZ8AYDybD13zUh0zajIC5WoCa3q5lUUaPTtqpcHJfAJvYeNSMKIrunH+chkcurqfhpI4AepTSxqnQ4+4preIQAzcg8j9dbYKkOShtuE/LrP3evQGcYiDBgNEmN0MgzQFjYMmsrl1/KWElzv1u22Frj0M1F0MghJE+R3AdrzhowhZp6Nd/wWOw/2nas6bkUzP0SdCVGh71CpbcYDoTytmXVmnoZrw10QYMoDdpn7EUCluGViccZ85mHZQeJ6eHDoBKglXt5G7ZZpfw7QXpBhWIF2AOZu2TmhyIEjHxcHYkKwXw4RB6h1wtB4Z3Q== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693383769; bh=7+H2apl/s7WMBGGG5bZX5m0VpQYosQkH2D/f+UuHeIV=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=nAFjCpIPMntmxdEDtq43sKorShQAimLo99X4rgoOrIe23h1mp61Demp1Kd3HFXh8X5HlQ2Il/lv7HJYv8iiZnVLcNQKWo2shdoNb5WYnESkUIM13WHjD8TIsGWiVxhfjBBe8imse73mL768SpgLYI9k6amwHelVoNvKxbUwTL8vy2HlL3w09XyIarCpRDMhKLskP1LRulfjH63SULXcXi53w8RYkOCTJbOKLDCBnCPTguuYG2VGvdFFp5WJGxjdGMLIdupiEFGT2Z4Snh/i5xoQHZ048kZ9MwAXOh8YVoYz8Cp4aGNvHav6niGMm3rDwgXCF0TvCbOoaYUkpTBrqbQ== X-YMail-OSG: AGItMBwVM1lw34wVd5a7GJ5mtZoczCZ_6BuFgiBTvy0PyN.2DT0kOaYdmS5QqqO yc9OJRH2AYQ6SGkg6KqKXPZL4bsFTVXzw.nZf8U5qaEJswHT4J5GpniWuzOoOnAR07zdRHOE5EFE GyCKkTUkX8mkzrX0hw4iLkMfx88AMl0EczHLj_scb03oxVDdT1RRiLFx16ooLfY8Fn_UmDDtuMaD TKYogXbhhssPcNDpRs1ivvmzE2rMG7C0NXcGjNXYPlTvgCdxLcpxsSVkdkyTs.7XcsSg1d3kJLqx jz_8F7ThF1DY6WxgCtYwQbFIOWec.7RoojgmVin0aoNvzSAthZefR7VybIg4R85coy3MzwW.Ov8j NXNigsBoym4TeKz1Are.ZcHYkNxE213RcNZbqk0xCL.Xc9q3iWWK.cQVvtHYPQ1enO1RZG9vpFqh aaNJP.FL_nI8SV9zSNJJ2Gfr7I3I8eeo21WJnSCP6oCtxCG.LsENe20tXL93aYaAgBPFyxhJfjXY z5N2QwOck8ZDnnaAzyKGHZyIm95ov5SAPg7memPguxGTjFD1mn9LFbQKvnbYVwQMafa2mPry2VvZ lpuneFnmQRbQbrwpmdZEyDxYgmJQrr_ANW3oCYt1LrNcYThAlWpFL1ucmM9suyoP3xxvPQxax82N 57uTq1jxmENU2IrX4BUkMm3gVugJwrT1iImEpgBND1HK_sheahTAYAXPzn6hh1h5RqOvSHeSmxpR 0Je4tME8QyOPm05aEqYKHPlRVexRNGeF.15gxti3_Rh.Oq7QAyc1f.z22Vk2i8PK34MOMP37dLY. _a.Ihz3orDjlYq8BYI.vvtOV1P3ogDNR1uk22hrYD2n85HSci5p7ICEFxH_c3q85i334LRWvgP8u VX6Lr4A4Rss8nDTn9c8D9qHVeO0MuOjPJq.823Wp5_VHljUvaoA40wRfLogjq.3LL52tjO53wd5m V9zMAA5rKyIMGrNYzFy6q59TuWd.zzHlQejGndhDTAfzlmRmAdI9bH9KcoeCd1958aAB_Ag2ZBso Tyky3fbEKzH2Z3C7kctn.tcWknnSPAqt89ujdktoVvX7rClFNbbh6cUZ7yTN34U_kkt2cdO_iDW3 .17iyERzg53IqVEsQdfO2Dz3ixuw3h97cpSfU3BkG8mPBY84UdH9mPfTP0wJg12e07zcP0TgpK9E 0_2Pb2zoHcMdJEy3Fjsrphvk5UU0aA9Df5rtUZuYo29q8zAqBE9RjZnoppAuqvM2YINXyo.8D_qh BcV0JX69BMl8dBy4AgiFqxccZzork5Ns3akO2XHl9N4CcYUCdcxAV891NoOaBkLFkLqPS1LcDINr KoQb3HQYVjuqtS1y8S4oXUW7ScWlINlQXr.9FX_FwBzwKfUWA5W2YFwxv31gO.iEhL9eFBq9Uz7S oCyENQHawNsfBeiLGACUK4.Q9g7h7cfL_9qD13tCnFxduURm1YojRwiy3vV1SVr5qvPFX79ZKaJo dx7kxHhSWKR5z_R7gQ_O8hS2HOzJDgyg6Rrpudz.VA02nt4d5SP9MtuG.VPAzocWBmTLuqpifeHx 0OV2JsDwTQpcVL6X07dGFUdhQH7bBh8p8nCizC8a7.3x0W5Ksy9pUTwkjC79crA4LP9Hk6RVjS7u wDCo.Wc5BeiKfHzv0kt1bvCQ7LeTMqj2HuYGjWX.RIdCt.KiHJJ8vrtLwNv68E.quuRQU3Rgscw5 nuyQZUgfrloDblQI3JT3EtYlEs2hOZS04nKbJ7SZsm6XLZDYghj1.YjcuLYZLTTsRQAqdwdCB63d .w8axkhlfOXe_PQhFStUXpKHiUDy1LS6PJTUt7b.eUQ07Bb_g.KMOJ.n4nUyhXA2XC91UJfxFmg6 aGA89JmlMTVrGZNTKuRumAkKWSJ6aFgOFPNRbczUVZWC9Nh68TcAPCrj8mwmlB6WuJPtCBkhiVWZ _G9D7nzBBIjjm5a1llg2KUWWs8y6rLhM1kpiPPf2._mH4YCaDfaNJRDevSa7pEx41ltBTd4YqmhS jsVzHxKddEwhz9qZnxUAIWq8VLRX9GkY7whA6UC_K8IQlhIqWRM8RjiOdXzLE4eVDhJ6e35fJwSC do6oj92X2KD2tjPQVEPKEvfwM7LBXmNhDEe45fce_kU1nDtyVlU8002tjDrxCbd6kBj00Od6f.ML rJNRTJuP0wiI7GbhVGBe1OlCv27ZZyNrlVIaUKsJhpF5a2ELxMApnakCAWoKWoeHWn9Tp3Je4W4s cASTcEOBu3aIDQe2.9qrGlc7fn6824BVPvgt69RRpy7XUaaIDjJsJHpS2Encv9EEKGsSZy.W0Kuk - X-Sonic-MF: X-Sonic-ID: c61a24ed-ac8c-487d-80af-5ff67c0eb7c5 Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Wed, 30 Aug 2023 08:22:49 +0000 Received: by hermes--production-ne1-7b767b77cc-84nnk (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID d8aa92076a43b9c0b151057159f4f232; Wed, 30 Aug 2023 08:22:47 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux Date: Wed, 30 Aug 2023 01:22:36 -0700 References: <87B95CDA-1812-44B8-9356-46631DEA9428@yahoo.com> To: weh@microsoft.com, FreeBSD Hackers In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.993]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.64.147:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.64.147:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RbHNV3R5dz3Hwy On Aug 30, 2023, at 01:17, Mark Millard wrote: > On Aug 29, 2023, at 12:52, Mark Millard wrote: >=20 >> Wei Hu wrote on >> Date: Tue, 29 Aug 2023 12:55:35 UTC : >>=20 >>> Thanks for the update. Seems the numbers are the same on zfs and = ufs. That's=20 >>> good to know.=20 >>>=20 >>> Yes, your numbers on ARM64 are better than mine on Intel. However, = my original >>> intention was to find out why scp on Linux is performing much better = than FreeBSD >>> under the same hardware env.=20 >>>=20 >>> Is it possible to try Linux in your ARM64 setting? I am using Ubuntu = 22.04 on ext4=20 >>> file system. >>=20 >>=20 >> I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023 >> to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the >> Hyper-V UEFI reports for the disk created: >>=20 >> 1. SCSI Disk 0,0 >> The boot loader did not load an operating system. >>=20 >> (It then reports the network adapter attempt found no >> boot image, but that is expected.) >>=20 >> That leaves me wondering if Hyper-V Quick Create >> established a VM file holding Intel/AMD material >> despite the aarch64 context. >>=20 >> Establishing a Ubuntu more directly is not familiar and >> will have to be a background activity and, so, likely >> will not be timely. If I did any experiments outside >> Hyper-V (native booting), they would be with slower >> USB3 SSD media than I use for FreeBSD. >>=20 >> I did notice that Hyper-V Quick Create did not create >> a fixed sized disk but a dynamic sized one. That is >> different than what I did for FreeBSD. >>=20 >> Also, it was not obvious if you were after aarch64 >> Hyper-V testing vs. native-boot testing vs. both. So >> I may have gone the wrong direction from the start. >> It is possible that I'd find establishing a native-boot >> easier and then be able to have a VM file created from >> the media, more like what I did with FreeBSD. >>=20 >> The Ubuntu activity likely would not be analogous to >> the FreeBSD builds having -mcpu=3D optimization used. >>=20 >> Back to $work. >>=20 >=20 > I found a sequence of UI operations that worked for > installing Ubuntu server 22.04.3 into Hyper-V in > Windows 11 Pro on the Windows Dev Kit 2023 via > use of a downloaded *.iso . >=20 > The kernel that results predates 6.0: >=20 > $ uname -ap > Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 = UTC 2023 aarch64 aarch64 aarch64 GNU/Linux >=20 > Using my usual rule of rebooting before the first scp: >=20 > $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 431.3MB/s 00:11=20 >=20 > $ rm FreeBSD-14-TEST.img > $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 482.2MB/s 00:10 >=20 > Definitely faster than the FreeBSD results that I reported > earlier, including faster than the ThreadRipper 1950X with > Optane in a PCIe slot (more like 300 MiBytes/sec). >=20 > I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual hard > disk under Hyper-V. >=20 > For reference: >=20 > $ lsblk -f > NAME FSTYPE FSVER LABEL UUID = FSAVAIL FSUSE% MOUNTPOINTS > loop0 squashfs 4.0 = 0 100% /snap/core20/1977 > loop1 squashfs 4.0 = 0 100% /snap/lxd/24326 > loop2 squashfs 4.0 = 0 100% /snap/snapd/19459 > sda = =E2=94=9C=E2=94=80sda1 vfat FAT32 F7E9-1344 = 1G 1% /boot/efi > =E2=94=94=E2=94=80sda2 ext4 1.0 = 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% / >=20 >=20 >=20 > An experiment would be to have a small amount if RAM relative > the file size. That would force it to actually write to media > for some part of the file copy. The wording was poor: "force it" here is just from the Ubuntu viewpoint. I make no claim to know if Hyper-V is actually writing the material out to media at the time vs. later. > So using 1024 MiByte of RAM assigned in Hyper-V: >=20 > $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 407.5MB/s 00:12 >=20 > $ rm FreeBSD-14-TEST.img > $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 404.7MB/s 00:12 >=20 > Still definitely faster than the FreeBSD results that I > reported earlier, including faster than the ThreadRipper > 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec). =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Wed Aug 30 08:49:54 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbJ04499Jz4rtTF for ; Wed, 30 Aug 2023 08:50:12 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic306-19.consmr.mail.gq1.yahoo.com (sonic306-19.consmr.mail.gq1.yahoo.com [98.137.68.82]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbJ033mxWz3LPV for ; Wed, 30 Aug 2023 08:50:11 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=MFVGNVE1; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.68.82 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693385409; bh=zk3RAs79DTPdy8bHcuEAu1SQuwV1Ljfgf/9f7QxrcBw=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=MFVGNVE18UPSYXhwNlVdMJl0cHVXuUftDLCE/1flWwYFsPtrid0slDkKzpw/u6nKLv0zT3TVpgaBp02hpyWu5PyM3nbxG5fs1vwUshxpbyiW0oWNyOirlCxgi1ui1mg3uhtsIY18jLXh/VKL576rtfUmgmmNSet6ES084q2+O7e4/TLMZeYC+P8NjVrWjJ/7vVs4GeTnxx88aRTFfflC7SYdHPl3DUK+/NCCqts4xeRADFCK8dC0DTy39uQ4RjvE61TapSERTMHDNaZpz8KB36x8tehU5E15LJx3g/Y8ODc7NQJ4uVjmtgckAnvB+AL3cdUv2swjBJFVNQXAtHt+9g== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693385409; bh=P5hIBoSBKIddN2V11Gd7Xs9TNWW4/1ulrXKKrF7l+xt=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=Fbn5qN+6KjxjeEyoXywu9aGn3T/p4ZsWAY52Caqp1EPtaoSqeCZUvrT3HEMI3LEyKCZt29QE4BlSnZdicT3qpAFnRHSHB78MMnNI3ihenNkKDYZ4/KfbDO/mHZFuLOtvxiLc+nksHgbyBDNIsnmz+PscoL9GY3e2Zf21p3BHt+kZVw0xWCi1IZzgnNVw88ijkq5e05wP5zBd3z57sCO0XOJwgeVtQPrgbofaqGDTZpCPIQOh/QefIsAlQZ8SqZ4dX7sexzi4Y4OiTwN81Fi3oim5Ih8RQe003z4M2zcfZllt96c15BQyzAsQGe554cTECRfvUsG0doPLMftqgg5c8A== X-YMail-OSG: Ffd2lCIVM1nqcsBdg_pklrGp1wszUo0CmggcSA4ukafBEiyyvw7PVIZnw7oGs.T ViwwYH.02VFOWYRw9DsZABCi9xKFlrGWcyX7bWn.INknR92KN6lGrqchEdqeAT1VNPlxeDG0Ng5m NmUaVYUIDwJ2uXJcNTcS_KEPHRgw79A7TLUbPHCEd1c6pCIX7mRB0vdFGbrHdugpU4Kawymn.aUn 098dic1.IkCo1hQ7bfLrr0HEQdmjC2QrwSs4mwaRdjB1Ez5Ujww_bIsXzGUDHHGQ.NLCgB_5kn7S f5qF5o7XqOp7w5HilEWyQzmUFb5B9_7LlnqzW3B600eYWlPJpKV5l5R9hc35b10t10pHtqmBB_nu 38cf884OuXTqYdAAaCbv7YqUzwvx1qR5.rtiWa23Ve85TlTMjeyOFhZM76LJolSe8J9648WaMk1s .KslRXGfJTIY2ZzsX411qoxZs6sIw08G9wLoAB5zowOlqaueP_sKYBpmKazRP5nthmVSMTUea9eU TCpNtDqg0od9haUjulgpAaq5C_i6qXkEINlndF92fdVD1f1JUIMLe9TdlDq7UU1IkdYvzwyMpnAQ zY57YobqqWYfhyJ9_I6ZKlHALpccvQ5wKbACmx9xQtWcMOLGhdRql9TIo0DCNhpQj_vT7hH2pr3f x12T7ree10RGBNIxzwhbbtVDT8gHiK8_340KYHfVQJOiDaBVtzWOFnrCxmrsdvoLltlxn_glQDP1 oiWpQrYHxOTqBQ4uRhDy9JuKGN5wpqkx59TipGEF8IzKx4THSiMsk61SCtQ4JmiEuzjpgZA.XBLl lcRP08PUdsR5_PFq.DZoIwhG0jAEPAGr6uuDsYe4H6NMXCouLG7pdpf_F9PR7jP3dXy5.I8ev3JU vc78PgoBk1xXhBnCjpB6cNKyhgAuyRpAH4cS_09uMNHs.xOu5dSEYT7CEr1GjDmJYczNoDWIO7kc XE1Fw9H6WBpftUuPyZQm9f7YEiaX.X3qg9VHPv5y.H92w6Zx0V2n.1MN5oyGo4x.LHyKMS.0m5z0 6hZm_QjIJdtikA.2v1o9SCy.YErHyQ64Ewco2oy3AJkumbZGqG9jR3mOFxmK79KS1y_Pk.i8wQwY UkgtEa6PtznDBnGI5pJP_Oz3kfxMzvpqiJXLnuLYQK7iTxGnTt5G4v8.41vFPANzl37moVjQ.57t FHPWGymna_WUSBzackCeL1CErux1bNpWOCCaKQspw__Gf9tyfUjB.AfBHbNwXPHG5lzGva7rEpYj LAHncrqqn3SDyhUZlRMzjIBVHwAejydQo6Em7W1ossw09vjpOfpxtMUc30e3r0BdlbKqJ4PaIyL8 pm2jU07t7l3S9rXfKMSJCgqoUzRp6L0MZXiA4wThnkaAFLlZDUVplDATPqZyUGdAyHTkE75WK1Ek iyTt4xWE2F7twBrndVQfX29KJCB91Y3WrkmxCM0AGV3duCGXyJtuE_wqBkGJa76Czjly_gQnyCwN XIOSuSD6u_9i4.RDPBj1bVq9Bdxs4wqKZHvzJv_irQeNtV.d7BhFP_lJjfRQ7hvvrG_GJbnqgqe. ZKBESwjLkzmmsIl1midI3iE9YOpajslOXOfmMyQAX7AQ0dulNGvaymmJpawKQ1HpJ71HgvRYLTIU mKdRHDZ4Av5.P0FDJSXVLI_3YZSZlaoY6GCOXFPaTAemrH8iuYNoOfL84ujHxHvIqnwPu8Hy_r8G yvI4r86tBODsf2MSXXxc1ncbD5gUSTnvmCHpVTeyJ6fkbJEJP9UWsYth2xFnMUU6IYKLVVjQ6JHb wPWVE9U_zIpfsNOdzm2m556Yxf4F4RUA_rNINBZ6AkeYXzp1S34UmNElyzu2_r_ic_RPSJiFcd.0 JjpYvd3BJ6TGR6TFYH4ALaaYKQ015sMm6iwc6YpOBdzJ_ub_Q7qHYa2AWGwuomnQ8f47s0gdUG0L N.LFozcXh09XeQ.sA4x0FSNAWoP3WpORYTJDtonEikc206heVY0aLVbCsvY3eZR2k8l9q.qJvR.o UEJjDtfmOgUHiETZd3pOsxfPuZqOXUTp2u9QX5ko6sc8BDyO7InjDR7yCPKcjPbJXjYlB5UsEHxd aZ1HNMN5M9iWchwpRqTRT0Pt.Lrsf6SeHS067BT.ccArB5LFfAmDk0BC27J_TeMes6CxLgmJyVVF Fujhb084tmePO0uFwZSm2xa4odlPriEEEpQnfbbMZTgb1UI5bidKXAdGxVqtgCWJf0lYHxz4bvI8 HpkA2WqB.X7gTqCQekM6x.OFi9Qt87P9wfhSYycZIjq7q34gO_uvyYCWn15VwvhCxVirXhQFnUBU - X-Sonic-MF: X-Sonic-ID: cce7fe73-194f-4c38-af10-3204c3d84a2a Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.gq1.yahoo.com with HTTP; Wed, 30 Aug 2023 08:50:09 +0000 Received: by hermes--production-gq1-6b7c87dcf5-cjxhs (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID c44cc49087e7af26cd526d0c7f457324; Wed, 30 Aug 2023 08:50:05 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux Date: Wed, 30 Aug 2023 01:49:54 -0700 References: <87B95CDA-1812-44B8-9356-46631DEA9428@yahoo.com> To: weh@microsoft.com, FreeBSD Hackers In-Reply-To: Message-Id: X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.99)[-0.994]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.68.82:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.68.82:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RbJ033mxWz3LPV On Aug 30, 2023, at 01:22, Mark Millard wrote: > On Aug 30, 2023, at 01:17, Mark Millard wrote: >=20 >> On Aug 29, 2023, at 12:52, Mark Millard wrote: >>=20 >>> Wei Hu wrote on >>> Date: Tue, 29 Aug 2023 12:55:35 UTC : >>>=20 >>>> Thanks for the update. Seems the numbers are the same on zfs and = ufs. That's=20 >>>> good to know.=20 >>>>=20 >>>> Yes, your numbers on ARM64 are better than mine on Intel. However, = my original >>>> intention was to find out why scp on Linux is performing much = better than FreeBSD >>>> under the same hardware env.=20 >>>>=20 >>>> Is it possible to try Linux in your ARM64 setting? I am using = Ubuntu 22.04 on ext4=20 >>>> file system. >>>=20 >>>=20 >>> I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023 >>> to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the >>> Hyper-V UEFI reports for the disk created: >>>=20 >>> 1. SCSI Disk 0,0 >>> The boot loader did not load an operating system. >>>=20 >>> (It then reports the network adapter attempt found no >>> boot image, but that is expected.) >>>=20 >>> That leaves me wondering if Hyper-V Quick Create >>> established a VM file holding Intel/AMD material >>> despite the aarch64 context. >>>=20 >>> Establishing a Ubuntu more directly is not familiar and >>> will have to be a background activity and, so, likely >>> will not be timely. If I did any experiments outside >>> Hyper-V (native booting), they would be with slower >>> USB3 SSD media than I use for FreeBSD. >>>=20 >>> I did notice that Hyper-V Quick Create did not create >>> a fixed sized disk but a dynamic sized one. That is >>> different than what I did for FreeBSD. >>>=20 >>> Also, it was not obvious if you were after aarch64 >>> Hyper-V testing vs. native-boot testing vs. both. So >>> I may have gone the wrong direction from the start. >>> It is possible that I'd find establishing a native-boot >>> easier and then be able to have a VM file created from >>> the media, more like what I did with FreeBSD. >>>=20 >>> The Ubuntu activity likely would not be analogous to >>> the FreeBSD builds having -mcpu=3D optimization used. >>>=20 >>> Back to $work. >>>=20 >>=20 >> I found a sequence of UI operations that worked for >> installing Ubuntu server 22.04.3 into Hyper-V in >> Windows 11 Pro on the Windows Dev Kit 2023 via >> use of a downloaded *.iso . >>=20 >> The kernel that results predates 6.0: >>=20 >> $ uname -ap >> Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 = UTC 2023 aarch64 aarch64 aarch64 GNU/Linux >>=20 >> Using my usual rule of rebooting before the first scp: >>=20 >> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >> . . . >> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 431.3MB/s 00:11=20 >>=20 >> $ rm FreeBSD-14-TEST.img >> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >> . . . >> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 482.2MB/s 00:10 >>=20 >> Definitely faster than the FreeBSD results that I reported >> earlier, including faster than the ThreadRipper 1950X with >> Optane in a PCIe slot (more like 300 MiBytes/sec). >>=20 >> I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual = hard >> disk under Hyper-V. >>=20 >> For reference: >>=20 >> $ lsblk -f >> NAME FSTYPE FSVER LABEL UUID = FSAVAIL FSUSE% MOUNTPOINTS >> loop0 squashfs 4.0 = 0 100% /snap/core20/1977 >> loop1 squashfs 4.0 = 0 100% /snap/lxd/24326 >> loop2 squashfs 4.0 = 0 100% /snap/snapd/19459 >> sda = =E2=94=9C=E2=94=80sda1 vfat FAT32 F7E9-1344 = 1G 1% /boot/efi >> =E2=94=94=E2=94=80sda2 ext4 1.0 = 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% / >>=20 >>=20 >>=20 >> An experiment would be to have a small amount if RAM relative >> the file size. That would force it to actually write to media >> for some part of the file copy. >=20 > The wording was poor: "force it" here is just from the > Ubuntu viewpoint. I make no claim to know if Hyper-V > is actually writing the material out to media at the > time vs. later. >=20 >> So using 1024 MiByte of RAM assigned in Hyper-V: >>=20 >> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >> . . . >> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 407.5MB/s 00:12 >>=20 >> $ rm FreeBSD-14-TEST.img >> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >> . . . >> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 404.7MB/s 00:12 >>=20 >> Still definitely faster than the FreeBSD results that I >> reported earlier, including faster than the ThreadRipper >> 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec). One more variation in ubuntu under Hyper-V, still with 1024 MiBytes of assigned RAM: use of localhost:/dev/null $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:/dev/null . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:/dev/null . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 492.9MB/s 00:10 The matching FreeBSD examples with 24576 MiBytes of RAM assigned (ZFS = context): # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:/dev/null . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:/dev/null . . . FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 198.7MB/s 00:25 Note: At most one VM running at a time, never both in overlapping times. =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Aug 31 00:09:34 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbhPJ4hSQz4rQjC for ; Thu, 31 Aug 2023 00:09:56 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic312-25.consmr.mail.gq1.yahoo.com (sonic312-25.consmr.mail.gq1.yahoo.com [98.137.69.206]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbhPG6RB9z4kNv for ; Thu, 31 Aug 2023 00:09:54 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b="q/ZuzZ87"; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.69.206 as permitted sender) smtp.mailfrom=marklmi@yahoo.com; dmarc=pass (policy=reject) header.from=yahoo.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693440592; bh=MJllwp50E9EKQgorX6ya2RHTvsQroKdlZcbpayQtdcI=; h=From:Subject:Date:References:To:In-Reply-To:From:Subject:Reply-To; b=q/ZuzZ87z2Q5fhuiLzQ+ksYsVu4ydGosDvfTuIYKUKN58yzEsPdrAA28Izo+NHHJtnMdsmFBTHUzgtV5fSnpLmXJiB/Bwax+xgbgRXPeW1T54opabODIvkYGYthAK2RlGjYIZJOTJPq8inbZQWf6Ke4xt18essmXJZfESqRMb5RPfTz9purX/EVHzbfJKaD9wzNlfl43sdYtVGlPaEqQMSrWYXiwHZoZ7EeQIbaoFxOMWLQLOONbQJbEislwxkt+mdsTVYLNk0Y0Isf5Zw7YPpluqwUfNkmPYbxqN9n7MUWCYlpz4VjVYp+QNrisdo9LeK6JFFPPZ9OYFcS+5mRXWw== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693440592; bh=cpTRQJDTnF4yE7F29Jmq9hq3eUkjgIX+kL8Xa9JwJG2=; h=X-Sonic-MF:From:Subject:Date:To:From:Subject; b=CParrwLiJvBkoMTutBSVljrfhH5vFVCCI1YnbF5lqTo4HWKkzRukMpbqGIQJvP9WxPM3LVnKgrJkUpFnzh1Ky7mZaJMZRQXB2+0lk2X25n8+cEF1UXtW/cu+oukRXRRjTKWyS7AFj1tF/DY4UWLIxHaL2p02pj03kNS85jWJG/Zu8C5LmmrhOEtrQvtBm8N+KssqBkQLMVlLljpITv6bwtGxZ++EzaMZ3W5QgCHUnQImVGG+Jn0t2LyBa0evp7LGrmo3YrdprQMDkcsDITWZPT9rjV8HvI0VR+xo3/RbbKwIymYkNmWN09ID/2ybGP3luI9gbFBx7sQ3V6EXdR8TCw== X-YMail-OSG: RBCr75cVM1kVCL_sinPIe87PexJauemzUCDTxaXq46uxMAkgzYh11GLJ.DXMDN8 Pp5wsJ_m6puHHOS8d5We3AOKDMTbBkAZAFGSYld4GTVGNgGGffJ5PHv5yugW.lIwbzDe7K6mNKyV BqVjSOOORm3xp_iB.VgNthhIrn1kTx1085bu4DikOTiUsC5niennEH2y9.7LE2rDF8k6ouE0ON4g UNMJs38H9KVQDz7ET.dANzxsWWgT2ae0FFtvI2uzzn8k90_qfyCUMRW8.pmHm2cOgnnpDk_K_cmo qlt.3xxEO1lm2oltr9iQvlDDEutA9wSrF45P2GhiU_nO3U0_mzC0caZxYx2eWsqX6PFJG7V0gK3q 2FdhjABL2TJ47I9XHc4qkwPRm8i3OZNij22MHzIJtp2MWlEl_VjCrNP5Q_hPvlu4XFxuBtRjbkDP tF6O9b.x_nyX4ungTFFvuLwFMCyEIipqQyMhPlgW1A_BwGym6RpciCCxgJTA4bCBhwxzZotGL0Zm 8VKBkGwuxa4OcT6ufFYY3LbLVdWCPZBm0Qnnm..5TljJ3HluhbqMgxIdKpiBhg5sl3WCPqQMLM7x pyxX.Qdn.m7hlxRcNFQH5ZL9pyc.yIysxBeDHSxDVFfQDpvWnU5z.uQixj_hlrpHmquKODzEqqVG 9GEpcvWvGmZtA5kmRxDy52PvmOIjAjOQy2O.pd4qbohXebSV.VXmXvY0DoOtG3AP.xU3Ilix4ET6 17FbLfAUHYL_fURcp1JjrjQbJUn39ijFbb2iSMpNQFeZxULmfSUEhOR8JvwhZihPNghUQHQxB.WW G1N6p71uuoPL5sgzCtAe3rw0L6dQ0dCsmDA2USiHRQLo4mePwfLcscsqEfDefGNRYCHMY3ss7Ww. .gxoP73jgz2e4LO_g4VxcvgKM1l4s8WRP0ZGNBsirIZJkFRDj5I03KWjz.3ejvap4CD28ZDJ7PpE oXNqXDqZFGHKsdp1Y5YEpZ2I19BQyxicGkxrB5AjZ.mlPo_m3qksACzj3hLcv5zeSFLV9fB2SLWE biPv3JvOKGzpZRdNQPJbeeitMj6YczEfxqwAKi_5EDlJjLoTtajmAsKjVGBHVweC.FliEkXVY9j2 p_Eh3IJyUt9diJxbxchSC9Qv20kuNhOt3ATLHPzfhg22_lE7pnjCWL_VvdcIu5UGWftRrck3Kkgm .budIrHOyCCHiI1f968ZrWeWWrZH7iQZYoewMVgsFsHtr.USu2PBKe8sDMxVNmXUdfonj1a35Rfg 08AyQMUJ.HbaYC0kUCgEuuwQg9J.J3nNYC9jZBCpR.5vhpbdMSTrCuPGJCj6PWt0BPe6NaQgZjWn UvPGL52HFhOOT4TwQ3H5jvrxYdZ2nA42fIRjDsl_wdJGq3k2tOmsAcdSZrrTmklGWJB1ztMeTHao 3no8cveV4FVibTdUiKMOEr3lk0sUW_pN_9c_JiXzdyedirMis2608odB04Lml_LuNbGjFOVXBftF f7PgOgnAK0rJPQv5d39D7LE6N0Uhgr1GINW8gr6Cl0JZrrF3RsiUONpv9rCZBcOBH593XgSdUFKJ 5B5CirNMJlaVKVllZNqTU6oBbiTz8ESRuj1NI5bZvxzz0gyQtsEXqpm2kHV9P6tRUcxqan7q7C.s C3P1Hg5t79IuIZHb.kFyab_6ztGTdu1aVocb0ypPHdw2jhEekK2NU6Y_Bumev8hufj_ip33zF6eb f3Td9hzPB9qyhuBtYEiV61EnbziLfoKqKgclzH_EjEc1abw62cVUYR_8ee0XAdQOdrAdbEdb58uN Nf0tfyiSik_aj32QQ6VJ4QsbQoT4YLTEy_QOdiA.ETE7KAude80IKl0pbCf0yODwnOJQj3stZc1w neQjYBBmkSHbsxXrZb7a7p5qAzrmm986MHvoenul.u6LqsxW0KsFhhk1z5lV1YoaL.eswDarNVh1 QMHahJ10undOnc3jdXKfrU8S7TIu5vSgA_RAYxKICwjuAqt.3pwlC02vVO_hAa4LK0cCXIHKcwd1 QxS3q3BpcWRJNb5Ufb0cXAJ6H6SmAjms8naX48CqkFIOwRHjnfSP8uDyefal9E0WXcizrXiHX7nr VXmzt2JZp2j0HVX8s_jamVMoWkootyMMc.upisnSbZkrE2cLON.VAWhY1x3ZaCnhsXVlbH2Hm9BF PNT4Vo08uHImGkRcYkyy7k0EAiUOq8eW9kGX5GRWqApRPu6wprcQ5u_s05jRYOPOEYioe9Y7p.du 1YvzfjBqhCxLbbvRWAmDS0ROqwMx_GXBZCS3N.FUYGjdFnKxOedDeJkaivZ.fZb72nJqYrLXoipo - X-Sonic-MF: X-Sonic-ID: 9ccea03b-b7c5-4f51-8721-18eab296f352 Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.gq1.yahoo.com with HTTP; Thu, 31 Aug 2023 00:09:52 +0000 Received: by hermes--production-bf1-865889d799-5m62n (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID e706621ac190da6f12d857cb6413dd52; Thu, 31 Aug 2023 00:09:47 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] Date: Wed, 30 Aug 2023 17:09:34 -0700 References: <87B95CDA-1812-44B8-9356-46631DEA9428@yahoo.com> To: weh@microsoft.com, FreeBSD Hackers In-Reply-To: Message-Id: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; MV_CASE(0.50)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[98.137.69.206:from]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[98.137.69.206:from]; DKIM_TRACE(0.00)[yahoo.com:+]; FREEMAIL_FROM(0.00)[yahoo.com]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RbhPG6RB9z4kNv On Aug 30, 2023, at 01:49, Mark Millard wrote: > On Aug 30, 2023, at 01:22, Mark Millard wrote: >=20 >> On Aug 30, 2023, at 01:17, Mark Millard wrote: >>=20 >>> On Aug 29, 2023, at 12:52, Mark Millard wrote: >>>=20 >>>> Wei Hu wrote on >>>> Date: Tue, 29 Aug 2023 12:55:35 UTC : >>>>=20 >>>>> Thanks for the update. Seems the numbers are the same on zfs and = ufs. That's=20 >>>>> good to know.=20 >>>>>=20 >>>>> Yes, your numbers on ARM64 are better than mine on Intel. However, = my original >>>>> intention was to find out why scp on Linux is performing much = better than FreeBSD >>>>> under the same hardware env.=20 >>>>>=20 >>>>> Is it possible to try Linux in your ARM64 setting? I am using = Ubuntu 22.04 on ext4=20 >>>>> file system. >>>>=20 >>>>=20 >>>> I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023 >>>> to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the >>>> Hyper-V UEFI reports for the disk created: >>>>=20 >>>> 1. SCSI Disk 0,0 >>>> The boot loader did not load an operating system. >>>>=20 >>>> (It then reports the network adapter attempt found no >>>> boot image, but that is expected.) >>>>=20 >>>> That leaves me wondering if Hyper-V Quick Create >>>> established a VM file holding Intel/AMD material >>>> despite the aarch64 context. >>>>=20 >>>> Establishing a Ubuntu more directly is not familiar and >>>> will have to be a background activity and, so, likely >>>> will not be timely. If I did any experiments outside >>>> Hyper-V (native booting), they would be with slower >>>> USB3 SSD media than I use for FreeBSD. >>>>=20 >>>> I did notice that Hyper-V Quick Create did not create >>>> a fixed sized disk but a dynamic sized one. That is >>>> different than what I did for FreeBSD. >>>>=20 >>>> Also, it was not obvious if you were after aarch64 >>>> Hyper-V testing vs. native-boot testing vs. both. So >>>> I may have gone the wrong direction from the start. >>>> It is possible that I'd find establishing a native-boot >>>> easier and then be able to have a VM file created from >>>> the media, more like what I did with FreeBSD. >>>>=20 >>>> The Ubuntu activity likely would not be analogous to >>>> the FreeBSD builds having -mcpu=3D optimization used. >>>>=20 >>>> Back to $work. >>>>=20 >>>=20 >>> I found a sequence of UI operations that worked for >>> installing Ubuntu server 22.04.3 into Hyper-V in >>> Windows 11 Pro on the Windows Dev Kit 2023 via >>> use of a downloaded *.iso . >>>=20 >>> The kernel that results predates 6.0: >>>=20 >>> $ uname -ap >>> Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 = UTC 2023 aarch64 aarch64 aarch64 GNU/Linux >>>=20 >>> Using my usual rule of rebooting before the first scp: >>>=20 >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 431.3MB/s 00:11=20 >>>=20 >>> $ rm FreeBSD-14-TEST.img >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 482.2MB/s 00:10 >>>=20 >>> Definitely faster than the FreeBSD results that I reported >>> earlier, including faster than the ThreadRipper 1950X with >>> Optane in a PCIe slot (more like 300 MiBytes/sec). >>>=20 >>> I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual = hard >>> disk under Hyper-V. >>>=20 >>> For reference: >>>=20 >>> $ lsblk -f >>> NAME FSTYPE FSVER LABEL UUID = FSAVAIL FSUSE% MOUNTPOINTS >>> loop0 squashfs 4.0 = 0 100% /snap/core20/1977 >>> loop1 squashfs 4.0 = 0 100% /snap/lxd/24326 >>> loop2 squashfs 4.0 = 0 100% /snap/snapd/19459 >>> sda = =E2=94=9C=E2=94=80sda1 vfat FAT32 F7E9-1344 = 1G 1% /boot/efi >>> =E2=94=94=E2=94=80sda2 ext4 1.0 = 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% / >>>=20 >>>=20 >>>=20 >>> An experiment would be to have a small amount if RAM relative >>> the file size. That would force it to actually write to media >>> for some part of the file copy. >>=20 >> The wording was poor: "force it" here is just from the >> Ubuntu viewpoint. I make no claim to know if Hyper-V >> is actually writing the material out to media at the >> time vs. later. >>=20 >>> So using 1024 MiByte of RAM assigned in Hyper-V: >>>=20 >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 407.5MB/s 00:12 >>>=20 >>> $ rm FreeBSD-14-TEST.img >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 404.7MB/s 00:12 >>>=20 >>> Still definitely faster than the FreeBSD results that I >>> reported earlier, including faster than the ThreadRipper >>> 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec). >=20 > One more variation in ubuntu under Hyper-V, still with 1024 MiBytes > of assigned RAM: use of localhost:/dev/null >=20 > $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:/dev/null > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >=20 > $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:/dev/null > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 492.9MB/s 00:10 >=20 >=20 > The matching FreeBSD examples with 24576 MiBytes of RAM assigned (ZFS = context): >=20 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:/dev/null > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >=20 > # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:/dev/null > . . . > = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 198.7MB/s 00:25 >=20 >=20 > Note: At most one VM running at a time, never both in overlapping = times. Avoiding having a cipher involved and even localhost involved: use dd . . . FreeBSD examples for Windows Dev Kit 2023 Hyper-V context, 24576 MiByts of RAM assigned): # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1m status=3Dprogress 2512388096 bytes (2512 MB, 2396 MiB) transferred 1.046s, 2402 MB/s 5120+0 records in 5120+0 records out 5368709120 bytes transferred in 1.627071 secs (3299614770 bytes/sec) CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1k status=3Dprogress 5233509376 bytes (5234 MB, 4991 MiB) transferred 14.022s, 373 MB/s 5242880+0 records in 5242880+0 records out 5368709120 bytes transferred in 14.365142 secs (373731714 bytes/sec) CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D512 status=3Dprogress 5285410816 bytes (5285 MB, 5041 MiB) transferred 27.029s, 196 MB/s 10485760+0 records in 10485760+0 records out 5368709120 bytes transferred in 27.432570 secs (195705657 bytes/sec) Ubuntu 22.04.3 for Windows Dev Kit 2023 Hyper-V context, only 1024 MiBytes of RAM assigned: $ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1M status=3Dprogress 4003463168 bytes (4.0 GB, 3.7 GiB) copied, 2 s, 2.0 GB/s 5120+0 records in 5120+0 records out 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 2.56342 s, 2.1 GB/s $ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1K status=3Dprogress 4793865216 bytes (4.8 GB, 4.5 GiB) copied, 6 s, 799 MB/s 5242880+0 records in 5242880+0 records out 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.60403 s, 813 MB/s markmi@ubwdk23s:~$ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D512 status=3Dprogress 4800102912 bytes (4.8 GB, 4.5 GiB) copied, 9 s, 533 MB/s 10485760+0 records in 10485760+0 records out 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 9.95606 s, 539 MB/s =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Aug 31 01:45:33 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbkWt4nmjz4rWJM for ; Thu, 31 Aug 2023 01:45:46 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbkWs6qDmz4vp8 for ; Thu, 31 Aug 2023 01:45:45 +0000 (UTC) (envelope-from nonesuch@longcount.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-qt1-x82f.google.com with SMTP id d75a77b69052e-4121788397fso6959271cf.0 for ; Wed, 30 Aug 2023 18:45:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=longcount.org; s=google; t=1693446345; x=1694051145; darn=freebsd.org; h=to:in-reply-to:cc:references:message-id:date:subject:mime-version :from:content-transfer-encoding:from:to:cc:subject:date:message-id :reply-to; bh=IvgBO4RWaXoZXAYKoqKC8OdNgp3E6yA1iJ+jV21d4Io=; b=nDfdGPojdHLKN174AcK0uu+/VRZfLj44Q7lJqH20OhTg5+iqdfjEoPL2skWWovT/QD DlH+9DoVJhPuwkwiF48P38v5FZOFuQECI2f4i8Mu2J1Z2tYEy1lb3vIMWUExEYXk9ykX aiImIwa/Bm20hFlkRIY6oHKbU3NZjPmr2KkiOL1Hv89kSnKmdOSTYdtCRxeMKlMk3uls 1cPRv0YBRdqZvC3el15X4mwEfbf+z60dtkdKhGM37T4hZ35qaRIs+BrfYA4eQe93zSso +bszGGIumWJKsL8FBBmC/HBJuwS16D15XVw73HO5XQ7boHc+BHsrPKTWrC5hcIg4mX3b A90A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693446345; x=1694051145; h=to:in-reply-to:cc:references:message-id:date:subject:mime-version :from:content-transfer-encoding:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IvgBO4RWaXoZXAYKoqKC8OdNgp3E6yA1iJ+jV21d4Io=; b=XzcvhPAfOIZSj5EsKKZMHQFIXgRv7+OYOtOvpoAj0wXQ+E9aGuPLntmn4MNn5/rrun xCLniEptU7ts9t5L5RJcbMdSN8hZsTP4nx2UwCG9upII99eU16R0/WRv6cJS/ioIBIh2 E7Pv6e9t5YH27cGzBA8yj36ZnvJjqr5V6BMCwrW5BePZg1Gf0kfq7jAI62qiCa15V2Vg kKQLXH7W6K6S2ahy6vAzBPgEXz+HS5xGoTW81QiNjJjfavmZmqVe1TCnkcJwBPqUI/Oo gFzIVUPeLLe3Vbl4KywcPsGusNeTR/FqZU2DMZAef9Trx6LaPzcPXWhcfvc1AHCQ4dO7 eBBA== X-Gm-Message-State: AOJu0YylzmrEk8NZTV6O5PPU0oG2b5tk8dIvOngGl+gaez+LYDv2RJeL xh3epuSz/W/TDsUKJ1FrcHy4Zg== X-Google-Smtp-Source: AGHT+IGtnSGGptsWOa3yiwcFysli+IM5ThAZpbGV9gSezDGReldX+6jfG53kphMUXluDfh8h0NffEQ== X-Received: by 2002:ac8:4d5e:0:b0:410:8ac5:3084 with SMTP id x30-20020ac84d5e000000b004108ac53084mr1284173qtv.7.1693446344722; Wed, 30 Aug 2023 18:45:44 -0700 (PDT) Received: from smtpclient.apple ([72.89.196.186]) by smtp.gmail.com with ESMTPSA id fu9-20020a05622a5d8900b00410910c6164sm149205qtb.50.2023.08.30.18.45.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 30 Aug 2023 18:45:44 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Mark Saad List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (1.0) Subject: Re: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] Date: Wed, 30 Aug 2023 21:45:33 -0400 Message-Id: <543FBABC-C75A-4AB0-BFB6-1C7C15ECBA4E@longcount.org> References: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> Cc: weh@microsoft.com, FreeBSD Hackers In-Reply-To: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> To: Mark Millard X-Mailer: iPhone Mail (20G75) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Queue-Id: 4RbkWs6qDmz4vp8 All Why not take scp out of the picture and try iperf? Why , we could be looki= ng at rss by default in Linux . --- Mark Saad | nonesuch@longcount.org > On Aug 30, 2023, at 8:10 PM, Mark Millard wrote: >=20 > =EF=BB=BFOn Aug 30, 2023, at 01:49, Mark Millard wrote= : >=20 >>> On Aug 30, 2023, at 01:22, Mark Millard wrote: >>>=20 >>>> On Aug 30, 2023, at 01:17, Mark Millard wrote: >>>=20 >>>> On Aug 29, 2023, at 12:52, Mark Millard wrote: >>>>=20 >>>>> Wei Hu wrote on >>>>> Date: Tue, 29 Aug 2023 12:55:35 UTC : >>>>>=20 >>>>>> Thanks for the update. Seems the numbers are the same on zfs and ufs.= That's=20 >>>>>> good to know.=20 >>>>>>=20 >>>>>> Yes, your numbers on ARM64 are better than mine on Intel. However, my= original >>>>>> intention was to find out why scp on Linux is performing much better t= han FreeBSD >>>>>> under the same hardware env.=20 >>>>>>=20 >>>>>> Is it possible to try Linux in your ARM64 setting? I am using Ubuntu 2= 2.04 on ext4=20 >>>>>> file system. >>>>>=20 >>>>>=20 >>>>> I tried to use the Hyper-V Quick Create on the Windows Dev Kit 2023 >>>>> to install a Ubuntu 22.04 . (No clue if ext4 would result.) But the >>>>> Hyper-V UEFI reports for the disk created: >>>>>=20 >>>>> 1. SCSI Disk 0,0 >>>>> The boot loader did not load an operating system. >>>>>=20 >>>>> (It then reports the network adapter attempt found no >>>>> boot image, but that is expected.) >>>>>=20 >>>>> That leaves me wondering if Hyper-V Quick Create >>>>> established a VM file holding Intel/AMD material >>>>> despite the aarch64 context. >>>>>=20 >>>>> Establishing a Ubuntu more directly is not familiar and >>>>> will have to be a background activity and, so, likely >>>>> will not be timely. If I did any experiments outside >>>>> Hyper-V (native booting), they would be with slower >>>>> USB3 SSD media than I use for FreeBSD. >>>>>=20 >>>>> I did notice that Hyper-V Quick Create did not create >>>>> a fixed sized disk but a dynamic sized one. That is >>>>> different than what I did for FreeBSD. >>>>>=20 >>>>> Also, it was not obvious if you were after aarch64 >>>>> Hyper-V testing vs. native-boot testing vs. both. So >>>>> I may have gone the wrong direction from the start. >>>>> It is possible that I'd find establishing a native-boot >>>>> easier and then be able to have a VM file created from >>>>> the media, more like what I did with FreeBSD. >>>>>=20 >>>>> The Ubuntu activity likely would not be analogous to >>>>> the FreeBSD builds having -mcpu=3D optimization used. >>>>>=20 >>>>> Back to $work. >>>>>=20 >>>>=20 >>>> I found a sequence of UI operations that worked for >>>> installing Ubuntu server 22.04.3 into Hyper-V in >>>> Windows 11 Pro on the Windows Dev Kit 2023 via >>>> use of a downloaded *.iso . >>>>=20 >>>> The kernel that results predates 6.0: >>>>=20 >>>> $ uname -ap >>>> Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 14:19:18 UTC= 2023 aarch64 aarch64 aarch64 GNU/Linux >>>>=20 >>>> Using my usual rule of rebooting before the first scp: >>>>=20 >>>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264= 841.img markmi@localhost:FreeBSD-14-TEST.img >>>> . . . >>>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.im= g = 100% 5120MB 431.3MB/s 00:11=20 >>>>=20 >>>> $ rm FreeBSD-14-TEST.img >>>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264= 841.img markmi@localhost:FreeBSD-14-TEST.img >>>> . . . >>>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.im= g = 100% 5120MB 482.2MB/s 00:10 >>>>=20 >>>> Definitely faster than the FreeBSD results that I reported >>>> earlier, including faster than the ThreadRipper 1950X with >>>> Optane in a PCIe slot (more like 300 MiBytes/sec). >>>>=20 >>>> I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual hard >>>> disk under Hyper-V. >>>>=20 >>>> For reference: >>>>=20 >>>> $ lsblk -f >>>> NAME FSTYPE FSVER LABEL UUID FSAVAI= L FSUSE% MOUNTPOINTS >>>> loop0 squashfs 4.0 0= 100% /snap/core20/1977 >>>> loop1 squashfs 4.0 0= 100% /snap/lxd/24326 >>>> loop2 squashfs 4.0 0= 100% /snap/snapd/19459 >>>> sda = =E2=94=9C=E2=94=80sda1 vfat FAT32 F7E9-1344 = 1G 1% /boot/efi >>>> =E2=94=94=E2=94=80sda2 ext4 1.0 48a0dbe6-5a99-4b6e-92dc-fe6= d8efc6ffe 99.3G 14% / >>>>=20 >>>>=20 >>>>=20 >>>> An experiment would be to have a small amount if RAM relative >>>> the file size. That would force it to actually write to media >>>> for some part of the file copy. >>>=20 >>> The wording was poor: "force it" here is just from the >>> Ubuntu viewpoint. I make no claim to know if Hyper-V >>> is actually writing the material out to media at the >>> time vs. later. >>>=20 >>>> So using 1024 MiByte of RAM assigned in Hyper-V: >>>>=20 >>>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264= 841.img markmi@localhost:FreeBSD-14-TEST.img >>>> . . . >>>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.im= g = 100% 5120MB 407.5MB/s 00:12 >>>>=20 >>>> $ rm FreeBSD-14-TEST.img >>>> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264= 841.img markmi@localhost:FreeBSD-14-TEST.img >>>> . . . >>>> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.im= g = 100% 5120MB 404.7MB/s 00:12 >>>>=20 >>>> Still definitely faster than the FreeBSD results that I >>>> reported earlier, including faster than the ThreadRipper >>>> 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec). >>=20 >> One more variation in ubuntu under Hyper-V, still with 1024 MiBytes >> of assigned RAM: use of localhost:/dev/null >>=20 >> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26484= 1.img markmi@localhost:/dev/null >> . . . >> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >>=20 >> $ scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26484= 1.img markmi@localhost:/dev/null >> . . . >> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 492.9MB/s 00:10 >>=20 >>=20 >> The matching FreeBSD examples with 24576 MiBytes of RAM assigned (ZFS con= text): >>=20 >> # scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26484= 1.img root@localhost:/dev/null >> . . . >> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >>=20 >> # scp FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26484= 1.img root@localhost:/dev/null >> . . . >> FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 198.7MB/s 00:25 >>=20 >>=20 >> Note: At most one VM running at a time, never both in overlapping times. >=20 > Avoiding having a cipher involved and even localhost > involved: use dd . . . >=20 >=20 > FreeBSD examples for Windows Dev Kit 2023 Hyper-V context, > 24576 MiByts of RAM assigned): >=20 > # dd if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26= 4841.img of=3D/dev/null bs=3D1m status=3Dprogress > 2512388096 bytes (2512 MB, 2396 MiB) transferred 1.046s, 2402 MB/s > 5120+0 records in > 5120+0 records out > 5368709120 bytes transferred in 1.627071 secs (3299614770 bytes/sec) > CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd if=3DFreeBSD-14.0-ALPHA2-ar= m-armv7-GENERICSD-20230818-77013f29d048-264841.img of=3D/dev/null bs=3D1k st= atus=3Dprogress > 5233509376 bytes (5234 MB, 4991 MiB) transferred 14.022s, 373 MB/s > 5242880+0 records in > 5242880+0 records out > 5368709120 bytes transferred in 14.365142 secs (373731714 bytes/sec) > CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd if=3DFreeBSD-14.0-ALPHA2-ar= m-armv7-GENERICSD-20230818-77013f29d048-264841.img of=3D/dev/null bs=3D512 s= tatus=3Dprogress > 5285410816 bytes (5285 MB, 5041 MiB) transferred 27.029s, 196 MB/s > 10485760+0 records in > 10485760+0 records out > 5368709120 bytes transferred in 27.432570 secs (195705657 bytes/sec) >=20 >=20 > Ubuntu 22.04.3 for Windows Dev Kit 2023 Hyper-V context, > only 1024 MiBytes of RAM assigned: >=20 > $ dd if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26= 4841.img of=3D/dev/null bs=3D1M status=3Dprogress > 4003463168 bytes (4.0 GB, 3.7 GiB) copied, 2 s, 2.0 GB/s > 5120+0 records in > 5120+0 records out > 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 2.56342 s, 2.1 GB/s > $ dd if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-26= 4841.img of=3D/dev/null bs=3D1K status=3Dprogress > 4793865216 bytes (4.8 GB, 4.5 GiB) copied, 6 s, 799 MB/s > 5242880+0 records in > 5242880+0 records out > 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.60403 s, 813 MB/s > markmi@ubwdk23s:~$ dd if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-2023081= 8-77013f29d048-264841.img of=3D/dev/null bs=3D512 status=3Dprogress > 4800102912 bytes (4.8 GB, 4.5 GiB) copied, 9 s, 533 MB/s > 10485760+0 records in > 10485760+0 records out > 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 9.95606 s, 539 MB/s >=20 >=20 > =3D=3D=3D > Mark Millard > marklmi at yahoo.com >=20 >=20 From nobody Thu Aug 31 02:17:58 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RblFN5fsBz4rYPR for ; Thu, 31 Aug 2023 02:18:16 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic310-22.consmr.mail.gq1.yahoo.com (sonic310-22.consmr.mail.gq1.yahoo.com [98.137.69.148]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RblFN2q6fz3M8g for ; Thu, 31 Aug 2023 02:18:16 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693448294; bh=FxthQSbbNo8rXjtf9Au29rmhT2jrdRtt4LmJWmm6HYk=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=HAIJZqvhQtXKWJlNhMBNjnyq3ENz4OP2MQatq2jg2EBaL1ZLD8kpqUdxrSFFLJZQwMR9/JzuODFMM20jtz54XSi+prPR3ZXFLdle9TCqL0++lUwpS1RA4CBXEg1rrrTwrcasR8idV25G9RY/wKAI9wNCB93TII8V45kzv0OjW6Szp/IStZMnvC3aG2NeAi91PA1otxuo1x++JqJpViPFIv31n0Mgs5NvMyRqeI6VIyspsus3lJSYlzaZxRcyXdmgM4Aro+3rKFJ/FMU8OS6d4UN626i/9mbftVS5nMk7oJAYEvIO15q92D10LV7RbxdN7s2dLU8GKkBEBpenrLr+7w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693448294; bh=XwrErb+EQGS0QTvkAobT4aO6HUPyzmNaNVky8X8CCj/=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=BfYC+PqZPcWOM21N+If7xBBJGxe3d3tphFsloHwtVK9T+ND8jgw90PAdE4vp7LIvoUBrhnjl1V1aWpg8BtFTDZwEqUB38ujtGsnM9XUCPkSZtuqC9Jsx/71vqKD6Z8Zm228AejcT5YvWVkHIs+CtNy7JSRaCmAz9nGKH/s67JyI5ww3pRwjOWbafSw2P2d2NjAPJ/ZcPNH82eHSDz70OWj68GdoC2x0wNjaZaebslOSNmUFzoJvF5iyZ7dgdkKapyj+ibOyCyzJEnzF06LUjrqQwBVOhu6Iimdu5J9Nq1eS2i6Rl2U6PaeVzPbK30jzxzOjDJqep9nrdoXLohS+oBg== X-YMail-OSG: 0BsUKrcVM1mWM9Q3Z5pXcbAUj2Ia6Hb5s0.NF6K_y_N6iU.T0exUFo5JGmh5jgO W_8.m6YJA_Ic9VjDIAAo6NO3jjg4ZYJwEW_diONgkXFkLKQkKOHBhR3xMLutDoXAmzTrdEdY.iGa Z917kHp9PB200DP9rkgCX3sloF04EDe5Ock7vpSM0LyMp8loBTf0PJ9IDNAEbiYoESrsmjP9lAnW JjNeXj2gUmQYof9FgX06whyeoqf6Q4BYWdRwvyotOyvGJ8xMqiMJTNJQ3Z4gIyrRDqtHxkV8A0zz yUPF7i6R.gKMIYCizyruU2tByFoL628oRNpExY9oVYucrKusQeBUNv5mXmAiekpZtBXxSQWGNN8a Fb_x5I9b1uf6uNwm5dy1rS3mblT1giWAXGUF0P14Mcoc0M8KSsvL2PlArGb9PHjsxj.Aa7lqFTw8 bOFgzLe81jjHGAVBuMHx9KGRWsdjEWXC_ZSL5VqVjRhvSHYc8lysxPkPaGqHEFalcj58B7dXFtek UknsXGjxHFg_JGM5TsgyrQ.E9.vjUm9I5cQLFTUBiJuD9X_svW8aswhdmVjQlvIOu.ezbA2G_op. DXXCkVAqYbfhc6jiFyJw5CKfJI_HbQUW2xVIqU91kjb2LjsjFRptkbOXAPUlxfLJ4YThzbTvFDbM rlRraK_.W_q2Oe2tlpezH9_ssh2Ty2p3HS8518m71mejIeT_ZwhTEdbN1Dr63SzDlUghE77QWkWA Se_QO9jd4Oi9y0Wzl_y583Pu0_iKVzuRVRHrW9OpGtWZV5C7UUyaMqzQ8WztYB70GFzNnQOVOiD8 hZvTnzoov8ZqRh0XmGmuDYOpmlyqDUMrNdgUdtrtfpVV_Uw7pBFWTrNRFUy7URkNAkiIQPeE2Snb ZS99J.qVWh5DUobmCh4NyC8c9FaRirnNezbAATtW_RCSCNwNTYJiiLco6SGyY1j72rAWOIhsgjS7 EIRLSm6I.chCZkGHVWOKFltuN51YQKgyqWaRuo00CV26z5iHUJomOJhw_Lti96N_VmlUt8F_T_41 QsPbw9MTLcxKSJwkuecwoMkOegZr94B7DZnXPEQoImCfFB.dN_1wReKCV_RxAmGvYf3ESJG1xqFj DNkJFb0NXqf4tI5dD0YDZrOD.hBlWE5eZG9OgAiJvptMeLbFez3D.acWU1abxmOPnjWu67Y0Ib2f UFKE_rDVo5Pfp12fje7fUDNRJ77TZDM15m0HI3uRPH.EABZoPisw9_FaYFwuIKO8i3J36gJGFFhq 46k7x7OUdqamkUREbeB1tdxrMawC2ZMW0K25gmh3NC5.OUKlEFwwH49xSqytU01QK37UEfYX467. f6r7oYFPjgLnC2fFvfGtGsVnI51Q3F3Af0utuQYM.D_F2Xi9Z9nJO7NplvqND76MLhf7WhePmD2N iIIFxGfQOQ7FZpmmFPSR7WCcTTXlVN4.6T0BKJd6_1ZjtXf53t2.4Knyt6DCzUuHFszi5sxmw3yJ 7QmnLU2gzUgspp56.I0EErWDHQj1etpVqeghy9PZLK7DuZpWYkoC38vxAitNDleIqGEoP6tO0ERX 3uy3vN_igEg6B7yD8vGD8e9AanBZctXmXZwK9ifv30Aygstx34Ns2ux6FSb8K6jZu7rhj2uR0oCH zUDfxDwDtOObnlofXS9.3aNqrzXA1TQ.SUMvB3YN9nQN0BArnLpx4UxTU4TdDtuX4Prfv6gEANlk PYjKyfQi1AFU22tR5C_YNzAae2NYQGBdcGWvSdxRQUh8MpZ74p09lOe9CNr3Ahf_f6aifwb1Z0RZ QxQfOgWIpukphht8mUFhOVQVUI1ZyyMmJykgtt4PKv5s3Pukm1VxT3ZsbndE4ZFROdknid1QOEN7 F70z4yPYFdmF.kTmS4VYkgngMe0Z45S5t.aD5cVf0PFQKCm3w18EUVyZ_UL8EnWRzD8UZgP0t0ld uLHVOPppuWKSwR1Wri.Hm8hQV6iL_O4AXI51mB2hQVETli0ejMUbDw5w8ermCA7igAmcTMc9d6Lt .Oe9hd_OM4E3E_qnw35o498Wt6vLZNnwEceLJlRyjoRPwhp2ZxxUQcINxtW1ID8n8Za9Lx6L7QXc hZiZbfPZkpDBjXdzb3SzfOtFJ5cdGs0C1LuhpLo3sA56zyOSaoYI9WtPQ26fvpd3AR3FOHwAqjby CGSwPxjlXUXrTb6z9QOOlTlkDNgHFstwNIe0H_Ro_aUfugTn8b0DnhquPngUhOq9XvFB9c07EhB4 SpADOW6vmzrl6Anz4J2s1ArFDsOEEblbnNI9Xq85kU_nqVXFQhwkH3wBWkEBk5XFnfKzxWjD3x_Q l X-Sonic-MF: X-Sonic-ID: 275bbaaa-2790-4b63-883e-04de88491482 Received: from sonic.gate.mail.ne1.yahoo.com by sonic310.consmr.mail.gq1.yahoo.com with HTTP; Thu, 31 Aug 2023 02:18:14 +0000 Received: by hermes--production-bf1-865889d799-k5x9p (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 0854b6417402866361f1300c9d96ae17; Thu, 31 Aug 2023 02:18:10 +0000 (UTC) Content-Type: text/plain; charset=utf-8 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] From: Mark Millard In-Reply-To: <543FBABC-C75A-4AB0-BFB6-1C7C15ECBA4E@longcount.org> Date: Wed, 30 Aug 2023 19:17:58 -0700 Cc: weh@microsoft.com, FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <96A99928-4E8E-4AB1-BF3A-D026B280EC0E@yahoo.com> References: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> <543FBABC-C75A-4AB0-BFB6-1C7C15ECBA4E@longcount.org> To: Mark Saad X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Queue-Id: 4RblFN2q6fz3M8g On Aug 30, 2023, at 18:45, Mark Saad wrote: > All > Why not take scp out of the picture and try iperf? Why , we could be = looking at rss by default in Linux . The explorations with ???@localhost:/dev/null and with dd suggest non-networking issues are a significant contributor to the data rate differences being observed on FreeBSD vs. Ubuntu 22.04.3 --including when no networking is involved at all. I also did experiments with cipher selections that I've not reported. As I've no clue why the original note was about specifically about scp performance, I've just been try to gather data that might be of some use, even for other contexts than just scp used over an actual network. Also having iperf figures for just the network contribution would be useful too --if the network had appropriate characteristics for comparison to the original context. The network that I'm using is limited to 1 Gbit/s and may not be a good match for comparison to the original context. I've just not gone that direction so far. > --- > Mark Saad | nonesuch@longcount.org >=20 >> On Aug 30, 2023, at 8:10 PM, Mark Millard wrote: >>=20 >> =EF=BB=BFOn Aug 30, 2023, at 01:49, Mark Millard = wrote: >>=20 >>>> On Aug 30, 2023, at 01:22, Mark Millard wrote: >>>>=20 >>>>> On Aug 30, 2023, at 01:17, Mark Millard wrote: >>>>=20 >>>>> On Aug 29, 2023, at 12:52, Mark Millard wrote: >>>>>=20 >>>>>> Wei Hu wrote on >>>>>> Date: Tue, 29 Aug 2023 12:55:35 UTC : >>>>>>=20 >>>>>>> Thanks for the update. Seems the numbers are the same on zfs and = ufs. That's=20 >>>>>>> good to know.=20 >>>>>>>=20 >>>>>>> Yes, your numbers on ARM64 are better than mine on Intel. = However, my original >>>>>>> intention was to find out why scp on Linux is performing much = better than FreeBSD >>>>>>> under the same hardware env.=20 >>>>>>>=20 >>>>>>> Is it possible to try Linux in your ARM64 setting? I am using = Ubuntu 22.04 on ext4=20 >>>>>>> file system. >>>>>>=20 >>>>>>=20 >>>>>> I tried to use the Hyper-V Quick Create on the Windows Dev Kit = 2023 >>>>>> to install a Ubuntu 22.04 . (No clue if ext4 would result.) But = the >>>>>> Hyper-V UEFI reports for the disk created: >>>>>>=20 >>>>>> 1. SCSI Disk 0,0 >>>>>> The boot loader did not load an operating system. >>>>>>=20 >>>>>> (It then reports the network adapter attempt found no >>>>>> boot image, but that is expected.) >>>>>>=20 >>>>>> That leaves me wondering if Hyper-V Quick Create >>>>>> established a VM file holding Intel/AMD material >>>>>> despite the aarch64 context. >>>>>>=20 >>>>>> Establishing a Ubuntu more directly is not familiar and >>>>>> will have to be a background activity and, so, likely >>>>>> will not be timely. If I did any experiments outside >>>>>> Hyper-V (native booting), they would be with slower >>>>>> USB3 SSD media than I use for FreeBSD. >>>>>>=20 >>>>>> I did notice that Hyper-V Quick Create did not create >>>>>> a fixed sized disk but a dynamic sized one. That is >>>>>> different than what I did for FreeBSD. >>>>>>=20 >>>>>> Also, it was not obvious if you were after aarch64 >>>>>> Hyper-V testing vs. native-boot testing vs. both. So >>>>>> I may have gone the wrong direction from the start. >>>>>> It is possible that I'd find establishing a native-boot >>>>>> easier and then be able to have a VM file created from >>>>>> the media, more like what I did with FreeBSD. >>>>>>=20 >>>>>> The Ubuntu activity likely would not be analogous to >>>>>> the FreeBSD builds having -mcpu=3D optimization used. >>>>>>=20 >>>>>> Back to $work. >>>>>>=20 >>>>>=20 >>>>> I found a sequence of UI operations that worked for >>>>> installing Ubuntu server 22.04.3 into Hyper-V in >>>>> Windows 11 Pro on the Windows Dev Kit 2023 via >>>>> use of a downloaded *.iso . >>>>>=20 >>>>> The kernel that results predates 6.0: >>>>>=20 >>>>> $ uname -ap >>>>> Linux ubwdk23s 5.15.0-82-generic #91-Ubuntu SMP Mon Aug 14 = 14:19:18 UTC 2023 aarch64 aarch64 aarch64 GNU/Linux >>>>>=20 >>>>> Using my usual rule of rebooting before the first scp: >>>>>=20 >>>>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>>>> . . . >>>>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 431.3MB/s 00:11=20 >>>>>=20 >>>>> $ rm FreeBSD-14-TEST.img >>>>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>>>> . . . >>>>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 482.2MB/s 00:10 >>>>>=20 >>>>> Definitely faster than the FreeBSD results that I reported >>>>> earlier, including faster than the ThreadRipper 1950X with >>>>> Optane in a PCIe slot (more like 300 MiBytes/sec). >>>>>=20 >>>>> I again used 6 cores, 24576 MiBytes of RAM, a fixed sized virtual = hard >>>>> disk under Hyper-V. >>>>>=20 >>>>> For reference: >>>>>=20 >>>>> $ lsblk -f >>>>> NAME FSTYPE FSVER LABEL UUID = FSAVAIL FSUSE% MOUNTPOINTS >>>>> loop0 squashfs 4.0 = 0 100% /snap/core20/1977 >>>>> loop1 squashfs 4.0 = 0 100% /snap/lxd/24326 >>>>> loop2 squashfs 4.0 = 0 100% /snap/snapd/19459 >>>>> sda = =E2=94=9C=E2=94=80sda1 vfat FAT32 F7E9-1344 = 1G 1% /boot/efi >>>>> =E2=94=94=E2=94=80sda2 ext4 1.0 = 48a0dbe6-5a99-4b6e-92dc-fe6d8efc6ffe 99.3G 14% / >>>>>=20 >>>>>=20 >>>>>=20 >>>>> An experiment would be to have a small amount if RAM relative >>>>> the file size. That would force it to actually write to media >>>>> for some part of the file copy. >>>>=20 >>>> The wording was poor: "force it" here is just from the >>>> Ubuntu viewpoint. I make no claim to know if Hyper-V >>>> is actually writing the material out to media at the >>>> time vs. later. >>>>=20 >>>>> So using 1024 MiByte of RAM assigned in Hyper-V: >>>>>=20 >>>>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>>>> . . . >>>>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 407.5MB/s 00:12 >>>>>=20 >>>>> $ rm FreeBSD-14-TEST.img >>>>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:FreeBSD-14-TEST.img >>>>> . . . >>>>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 404.7MB/s 00:12 >>>>>=20 >>>>> Still definitely faster than the FreeBSD results that I >>>>> reported earlier, including faster than the ThreadRipper >>>>> 1950X with Optane in a PCIe slot (more like 300 MiBytes/sec). >>>=20 >>> One more variation in ubuntu under Hyper-V, still with 1024 MiBytes >>> of assigned RAM: use of localhost:/dev/null >>>=20 >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:/dev/null >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >>>=20 >>> $ scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = markmi@localhost:/dev/null >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 492.9MB/s 00:10 >>>=20 >>>=20 >>> The matching FreeBSD examples with 24576 MiBytes of RAM assigned = (ZFS context): >>>=20 >>> # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:/dev/null >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = =20 >>>=20 >>> # scp = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = root@localhost:/dev/null >>> . . . >>> = FreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.img = = 100% 5120MB 198.7MB/s 00:25 >>>=20 >>>=20 >>> Note: At most one VM running at a time, never both in overlapping = times. >>=20 >> Avoiding having a cipher involved and even localhost >> involved: use dd . . . >>=20 >>=20 >> FreeBSD examples for Windows Dev Kit 2023 Hyper-V context, >> 24576 MiByts of RAM assigned): >>=20 >> # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1m status=3Dprogress >> 2512388096 bytes (2512 MB, 2396 MiB) transferred 1.046s, 2402 MB/s >> 5120+0 records in >> 5120+0 records out >> 5368709120 bytes transferred in 1.627071 secs (3299614770 bytes/sec) >> CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1k status=3Dprogress >> 5233509376 bytes (5234 MB, 4991 MiB) transferred 14.022s, 373 MB/s >> 5242880+0 records in >> 5242880+0 records out >> 5368709120 bytes transferred in 14.365142 secs (373731714 bytes/sec) >> CA78C-WDK23s-ZFS aarch64 1500000 1500000 # dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D512 status=3Dprogress >> 5285410816 bytes (5285 MB, 5041 MiB) transferred 27.029s, 196 MB/s >> 10485760+0 records in >> 10485760+0 records out >> 5368709120 bytes transferred in 27.432570 secs (195705657 bytes/sec) >>=20 >>=20 >> Ubuntu 22.04.3 for Windows Dev Kit 2023 Hyper-V context, >> only 1024 MiBytes of RAM assigned: >>=20 >> $ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1M status=3Dprogress >> 4003463168 bytes (4.0 GB, 3.7 GiB) copied, 2 s, 2.0 GB/s >> 5120+0 records in >> 5120+0 records out >> 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 2.56342 s, 2.1 GB/s >> $ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D1K status=3Dprogress >> 4793865216 bytes (4.8 GB, 4.5 GiB) copied, 6 s, 799 MB/s >> 5242880+0 records in >> 5242880+0 records out >> 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 6.60403 s, 813 MB/s >> markmi@ubwdk23s:~$ dd = if=3DFreeBSD-14.0-ALPHA2-arm-armv7-GENERICSD-20230818-77013f29d048-264841.= img of=3D/dev/null bs=3D512 status=3Dprogress >> 4800102912 bytes (4.8 GB, 4.5 GiB) copied, 9 s, 533 MB/s >> 10485760+0 records in >> 10485760+0 records out >> 5368709120 bytes (5.4 GB, 5.0 GiB) copied, 9.95606 s, 539 MB/s >=20 =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Aug 31 03:59:52 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbnVl5lwhz4rfnj for ; Thu, 31 Aug 2023 03:59:59 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from HK2P15301CU002.outbound.protection.outlook.com (mail-eastasiaazon11020026.outbound.protection.outlook.com [52.101.128.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbnVl0JXlz3c85 for ; Thu, 31 Aug 2023 03:59:59 +0000 (UTC) (envelope-from weh@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eqmjPRjz5UNoRlvSgOOxAf2FxBLmsmsGXpGTLn+JuEr4dCsj258lGskx/a7vem3KWNpXwJ3lcbI3sooTPZ1D7MeolgZ2xBAhojEvKAvpHgXYsqNtzUjpdr34yCPMV6aTZREJBzXKcS35+Pn+WzZcEicRTH8rJn3fovFd8mob+lRg/9kCkBsPOBy2NGiQtoLl8thr+nLE8m4FAQCyfgribH1K/9CNIbNa16N/Kd4sMRsmOonW3CMtYO7PNXZopPvNS7EK48hoxPNZ5xOJODJzme4F74gFdg5Y6OO7/nzQzfTW/0RKVfkHxEn+J0P8EWW3sT/r0ZVnL0og2iNQAWakEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=QHE7MXi+LEOj80JbHv+uAh/jNh5HhOhXSbHTb4nFp/Q=; b=fEivEYP4DQQN+62Ht3Q+RlWuw/J4Si/OZeMZhSKe/fFaS87YIYGptiSZeAQbWrt2RyOLK1g+dqG90gOvFHmqX3xz/CNvimiDuhvpeSxvJpVO1Ckpkm/LJZP9ntb/n28OFTnYf5OBSdMzn8+J0KwqcT0TMRvSMpyF+qW0dgAE8kykjn0C7p/YSmP2ApIYMA2krsKKQH6VX7I8T8Z9RcnfVwCSHUVUcSqsH+/B2YBNs7MlKgkRSWBVWODwB4lEOrFA5V82JCl2jbXo1fAVeYdIDm+iIhSSNOXhsT2yHEWBHK5R56N5DQhHi12RRPv/XDDIKwEIciyEXp6OSYC/BfWpqg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QHE7MXi+LEOj80JbHv+uAh/jNh5HhOhXSbHTb4nFp/Q=; b=cQ4T+Rbw+b/Cb0IETitRQlH7Oowfd/0abqxGaKrqY5JaRVC/s4KYQREIili5BgugtiP8X7MCT74xWXptSyHlYOXSl/rpIGMduPzJcy3RSwzg9dUx7MFFMaKs94HeLi9vHD59Tq0K89IjCobi6stMQojndonZFiOorJzHCfqPHyY= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by TYZP153MB0658.APCP153.PROD.OUTLOOK.COM (2603:1096:400:260::5) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.12; Thu, 31 Aug 2023 03:59:53 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83%7]) with mapi id 15.20.6768.012; Thu, 31 Aug 2023 03:59:53 +0000 From: Wei Hu To: Mark Millard , Mark Saad CC: FreeBSD Hackers Subject: RE: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] Thread-Topic: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] Thread-Index: AQHZ2597zvq5AgzMUkaYwDpn0euEM7ADomqAgAAJDgCAABvOYA== Date: Thu, 31 Aug 2023 03:59:52 +0000 Message-ID: References: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> <543FBABC-C75A-4AB0-BFB6-1C7C15ECBA4E@longcount.org> <96A99928-4E8E-4AB1-BF3A-D026B280EC0E@yahoo.com> In-Reply-To: <96A99928-4E8E-4AB1-BF3A-D026B280EC0E@yahoo.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=af5c166e-93b7-49c5-95eb-3c5252001651;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-08-31T03:57:28Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2P153MB0441:EE_|TYZP153MB0658:EE_ x-ms-office365-filtering-correlation-id: 9eefb7ce-342c-420b-cace-08dba9d6baf7 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: k6yKNYhMD3w2H8KctGlrE3ehFuNQn6QrTCR4xJarWt/eAkYu9XXUPxBRfVyh8qcQz/PlDFiSS4AHclycxaAO0fC6PXLPCm5CEAUfXz9yZrMbZ5vXYjWB0pgk/JlsGoxZWDnCzLQekSAC+KobYoJtWeY3dapi49oDj6KxX2qeCwy7Afi0+WEZiV5sAhNjprU6A0BBR+IXD8++molc+XW5FCAtr7zqSRud8YpS6sZE5fkMsUhjzOue157U1jq6Fo+IZ7xLgLU2iwYHuII/cv2lt1y+03Yd9Az2psWfNW2HLdBEHbX/DD1/ZoN4Oy8xTspuHjJUrjZQ0ghoFSMkDLH9Y/5qSMzD6ncPpxOJzEEClD8r+ohCiA5mTljMQfdWIQ+GFj/A10dYQH6qpofW97rduAIcnc1vr0nXnl5Q5n4FLO4UANTPfsSiE0QYDqzC/5o362RNWDT4FgLS299ueCv4Kz0CPKKblTabaCDtYZJ76rR4zVyDMqySx/zf17ikquvU8xw/vEJqJ7aUQoh9wclvCGttvTr3ttAgMdkVV/zXOJJExHiavNjjs6KzNu/OWKLHSHtZ7nnyXuN2wE6wKKdlHH8rRC11c3bWowkHJrBtqjq2YAB9T5Iic8aFDXKh1s2KXumaeBiLKQZFr9ZotOAumNaYWHsSpwvoSOomaFFsRzQ= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(376002)(346002)(366004)(136003)(396003)(39860400002)(186009)(1800799009)(451199024)(7696005)(71200400001)(6506007)(9686003)(53546011)(19627235002)(478600001)(10290500003)(83380400001)(26005)(30864003)(2906002)(8990500004)(66446008)(64756008)(66476007)(66556008)(316002)(41300700001)(66946007)(76116006)(110136005)(52536014)(5660300002)(4326008)(8936002)(8676002)(38070700005)(38100700002)(33656002)(86362001)(55016003)(122000001)(82950400001)(82960400001);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?utf-8?B?RTkzYmpLalNkb2xmRkduYjdraEhRVXErR0VoMFF6cTZQQ1VxVlAvVElXZDI5?= =?utf-8?B?TUpUWnBDSmdmUDVZZ0ljcmUwTnNSc0dRcXNtRVdhbW9BREM3NDdWcU9CSTFz?= =?utf-8?B?aHVBSEhzc3Zrc3N6ZUVqWFZhV2Z2eTA2MkZSeXhreXNHRm5hWkJZRUVsa3Q5?= =?utf-8?B?b1ZKWlFuSk5lQS9iQlB4VkdyUkhSMTVvUTF2Si80U1RTZG1EZzVPc3ljbE1n?= =?utf-8?B?aHZwQmtaTStIeGJWamExM3NWN2dOcC9OcVlIVHU2bkg5VENRQVplMVhZY1RG?= =?utf-8?B?SElpbHdJSnNMM05aQVhrQ3hmK3RWRSt1M01OdEczTklEWERmTi84SVhQZmpC?= =?utf-8?B?KzVTSmV2V25UNE9vVlQvc1dGTmF0UUsvK0NkQmV5VitHL2ZqT2VQVWlxUkEw?= =?utf-8?B?Qm1IZmcvdlo0d1lVaGlvSG9ueHMvZyt6Z3ZadlVoUnJ5ZitVbXdyYkc2eHNx?= =?utf-8?B?R1U4K3E4OGMyVG84WmZqaVhCOFBDNkFoQStGeG5MaXBLalJMSk5ZMkIwb2Iy?= =?utf-8?B?UnlRci80V1lEZFJIaUlCV1VLRXM5R1d2YTJIZ2tYV0JYdm1tblVWRkswWXZH?= =?utf-8?B?aWxwNkIvZTdsbS9veCtpY2l4OXRMelpwNkE1bnZ6UGlpZnFWRTJFZ01oUUNV?= =?utf-8?B?WmRwODI2STBNSlN4MEJlZ2tvUnpoeXM0WkZmSEFVQkN0TlV1WEJubWZwQ0ln?= =?utf-8?B?ZmVCNTZRQ2JuR3l3empXaWpqVzFNR1hFQTMxNjNvdlZJR293dHlmYnJFNnFw?= =?utf-8?B?RktJeEVrU3FCVWNxWllBclpMM0RvSkxuQ2F3SzQ4RHhQWDNCNUdNaFlnMWRl?= =?utf-8?B?eGdZbkszaXh3Sjd3bDMrcktvc1FwRkkxcEN5SFdDaDlXSmRRMzlka0I2ZSsw?= =?utf-8?B?ejF1eHdTWHJxU0lCYUZvbjRkTjlUekRxRUR1Y2twUXhTZWVmbFJaN29VeUpU?= =?utf-8?B?SzIrUHBLbW52K1k0SXhIUzNoalhHUWlRMzBOUUVIZjM4UlBVUytocGhwaGJE?= =?utf-8?B?MDNranUxSDFuQktYQnhLQ0lrc1BNLy9oOFF2aVFWbWUwQjEreDJUMGs3YXRU?= =?utf-8?B?ck95Tm9Wb0paZ2VlbGFHTzZZYSs3TUNPUFdxcTkvZEJuTlpLVE1SaEtJTFJT?= =?utf-8?B?WHZpcGtYL3JEeitWbTZpd2hRNFZFN2Vscy9HVDhXeURyZXkreVlaUTVydHZj?= =?utf-8?B?YS8zZnRxa044Y2hnb29sQnNMK1VhSGhrMzdzaFJ1NDlFakxaUzRoTUxYMkNZ?= =?utf-8?B?U20yeWZWZHcwSGxDREIvNHlpMmR5MjQ2NURpK2VqVUVwcE1FUWN3dU9YZWZD?= =?utf-8?B?U3dKT0lXdW1FUXB1czBIZkVKaU44SFJaZ0gra0FVYXhuY3VlQ0ViN3U5bVd6?= =?utf-8?B?cE8xbEZicnlXOFBDQmRyTmhPbnVFMHBnR3VmQ2hTbVF5ZTA3YXZPSWtubmhZ?= =?utf-8?B?WHY4bDcwRVdNMjlETWp4THU1bmw0U0NpSlpGYXg1NUhieUdpdUpkcXV3YXZK?= =?utf-8?B?dVYzZHlxNE9vMHh5WFV1ZFhGOTdENzJGQXdQNytzQjREQmZQd3dHc0UybE9U?= =?utf-8?B?MjVVMW5QZ25kWUhYSjdyZnVObzUyaUtGMDZGVExQa1ROdCtzWXZtVzY0SXJZ?= =?utf-8?B?TnE3WWM4WkZJSGh0SlJQK2NEZTF2L0l3eUVzbC8yT1FPMTlKWUQydGp0c21J?= =?utf-8?B?NjdFRVAzR0pKWUt1ekU5ZExOZ0V4L2N0bEFGSVZ3YmpaditzYnZOZlZXcUZm?= =?utf-8?B?dlI0TXp5ODdlKzZHb3FDUzhJVHlZMzh4RDErR2lvTll2cFE4eFRIdnpxMTRr?= =?utf-8?B?a0ExcHZFR0hqZWJDbG1rcFVNSUcyd25wcXFuYUxYOUpMRmdNUHU2MEUrQzRM?= =?utf-8?B?R09lOCtLYVZ2UUoweDJ2NDhCQ1N4MHhFMTlsUnVVdzR4T2lvZm52OEhuc1Yx?= =?utf-8?B?blFqbk9ROTFCRG1JTXRKZHJYMTFwUXZEaVdnT1hnNXI4YjVJWVl2dU9sdThj?= =?utf-8?B?Tk1pVEpXM3cyUXRRNXNrcUY3M1h5VmtxdzdiVjNmcWNVTVVWK2pkN1JNQThM?= =?utf-8?B?ZXFQcy8zNHl1akVHRGlyb01xVFBvYU5VL2FlM0JuazZqTHFpZmU5bEVtM2lW?= =?utf-8?Q?RMJw=3D?= Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: 9eefb7ce-342c-420b-cace-08dba9d6baf7 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Aug 2023 03:59:52.7361 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: SXtHfRWD05VwtjIKwcLlPp6zmQ/L5T2pc0TzEX6W/YGYILHEk8j7t/aH3J3BtPG4ZTjShp0dR+h2mBfUwvv+KA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: TYZP153MB0658 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:52.96.0.0/12, country:US] X-Rspamd-Queue-Id: 4RbnVl0JXlz3c85 DQoNCj4gLS0tLS1PcmlnaW5hbCBNZXNzYWdlLS0tLS0NCj4gRnJvbTogTWFyayBNaWxsYXJkIDxt YXJrbG1pQHlhaG9vLmNvbT4NCj4gU2VudDogVGh1cnNkYXksIEF1Z3VzdCAzMSwgMjAyMyAxMDox OCBBTQ0KPiBUbzogTWFyayBTYWFkIDxub25lc3VjaEBsb25nY291bnQub3JnPg0KPiBDYzogV2Vp IEh1IDx3ZWhAbWljcm9zb2Z0LmNvbT47IEZyZWVCU0QgSGFja2VycyA8ZnJlZWJzZC0NCj4gaGFj a2Vyc0BmcmVlYnNkLm9yZz4NCj4gU3ViamVjdDogUmU6IFZlcnkgc2xvdyBzY3AgcGVyZm9ybWFu Y2UgY29tcGFyaW5nIHRvIExpbnV4IFtkZCB0byAvZGV2L251bGwNCj4gc2hvd3Mgc3Vic3RhbnRp YWwgRnJlZUJTRCB2cy4gVWJ1bnR1IGRpZmZlcmVuY2VzIGZvciBicz0xayAob3IgMUspIGFuZA0K PiBicz01MTJdDQo+IA0KPiBPbiBBdWcgMzAsIDIwMjMsIGF0IDE4OjQ1LCBNYXJrIFNhYWQgPG5v bmVzdWNoQGxvbmdjb3VudC5vcmc+IHdyb3RlOg0KPiANCj4gPiBBbGwNCj4gPiAgV2h5IG5vdCB0 YWtlIHNjcCBvdXQgb2YgdGhlIHBpY3R1cmUgYW5kIHRyeSBpcGVyZj8gV2h5ICwgd2UgY291bGQg YmUgbG9va2luZw0KPiBhdCByc3MgYnkgZGVmYXVsdCBpbiBMaW51eCAuDQoNCkFjdHVhbGx5IEkg ZGlkIHRoZSBpcGVyZjMgdGVzdCBhcyB3ZWxsIGFuZCBwb3N0ZWQgcmVzdWx0cyBhIGNvdXBsZSBk YXlzIGFnby4gDQpQYXN0aW5nIGhlcmU6DQoNCkZyZWVCU0QgaXBlcmYzIHRvIGxvY2FsaG9zdCwg c2luZ2xlIHN0cmVhbTogMzAuOSBHYi9zIA0KTGludXggaXBlcmYzIHRvIGxvY2FsaG9zdCwgc2lu Z2xlIHN0cmVhbTogNDguOCBHYi9zDQoNCk5laXRoZXIgb2YgdGhlbSBoYXMgYW55IHRjcCByZXRy eS4gDQoNCkJvdGggVk1zIHJ1biBvbiBJbnRlbChSKSBYZW9uKFIpIFBsYXRpbnVtIDg0NzNDICgy MTAwLjAwLU1IeiBLOC1jbGFzcyBDUFUpLiANClRoZSBGcmVlQlNEIFZNIGlzIDE2IHZjcHUsIHdp dGggMTI4IEdCIG1lbW9yeS4NClRoZSBMaW51eCBWTSBpcyA0IHZjcHUsIHdpdGggMzIgR0IgbWVt b3J5Lg0KDQpXZWkNCg0KDQo+IA0KPiBUaGUgZXhwbG9yYXRpb25zIHdpdGggPz8/QGxvY2FsaG9z dDovZGV2L251bGwgYW5kIHdpdGggZGQgc3VnZ2VzdCBub24tDQo+IG5ldHdvcmtpbmcgaXNzdWVz IGFyZSBhIHNpZ25pZmljYW50IGNvbnRyaWJ1dG9yIHRvIHRoZSBkYXRhIHJhdGUgZGlmZmVyZW5j ZXMNCj4gYmVpbmcgb2JzZXJ2ZWQgb24gRnJlZUJTRCB2cy4gVWJ1bnR1IDIyLjA0LjMgLS1pbmNs dWRpbmcgd2hlbiBubw0KPiBuZXR3b3JraW5nIGlzIGludm9sdmVkIGF0IGFsbC4NCj4gDQo+IEkg YWxzbyBkaWQgZXhwZXJpbWVudHMgd2l0aCBjaXBoZXIgc2VsZWN0aW9ucyB0aGF0IEkndmUgbm90 IHJlcG9ydGVkLg0KPiANCj4gQXMgSSd2ZSBubyBjbHVlIHdoeSB0aGUgb3JpZ2luYWwgbm90ZSB3 YXMgYWJvdXQgc3BlY2lmaWNhbGx5IGFib3V0IHNjcA0KPiBwZXJmb3JtYW5jZSwgSSd2ZSBqdXN0 IGJlZW4gdHJ5IHRvIGdhdGhlciBkYXRhIHRoYXQgbWlnaHQgYmUgb2Ygc29tZSB1c2UsIGV2ZW4N Cj4gZm9yIG90aGVyIGNvbnRleHRzIHRoYW4ganVzdCBzY3AgdXNlZCBvdmVyIGFuIGFjdHVhbCBu ZXR3b3JrLg0KPiANCj4gQWxzbyBoYXZpbmcgaXBlcmYgZmlndXJlcyBmb3IganVzdCB0aGUgbmV0 d29yayBjb250cmlidXRpb24gd291bGQgYmUgdXNlZnVsIHRvbw0KPiAtLWlmIHRoZSBuZXR3b3Jr IGhhZCBhcHByb3ByaWF0ZSBjaGFyYWN0ZXJpc3RpY3MgZm9yIGNvbXBhcmlzb24gdG8gdGhlIG9y aWdpbmFsDQo+IGNvbnRleHQuIFRoZSBuZXR3b3JrIHRoYXQgSSdtIHVzaW5nIGlzIGxpbWl0ZWQg dG8gMSBHYml0L3MgYW5kIG1heSBub3QgYmUgYQ0KPiBnb29kIG1hdGNoIGZvciBjb21wYXJpc29u IHRvIHRoZSBvcmlnaW5hbCBjb250ZXh0LiBJJ3ZlIGp1c3Qgbm90IGdvbmUgdGhhdA0KPiBkaXJl Y3Rpb24gc28gZmFyLg0KPiANCj4gPiAtLS0NCj4gPiBNYXJrIFNhYWQgfCBub25lc3VjaEBsb25n Y291bnQub3JnDQo+ID4NCj4gPj4gT24gQXVnIDMwLCAyMDIzLCBhdCA4OjEwIFBNLCBNYXJrIE1p bGxhcmQgPG1hcmtsbWlAeWFob28uY29tPiB3cm90ZToNCj4gPj4NCj4gPj4g77u/T24gQXVnIDMw LCAyMDIzLCBhdCAwMTo0OSwgTWFyayBNaWxsYXJkIDxtYXJrbG1pQHlhaG9vLmNvbT4gd3JvdGU6 DQo+ID4+DQo+ID4+Pj4gT24gQXVnIDMwLCAyMDIzLCBhdCAwMToyMiwgTWFyayBNaWxsYXJkIDxt YXJrbG1pQHlhaG9vLmNvbT4gd3JvdGU6DQo+ID4+Pj4NCj4gPj4+Pj4gT24gQXVnIDMwLCAyMDIz LCBhdCAwMToxNywgTWFyayBNaWxsYXJkIDxtYXJrbG1pQHlhaG9vLmNvbT4NCj4gd3JvdGU6DQo+ ID4+Pj4NCj4gPj4+Pj4gT24gQXVnIDI5LCAyMDIzLCBhdCAxMjo1MiwgTWFyayBNaWxsYXJkIDxt YXJrbG1pQHlhaG9vLmNvbT4NCj4gd3JvdGU6DQo+ID4+Pj4+DQo+ID4+Pj4+PiBXZWkgSHUgPHdl aF9hdF9taWNyb3NvZnQuY29tPiB3cm90ZSBvbg0KPiA+Pj4+Pj4gRGF0ZTogVHVlLCAyOSBBdWcg MjAyMyAxMjo1NTozNSBVVEMgOg0KPiA+Pj4+Pj4NCj4gPj4+Pj4+PiBUaGFua3MgZm9yIHRoZSB1 cGRhdGUuIFNlZW1zIHRoZSBudW1iZXJzIGFyZSB0aGUgc2FtZSBvbiB6ZnMgYW5kDQo+ID4+Pj4+ Pj4gdWZzLiBUaGF0J3MgZ29vZCB0byBrbm93Lg0KPiA+Pj4+Pj4+DQo+ID4+Pj4+Pj4gWWVzLCB5 b3VyIG51bWJlcnMgb24gQVJNNjQgYXJlIGJldHRlciB0aGFuIG1pbmUgb24gSW50ZWwuDQo+ID4+ Pj4+Pj4gSG93ZXZlciwgbXkgb3JpZ2luYWwgaW50ZW50aW9uIHdhcyB0byBmaW5kIG91dCB3aHkg c2NwIG9uIExpbnV4DQo+ID4+Pj4+Pj4gaXMgcGVyZm9ybWluZyBtdWNoIGJldHRlciB0aGFuIEZy ZWVCU0QgdW5kZXIgdGhlIHNhbWUgaGFyZHdhcmUNCj4gZW52Lg0KPiA+Pj4+Pj4+DQo+ID4+Pj4+ Pj4gSXMgaXQgcG9zc2libGUgdG8gdHJ5IExpbnV4IGluIHlvdXIgQVJNNjQgc2V0dGluZz8gSSBh bSB1c2luZw0KPiA+Pj4+Pj4+IFVidW50dSAyMi4wNCBvbiBleHQ0IGZpbGUgc3lzdGVtLg0KPiA+ Pj4+Pj4NCj4gPj4+Pj4+DQo+ID4+Pj4+PiBJIHRyaWVkIHRvIHVzZSB0aGUgSHlwZXItViBRdWlj ayBDcmVhdGUgb24gdGhlIFdpbmRvd3MgRGV2IEtpdA0KPiA+Pj4+Pj4gMjAyMyB0byBpbnN0YWxs IGEgVWJ1bnR1IDIyLjA0IC4gKE5vIGNsdWUgaWYgZXh0NCB3b3VsZCByZXN1bHQuKQ0KPiA+Pj4+ Pj4gQnV0IHRoZSBIeXBlci1WIFVFRkkgcmVwb3J0cyBmb3IgdGhlIGRpc2sgY3JlYXRlZDoNCj4g Pj4+Pj4+DQo+ID4+Pj4+PiAxLiBTQ1NJIERpc2sgMCwwDQo+ID4+Pj4+PiBUaGUgYm9vdCBsb2Fk ZXIgZGlkIG5vdCBsb2FkIGFuIG9wZXJhdGluZyBzeXN0ZW0uDQo+ID4+Pj4+Pg0KPiA+Pj4+Pj4g KEl0IHRoZW4gcmVwb3J0cyB0aGUgbmV0d29yayBhZGFwdGVyIGF0dGVtcHQgZm91bmQgbm8gYm9v dCBpbWFnZSwNCj4gPj4+Pj4+IGJ1dCB0aGF0IGlzIGV4cGVjdGVkLikNCj4gPj4+Pj4+DQo+ID4+ Pj4+PiBUaGF0IGxlYXZlcyBtZSB3b25kZXJpbmcgaWYgSHlwZXItViBRdWljayBDcmVhdGUgZXN0 YWJsaXNoZWQgYSBWTQ0KPiA+Pj4+Pj4gZmlsZSBob2xkaW5nIEludGVsL0FNRCBtYXRlcmlhbCBk ZXNwaXRlIHRoZSBhYXJjaDY0IGNvbnRleHQuDQo+ID4+Pj4+Pg0KPiA+Pj4+Pj4gRXN0YWJsaXNo aW5nIGEgVWJ1bnR1IG1vcmUgZGlyZWN0bHkgaXMgbm90IGZhbWlsaWFyIGFuZCB3aWxsIGhhdmUN Cj4gPj4+Pj4+IHRvIGJlIGEgYmFja2dyb3VuZCBhY3Rpdml0eSBhbmQsIHNvLCBsaWtlbHkgd2ls bCBub3QgYmUgdGltZWx5Lg0KPiA+Pj4+Pj4gSWYgSSBkaWQgYW55IGV4cGVyaW1lbnRzIG91dHNp ZGUgSHlwZXItViAobmF0aXZlIGJvb3RpbmcpLCB0aGV5DQo+ID4+Pj4+PiB3b3VsZCBiZSB3aXRo IHNsb3dlcg0KPiA+Pj4+Pj4gVVNCMyBTU0QgbWVkaWEgdGhhbiBJIHVzZSBmb3IgRnJlZUJTRC4N Cj4gPj4+Pj4+DQo+ID4+Pj4+PiBJIGRpZCBub3RpY2UgdGhhdCBIeXBlci1WIFF1aWNrIENyZWF0 ZSBkaWQgbm90IGNyZWF0ZSBhIGZpeGVkDQo+ID4+Pj4+PiBzaXplZCBkaXNrIGJ1dCBhIGR5bmFt aWMgc2l6ZWQgb25lLiBUaGF0IGlzIGRpZmZlcmVudCB0aGFuIHdoYXQgSQ0KPiA+Pj4+Pj4gZGlk IGZvciBGcmVlQlNELg0KPiA+Pj4+Pj4NCj4gPj4+Pj4+IEFsc28sIGl0IHdhcyBub3Qgb2J2aW91 cyBpZiB5b3Ugd2VyZSBhZnRlciBhYXJjaDY0IEh5cGVyLVYNCj4gPj4+Pj4+IHRlc3RpbmcgdnMu IG5hdGl2ZS1ib290IHRlc3RpbmcgdnMuIGJvdGguIFNvIEkgbWF5IGhhdmUgZ29uZSB0aGUNCj4g Pj4+Pj4+IHdyb25nIGRpcmVjdGlvbiBmcm9tIHRoZSBzdGFydC4NCj4gPj4+Pj4+IEl0IGlzIHBv c3NpYmxlIHRoYXQgSSdkIGZpbmQgZXN0YWJsaXNoaW5nIGEgbmF0aXZlLWJvb3QgZWFzaWVyDQo+ ID4+Pj4+PiBhbmQgdGhlbiBiZSBhYmxlIHRvIGhhdmUgYSBWTSBmaWxlIGNyZWF0ZWQgZnJvbSB0 aGUgbWVkaWEsIG1vcmUNCj4gPj4+Pj4+IGxpa2Ugd2hhdCBJIGRpZCB3aXRoIEZyZWVCU0QuDQo+ ID4+Pj4+Pg0KPiA+Pj4+Pj4gVGhlIFVidW50dSBhY3Rpdml0eSBsaWtlbHkgd291bGQgbm90IGJl IGFuYWxvZ291cyB0byB0aGUgRnJlZUJTRA0KPiA+Pj4+Pj4gYnVpbGRzIGhhdmluZyAtbWNwdT0g b3B0aW1pemF0aW9uIHVzZWQuDQo+ID4+Pj4+Pg0KPiA+Pj4+Pj4gQmFjayB0byAkd29yay4NCj4g Pj4+Pj4+DQo+ID4+Pj4+DQo+ID4+Pj4+IEkgZm91bmQgYSBzZXF1ZW5jZSBvZiBVSSBvcGVyYXRp b25zIHRoYXQgd29ya2VkIGZvciBpbnN0YWxsaW5nDQo+ID4+Pj4+IFVidW50dSBzZXJ2ZXIgMjIu MDQuMyBpbnRvIEh5cGVyLVYgaW4gV2luZG93cyAxMSBQcm8gb24gdGhlDQo+ID4+Pj4+IFdpbmRv d3MgRGV2IEtpdCAyMDIzIHZpYSB1c2Ugb2YgYSBkb3dubG9hZGVkICouaXNvIC4NCj4gPj4+Pj4N Cj4gPj4+Pj4gVGhlIGtlcm5lbCB0aGF0IHJlc3VsdHMgcHJlZGF0ZXMgNi4wOg0KPiA+Pj4+Pg0K PiA+Pj4+PiAkIHVuYW1lIC1hcA0KPiA+Pj4+PiBMaW51eCB1YndkazIzcyA1LjE1LjAtODItZ2Vu ZXJpYyAjOTEtVWJ1bnR1IFNNUCBNb24gQXVnIDE0DQo+ID4+Pj4+IDE0OjE5OjE4IFVUQyAyMDIz IGFhcmNoNjQgYWFyY2g2NCBhYXJjaDY0IEdOVS9MaW51eA0KPiA+Pj4+Pg0KPiA+Pj4+PiBVc2lu ZyBteSB1c3VhbCBydWxlIG9mIHJlYm9vdGluZyBiZWZvcmUgdGhlIGZpcnN0IHNjcDoNCj4gPj4+ Pj4NCj4gPj4+Pj4gJCBzY3ANCj4gPj4+Pj4gRnJlZUJTRC0xNC4wLUFMUEhBMi1hcm0tYXJtdjct R0VORVJJQ1NELTIwMjMwODE4LQ0KPiA3NzAxM2YyOWQwNDgtMjY0OA0KPiA+Pj4+PiA0MS5pbWcg bWFya21pQGxvY2FsaG9zdDpGcmVlQlNELTE0LVRFU1QuaW1nDQo+ID4+Pj4+IC4gLiAuDQo+ID4+ Pj4+IEZyZWVCU0QtMTQuMC1BTFBIQTItYXJtLWFybXY3LUdFTkVSSUNTRC0yMDIzMDgxOC0NCj4g NzcwMTNmMjlkMDQ4LTI2NDg0MS5pbWcNCj4gMTAwJSA1MTIwTUIgNDMxLjNNQi9zICAgMDA6MTEN Cj4gPj4+Pj4NCj4gPj4+Pj4gJCBybSBGcmVlQlNELTE0LVRFU1QuaW1nDQo+ID4+Pj4+ICQgc2Nw DQo+ID4+Pj4+IEZyZWVCU0QtMTQuMC1BTFBIQTItYXJtLWFybXY3LUdFTkVSSUNTRC0yMDIzMDgx OC0NCj4gNzcwMTNmMjlkMDQ4LTI2NDgNCj4gPj4+Pj4gNDEuaW1nIG1hcmttaUBsb2NhbGhvc3Q6 RnJlZUJTRC0xNC1URVNULmltZw0KPiA+Pj4+PiAuIC4gLg0KPiA+Pj4+PiBGcmVlQlNELTE0LjAt QUxQSEEyLWFybS1hcm12Ny1HRU5FUklDU0QtMjAyMzA4MTgtDQo+IDc3MDEzZjI5ZDA0OC0yNjQ4 NDEuaW1nDQo+IDEwMCUgNTEyME1CIDQ4Mi4yTUIvcyAgIDAwOjEwDQo+ID4+Pj4+DQo+ID4+Pj4+ IERlZmluaXRlbHkgZmFzdGVyIHRoYW4gdGhlIEZyZWVCU0QgcmVzdWx0cyB0aGF0IEkgcmVwb3J0 ZWQNCj4gPj4+Pj4gZWFybGllciwgaW5jbHVkaW5nIGZhc3RlciB0aGFuIHRoZSBUaHJlYWRSaXBw ZXIgMTk1MFggd2l0aCBPcHRhbmUNCj4gPj4+Pj4gaW4gYSBQQ0llIHNsb3QgKG1vcmUgbGlrZSAz MDAgTWlCeXRlcy9zZWMpLg0KPiA+Pj4+Pg0KPiA+Pj4+PiBJIGFnYWluIHVzZWQgNiBjb3Jlcywg MjQ1NzYgTWlCeXRlcyBvZiBSQU0sIGEgZml4ZWQgc2l6ZWQgdmlydHVhbA0KPiA+Pj4+PiBoYXJk IGRpc2sgdW5kZXIgSHlwZXItVi4NCj4gPj4+Pj4NCj4gPj4+Pj4gRm9yIHJlZmVyZW5jZToNCj4g Pj4+Pj4NCj4gPj4+Pj4gJCBsc2JsayAtZg0KPiA+Pj4+PiBOQU1FICAgRlNUWVBFICAgRlNWRVIg TEFCRUwgVVVJRCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIEZTQVZBSUwgRlNVU0Ul DQo+IE1PVU5UUE9JTlRTDQo+ID4+Pj4+IGxvb3AwICBzcXVhc2hmcyA0LjAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMCAgIDEwMCUNCj4gL3NuYXAv Y29yZTIwLzE5NzcNCj4gPj4+Pj4gbG9vcDEgIHNxdWFzaGZzIDQuMCAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAwICAgMTAwJSAvc25hcC9seGQvMjQz MjYNCj4gPj4+Pj4gbG9vcDIgIHNxdWFzaGZzIDQuMCAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAwICAgMTAwJQ0KPiAvc25hcC9zbmFwZC8xOTQ1OQ0K PiA+Pj4+PiBzZGEgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICDilJzilIBzZGExIHZmYXQgICAgIEZBVDMy DQo+IEY3RTktMTM0NCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDFHICAgICAxJSAv Ym9vdC9lZmkNCj4gPj4+Pj4g4pSU4pSAc2RhMiBleHQ0ICAgICAxLjAgICAgICAgICA0OGEwZGJl Ni01YTk5LTRiNmUtOTJkYy1mZTZkOGVmYzZmZmUNCj4gOTkuM0cgICAgMTQlIC8NCj4gPj4+Pj4N Cj4gPj4+Pj4NCj4gPj4+Pj4NCj4gPj4+Pj4gQW4gZXhwZXJpbWVudCB3b3VsZCBiZSB0byBoYXZl IGEgc21hbGwgYW1vdW50IGlmIFJBTSByZWxhdGl2ZSB0aGUNCj4gPj4+Pj4gZmlsZSBzaXplLiBU aGF0IHdvdWxkIGZvcmNlIGl0IHRvIGFjdHVhbGx5IHdyaXRlIHRvIG1lZGlhIGZvciBzb21lDQo+ ID4+Pj4+IHBhcnQgb2YgdGhlIGZpbGUgY29weS4NCj4gPj4+Pg0KPiA+Pj4+IFRoZSB3b3JkaW5n IHdhcyBwb29yOiAiZm9yY2UgaXQiIGhlcmUgaXMganVzdCBmcm9tIHRoZSBVYnVudHUNCj4gPj4+ PiB2aWV3cG9pbnQuIEkgbWFrZSBubyBjbGFpbSB0byBrbm93IGlmIEh5cGVyLVYgaXMgYWN0dWFs bHkgd3JpdGluZw0KPiA+Pj4+IHRoZSBtYXRlcmlhbCBvdXQgdG8gbWVkaWEgYXQgdGhlIHRpbWUg dnMuIGxhdGVyLg0KPiA+Pj4+DQo+ID4+Pj4+IFNvIHVzaW5nIDEwMjQgTWlCeXRlIG9mIFJBTSBh c3NpZ25lZCBpbiBIeXBlci1WOg0KPiA+Pj4+Pg0KPiA+Pj4+PiAkIHNjcA0KPiA+Pj4+PiBGcmVl QlNELTE0LjAtQUxQSEEyLWFybS1hcm12Ny1HRU5FUklDU0QtMjAyMzA4MTgtDQo+IDc3MDEzZjI5 ZDA0OC0yNjQ4DQo+ID4+Pj4+IDQxLmltZyBtYXJrbWlAbG9jYWxob3N0OkZyZWVCU0QtMTQtVEVT VC5pbWcNCj4gPj4+Pj4gLiAuIC4NCj4gPj4+Pj4gRnJlZUJTRC0xNC4wLUFMUEhBMi1hcm0tYXJt djctR0VORVJJQ1NELTIwMjMwODE4LQ0KPiA3NzAxM2YyOWQwNDgtMjY0ODQxLmltZw0KPiAxMDAl IDUxMjBNQiA0MDcuNU1CL3MgICAwMDoxMg0KPiA+Pj4+Pg0KPiA+Pj4+PiAkIHJtIEZyZWVCU0Qt MTQtVEVTVC5pbWcNCj4gPj4+Pj4gJCBzY3ANCj4gPj4+Pj4gRnJlZUJTRC0xNC4wLUFMUEhBMi1h cm0tYXJtdjctR0VORVJJQ1NELTIwMjMwODE4LQ0KPiA3NzAxM2YyOWQwNDgtMjY0OA0KPiA+Pj4+ PiA0MS5pbWcgbWFya21pQGxvY2FsaG9zdDpGcmVlQlNELTE0LVRFU1QuaW1nDQo+ID4+Pj4+IC4g LiAuDQo+ID4+Pj4+IEZyZWVCU0QtMTQuMC1BTFBIQTItYXJtLWFybXY3LUdFTkVSSUNTRC0yMDIz MDgxOC0NCj4gNzcwMTNmMjlkMDQ4LTI2NDg0MS5pbWcNCj4gMTAwJSA1MTIwTUIgNDA0LjdNQi9z ICAgMDA6MTINCj4gPj4+Pj4NCj4gPj4+Pj4gU3RpbGwgZGVmaW5pdGVseSBmYXN0ZXIgdGhhbiB0 aGUgRnJlZUJTRCByZXN1bHRzIHRoYXQgSSByZXBvcnRlZA0KPiA+Pj4+PiBlYXJsaWVyLCBpbmNs dWRpbmcgZmFzdGVyIHRoYW4gdGhlIFRocmVhZFJpcHBlciAxOTUwWCB3aXRoIE9wdGFuZQ0KPiA+ Pj4+PiBpbiBhIFBDSWUgc2xvdCAobW9yZSBsaWtlIDMwMCBNaUJ5dGVzL3NlYykuDQo+ID4+Pg0K PiA+Pj4gT25lIG1vcmUgdmFyaWF0aW9uIGluIHVidW50dSB1bmRlciBIeXBlci1WLCBzdGlsbCB3 aXRoIDEwMjQgTWlCeXRlcw0KPiA+Pj4gb2YgYXNzaWduZWQgUkFNOiB1c2Ugb2YgbG9jYWxob3N0 Oi9kZXYvbnVsbA0KPiA+Pj4NCj4gPj4+ICQgc2NwDQo+ID4+PiBGcmVlQlNELTE0LjAtQUxQSEEy LWFybS1hcm12Ny1HRU5FUklDU0QtMjAyMzA4MTgtDQo+IDc3MDEzZjI5ZDA0OC0yNjQ4NDEuaW1n IG1hcmttaUBsb2NhbGhvc3Q6L2Rldi9udWxsIC4gLiAuDQo+ID4+PiBGcmVlQlNELTE0LjAtQUxQ SEEyLWFybS1hcm12Ny1HRU5FUklDU0QtMjAyMzA4MTgtDQo+IDc3MDEzZjI5ZDA0OC0yNjQ4NDEu aW1nDQo+ID4+Pg0KPiA+Pj4gJCBzY3ANCj4gPj4+IEZyZWVCU0QtMTQuMC1BTFBIQTItYXJtLWFy bXY3LUdFTkVSSUNTRC0yMDIzMDgxOC0NCj4gNzcwMTNmMjlkMDQ4LTI2NDg0MS5pbWcgbWFya21p QGxvY2FsaG9zdDovZGV2L251bGwgLiAuIC4NCj4gPj4+IEZyZWVCU0QtMTQuMC1BTFBIQTItYXJt LWFybXY3LUdFTkVSSUNTRC0yMDIzMDgxOC0NCj4gNzcwMTNmMjlkMDQ4LTI2NDg0MS5pbWcNCj4g MTAwJSA1MTIwTUIgNDkyLjlNQi9zICAgMDA6MTANCj4gPj4+DQo+ID4+Pg0KPiA+Pj4gVGhlIG1h dGNoaW5nIEZyZWVCU0QgZXhhbXBsZXMgd2l0aCAyNDU3NiBNaUJ5dGVzIG9mIFJBTSBhc3NpZ25l ZA0KPiAoWkZTIGNvbnRleHQpOg0KPiA+Pj4NCj4gPj4+ICMgc2NwDQo+ID4+PiBGcmVlQlNELTE0 LjAtQUxQSEEyLWFybS1hcm12Ny1HRU5FUklDU0QtMjAyMzA4MTgtDQo+IDc3MDEzZjI5ZDA0OC0y NjQ4NDEuaW1nIHJvb3RAbG9jYWxob3N0Oi9kZXYvbnVsbCAuIC4gLg0KPiA+Pj4gRnJlZUJTRC0x NC4wLUFMUEhBMi1hcm0tYXJtdjctR0VORVJJQ1NELTIwMjMwODE4LQ0KPiA3NzAxM2YyOWQwNDgt MjY0ODQxLmltZw0KPiA+Pj4NCj4gPj4+ICMgc2NwDQo+ID4+PiBGcmVlQlNELTE0LjAtQUxQSEEy LWFybS1hcm12Ny1HRU5FUklDU0QtMjAyMzA4MTgtDQo+IDc3MDEzZjI5ZDA0OC0yNjQ4NDEuaW1n IHJvb3RAbG9jYWxob3N0Oi9kZXYvbnVsbCAuIC4gLg0KPiA+Pj4gRnJlZUJTRC0xNC4wLUFMUEhB Mi1hcm0tYXJtdjctR0VORVJJQ1NELTIwMjMwODE4LQ0KPiA3NzAxM2YyOWQwNDgtMjY0ODQxLmlt Zw0KPiAxMDAlIDUxMjBNQiAxOTguN01CL3MgICAwMDoyNQ0KPiA+Pj4NCj4gPj4+DQo+ID4+PiBO b3RlOiBBdCBtb3N0IG9uZSBWTSBydW5uaW5nIGF0IGEgdGltZSwgbmV2ZXIgYm90aCBpbiBvdmVy bGFwcGluZyB0aW1lcy4NCj4gPj4NCj4gPj4gQXZvaWRpbmcgaGF2aW5nIGEgY2lwaGVyIGludm9s dmVkIGFuZCBldmVuIGxvY2FsaG9zdA0KPiA+PiBpbnZvbHZlZDogdXNlIGRkIC4gLiAuDQo+ID4+ DQo+ID4+DQo+ID4+IEZyZWVCU0QgZXhhbXBsZXMgZm9yIFdpbmRvd3MgRGV2IEtpdCAyMDIzIEh5 cGVyLVYgY29udGV4dCwNCj4gPj4gMjQ1NzYgTWlCeXRzIG9mIFJBTSBhc3NpZ25lZCk6DQo+ID4+ DQo+ID4+ICMgZGQNCj4gPj4gaWY9RnJlZUJTRC0xNC4wLUFMUEhBMi1hcm0tYXJtdjctR0VORVJJ Q1NELTIwMjMwODE4LQ0KPiA3NzAxM2YyOWQwNDgtMjY0OA0KPiA+PiA0MS5pbWcgb2Y9L2Rldi9u dWxsIGJzPTFtIHN0YXR1cz1wcm9ncmVzcw0KPiA+PiAyNTEyMzg4MDk2IGJ5dGVzICgyNTEyIE1C LCAyMzk2IE1pQikgdHJhbnNmZXJyZWQgMS4wNDZzLCAyNDAyIE1CL3MNCj4gPj4gNTEyMCswIHJl Y29yZHMgaW4NCj4gPj4gNTEyMCswIHJlY29yZHMgb3V0DQo+ID4+IDUzNjg3MDkxMjAgYnl0ZXMg dHJhbnNmZXJyZWQgaW4gMS42MjcwNzEgc2VjcyAoMzI5OTYxNDc3MCBieXRlcy9zZWMpDQo+ID4+ IENBNzhDLVdESzIzcy1aRlMgYWFyY2g2NCAgMTUwMDAwMCAxNTAwMDAwICMgZGQNCj4gPj4gaWY9 RnJlZUJTRC0xNC4wLUFMUEhBMi1hcm0tYXJtdjctR0VORVJJQ1NELTIwMjMwODE4LQ0KPiA3NzAx M2YyOWQwNDgtMjY0OA0KPiA+PiA0MS5pbWcgb2Y9L2Rldi9udWxsIGJzPTFrIHN0YXR1cz1wcm9n cmVzcw0KPiA+PiA1MjMzNTA5Mzc2IGJ5dGVzICg1MjM0IE1CLCA0OTkxIE1pQikgdHJhbnNmZXJy ZWQgMTQuMDIycywgMzczIE1CL3MNCj4gPj4gNTI0Mjg4MCswIHJlY29yZHMgaW4NCj4gPj4gNTI0 Mjg4MCswIHJlY29yZHMgb3V0DQo+ID4+IDUzNjg3MDkxMjAgYnl0ZXMgdHJhbnNmZXJyZWQgaW4g MTQuMzY1MTQyIHNlY3MgKDM3MzczMTcxNCBieXRlcy9zZWMpDQo+ID4+IENBNzhDLVdESzIzcy1a RlMgYWFyY2g2NCAgMTUwMDAwMCAxNTAwMDAwICMgZGQNCj4gPj4gaWY9RnJlZUJTRC0xNC4wLUFM UEhBMi1hcm0tYXJtdjctR0VORVJJQ1NELTIwMjMwODE4LQ0KPiA3NzAxM2YyOWQwNDgtMjY0OA0K PiA+PiA0MS5pbWcgb2Y9L2Rldi9udWxsIGJzPTUxMiBzdGF0dXM9cHJvZ3Jlc3MNCj4gPj4gNTI4 NTQxMDgxNiBieXRlcyAoNTI4NSBNQiwgNTA0MSBNaUIpIHRyYW5zZmVycmVkIDI3LjAyOXMsIDE5 NiBNQi9zDQo+ID4+IDEwNDg1NzYwKzAgcmVjb3JkcyBpbg0KPiA+PiAxMDQ4NTc2MCswIHJlY29y ZHMgb3V0DQo+ID4+IDUzNjg3MDkxMjAgYnl0ZXMgdHJhbnNmZXJyZWQgaW4gMjcuNDMyNTcwIHNl Y3MgKDE5NTcwNTY1NyBieXRlcy9zZWMpDQo+ID4+DQo+ID4+DQo+ID4+IFVidW50dSAyMi4wNC4z IGZvciBXaW5kb3dzIERldiBLaXQgMjAyMyBIeXBlci1WIGNvbnRleHQsIG9ubHkgMTAyNA0KPiA+ PiBNaUJ5dGVzIG9mIFJBTSBhc3NpZ25lZDoNCj4gPj4NCj4gPj4gJCBkZA0KPiA+PiBpZj1GcmVl QlNELTE0LjAtQUxQSEEyLWFybS1hcm12Ny1HRU5FUklDU0QtMjAyMzA4MTgtDQo+IDc3MDEzZjI5 ZDA0OC0yNjQ4DQo+ID4+IDQxLmltZyBvZj0vZGV2L251bGwgYnM9MU0gc3RhdHVzPXByb2dyZXNz DQo+ID4+IDQwMDM0NjMxNjggYnl0ZXMgKDQuMCBHQiwgMy43IEdpQikgY29waWVkLCAyIHMsIDIu MCBHQi9zDQo+ID4+IDUxMjArMCByZWNvcmRzIGluDQo+ID4+IDUxMjArMCByZWNvcmRzIG91dA0K PiA+PiA1MzY4NzA5MTIwIGJ5dGVzICg1LjQgR0IsIDUuMCBHaUIpIGNvcGllZCwgMi41NjM0MiBz LCAyLjEgR0IvcyAkIGRkDQo+ID4+IGlmPUZyZWVCU0QtMTQuMC1BTFBIQTItYXJtLWFybXY3LUdF TkVSSUNTRC0yMDIzMDgxOC0NCj4gNzcwMTNmMjlkMDQ4LTI2NDgNCj4gPj4gNDEuaW1nIG9mPS9k ZXYvbnVsbCBicz0xSyBzdGF0dXM9cHJvZ3Jlc3MNCj4gPj4gNDc5Mzg2NTIxNiBieXRlcyAoNC44 IEdCLCA0LjUgR2lCKSBjb3BpZWQsIDYgcywgNzk5IE1CL3MNCj4gPj4gNTI0Mjg4MCswIHJlY29y ZHMgaW4NCj4gPj4gNTI0Mjg4MCswIHJlY29yZHMgb3V0DQo+ID4+IDUzNjg3MDkxMjAgYnl0ZXMg KDUuNCBHQiwgNS4wIEdpQikgY29waWVkLCA2LjYwNDAzIHMsIDgxMyBNQi9zDQo+ID4+IG1hcmtt aUB1YndkazIzczp+JCBkZA0KPiA+PiBpZj1GcmVlQlNELTE0LjAtQUxQSEEyLWFybS1hcm12Ny1H RU5FUklDU0QtMjAyMzA4MTgtDQo+IDc3MDEzZjI5ZDA0OC0yNjQ4DQo+ID4+IDQxLmltZyBvZj0v ZGV2L251bGwgYnM9NTEyIHN0YXR1cz1wcm9ncmVzcw0KPiA+PiA0ODAwMTAyOTEyIGJ5dGVzICg0 LjggR0IsIDQuNSBHaUIpIGNvcGllZCwgOSBzLCA1MzMgTUIvcw0KPiA+PiAxMDQ4NTc2MCswIHJl Y29yZHMgaW4NCj4gPj4gMTA0ODU3NjArMCByZWNvcmRzIG91dA0KPiA+PiA1MzY4NzA5MTIwIGJ5 dGVzICg1LjQgR0IsIDUuMCBHaUIpIGNvcGllZCwgOS45NTYwNiBzLCA1MzkgTUIvcw0KPiA+DQo+ IA0KPiA9PT0NCj4gTWFyayBNaWxsYXJkDQo+IG1hcmtsbWkgYXQgeWFob28uY29tDQoNCg== From nobody Thu Aug 31 06:16:52 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RbrY55255z4rn0s for ; Thu, 31 Aug 2023 06:17:13 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic301-20.consmr.mail.gq1.yahoo.com (sonic301-20.consmr.mail.gq1.yahoo.com [98.137.64.146]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RbrY50kXMz4LLH for ; Thu, 31 Aug 2023 06:17:13 +0000 (UTC) (envelope-from marklmi@yahoo.com) Authentication-Results: mx1.freebsd.org; none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693462630; bh=f7jJxsq9YMt+FEU02hVvSlRP5PpxXNqMNUmzdBHsueE=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=ZFStYIbikTntEVQtFA2dDyZsTHc6YYIWZ7oik7cGCzt+18CRZkV7Gj3jvnaS3F89RKEby3qFMlfPmZ5NKkDipMomE0n32uGFIs4G/miPm2x+5JfVeCfBlfV/QfpkYEXG6Cj2BV27QmY8mKUH5GPegWmsK7Sv6IYa3gnttCT43nl9XbsF2YT9WGlEaPeZTohEsFOKeB5+aMpWOy/+LrPWHMOURkUIp77pXlXy/Ag/7Q5YD46nY9e1chIQAJFjE4QzC6WEmjT4J16vsHuNdwHy410U8reifi6J9t3X+2PAFJnk4Nlfv7l5ZGmHtar06EawfJACgmaNWa4bLZpafYL57w== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1693462630; bh=bB4/XL452EM3fWMN4Q/WpZSi37PL9rvs3Z1cVD2Ki9w=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=flX0IylnGrvEB7ZUfqxX9UUm0CSvYt3r77GT4Z0OVooXWOkHLqC0F4GToM1F2PaXiimeAZSB4PkfjistRrId70gi5H/0+duquwiJf6RkknjafqwVTsfalLalEs0AnvAGBUX5Xc2pTPcpS37BFvM97o4cSpUMdp/0mPAF8lHjK4Zsf+JiKb4dAOy/2ouKiOS2A1WLNC0G94SbN6aICvIat/agZXis2OCnGEDfVUsk1Mzjze8TCrlBLciBbRT21ngEJ9Fnx2ZOENvibPSB1d9cgQRLhJdsj8OzWWMIgZoJHLef/o3Ewjw1KUGh1P2evjy1yRvq66PExuokB6nmK366bA== X-YMail-OSG: VI3UhWMVM1nEtR3C7E6.pYCZIpM3s3ySn8PXXsZZAcHRZh8XMSkQ8peJhLWR5gq mj8.DrbQOyrkRToiwOjwNsvhWnZH5XQ3geu3Buke3xCGG.1nF4ZyiVIibKtTpTejNpOi_9FgwJKK iftLySrfyKo1CJSvnoLgmUvFHY_x1DfRePYe6HzFp562mLEdUIHJguI33UrOCxrY31sRh8Ku9xu7 GoDB6igKRfW04nkqPopt1.bbXBQAq1vWpBOUUO7DdpRYUtWQITgOC.zmBpEaOUMzK4Yr5Jx5b7r6 o85lR7WdPvT.PXphfU5Xh8u7GmAgtGiFggihB8LQejA5Cs_tong97afhuFwHEHfQSCmmIXrHL1fO CCmR6bkpKpwth4foDWRDgbrAsr5AClrScN_FcXpBkHb8ifpWnyfDcwhMY9hy3QKzXmxPxqpNJFeU QZhWsTfxtNZJK2m3sXwGESHxRu71fclF8s3OPVl3EqaDVPCtbimGfxpGBdvXBwjUB8.B0i_jVdUZ iQl6at10qqyeUHnUnjaqYm33QAzRGRsd7RdnbNfNB9tR.eORGTa9d9kQ0DH9hhZCaD6YW71Rr4Az 0b4QW4WZaEe0myjnV44al51cfeiFdYS4wLIdGqG8Fl62oWlD6PDp8hPHTLnSILXEJP7KnqY6Ifmi bb9oubgVlKMHOATulaEzwrp2kO8vlujbmmRVVxlb2PhZeCa4iplLtfzC4bdVFyBcg.2eP6LT4079 dZ1C.TrRKz3ZKCzPslaIb6KYcrQpFyYwmC86WkP08JcixFqJNJoi4b4GlrhHlBr2TjuXjFs6iTPX iaOwkpTfcZOewEUi0hqpDF4binGW.xHiqxip1Rrv3gbaRCVQbh09ssV.DvCHBLN..Ymb4c7q55B2 c2.T8rkSdoYGZJtB6h_H8qLWuFeHhTBN6uJHZZa2LMqXxfWJIPgEr2A8jge6UWcKBErYILROzhzZ rgTna6kuq0J.sm3_JlApUo7EG6IdKphM2ciXixPsfXHO5Cn7nIMsWiXblJ3pOTe6851ZX1A4i0hu 56kSXQc4Q7N992LGw0xMp38O09c6WsQjd1d_H8Z.UfrXo7BT4svBOUbFuaeUjovt6j5QPKt8pS2t y_6E1poM7K.D76Q4eElinyWPEuGThuPSyazchVgSEp2sGuhRZFWKlB4JLisFEJk_g11Iy0XgFDEY xh5.5dCLpCMn6fJ8SEpQ8hqC6xi1AOHmQzj9eR8M8NzD6WxXYqr2lkSpitYss6vAXkWgZJVB2bjL AtGklWG4VP32zaW50OKoPsh8sB3dYL10w6ZlY1.c_1XYBkmJjlPZU2j1OJrVgGRM7tZn5Bl71MeL pMo_q8cYKTbLdNHPGEf2TkEBEQGfw828tPznGPjCpsDRcmtLFG_NWwDlrC5YrzGoioDTTcrilL3m bbeKbX7iTQxNkXHMtIzFo0E4LByp8plBBY2Tz.840zZmwLgfDtybfRyQfdPX9cWe1aIpuMQG_foO tfMKnok.U90CTJu4syLVQ1_4_o0Pbe3ma8mEG0IuSvk4iEZvHxn_RHO7XraqUaoalOxadvmxj_1h PXSH.56HvcRjBH2HsXfAU5iRzYuYiOY1lXJN3voP6FMfByN9QJlYncCvobZM8mj55u66aLzPmK8i AwBjh49lE28_jtmeIIzGiJZok03AHwyzVjebyM55.PGZyf3MakKSiAVVXPaEFJrEkAfIsJQRUMlw w1z5S6QIyR9dDwXn_Z5v.Su5ab.tMtn36bWmXsCd31._uP3Z_xX7QuIh6dVoi6agoawkQaEdBpu6 jBxc4DvvP9oKLRoig.BQymSzPHm6yA3WhkAFyYF6fdX9ZZDx.OxvBZsus4EA12JsAlVY7XXXmpEi g_gH9LmRIsxW4GGFkv0l6UqH00ZZYIZU_SBW5BoA522Oh6bi3yOsQ55Rihp5PCsXxX8aBahQDHmK ZQ_Ngms0Lkg9b4.VerAzh2oXmRTJaj.ohRmyjcYHqGYYrztjf3lA.aueRJHSRu6JvZCRZmFx4n5Y NepWOLUuZUFE7F5R.wZrfa8Tbne8OvUzTOrPFvGism_PX4ipCZ3LLSn93cjnOhTfCd5JgnSU2s6p 4dLad_0G1GpYTULIbSLJPAf2Ifm6Dke51JTppfv03KTu9oAdkDGoZb9_ziSemV.l0BOHNNzD2Dhe LEC1dIg6jxNeSA80.lhaWb_goq_6gf8rlKLfLfKRlYs2SM6Wb5HgCL8iVp5CyvmMdPAQgc7hg2Sz ApOttqQSJ5gJQwAz8t7riQeMWjIrdl1R47xYM5Yh8mFwoMOU27neND5OeI7sFKviiCT4v.FgxCaQ - X-Sonic-MF: X-Sonic-ID: 6830c2fe-9ced-41d5-82ef-2c5784f6789d Received: from sonic.gate.mail.ne1.yahoo.com by sonic301.consmr.mail.gq1.yahoo.com with HTTP; Thu, 31 Aug 2023 06:17:10 +0000 Received: by hermes--production-bf1-865889d799-r6v2w (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 69861597924a96dff4bcabb35f394169; Thu, 31 Aug 2023 06:17:05 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.700.6\)) Subject: Re: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] From: Mark Millard In-Reply-To: Date: Wed, 30 Aug 2023 23:16:52 -0700 Cc: Wei Hu , FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> <543FBABC-C75A-4AB0-BFB6-1C7C15ECBA4E@longcount.org> <96A99928-4E8E-4AB1-BF3A-D026B280EC0E@yahoo.com> To: Mark Saad X-Mailer: Apple Mail (2.3731.700.6) X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US] X-Rspamd-Queue-Id: 4RbrY50kXMz4LLH On Aug 30, 2023, at 20:59, Wei Hu wrote: >> -----Original Message----- >> From: Mark Millard >> Sent: Thursday, August 31, 2023 10:18 AM >> To: Mark Saad >> Cc: Wei Hu ; FreeBSD Hackers > hackers@freebsd.org> >> Subject: Re: Very slow scp performance comparing to Linux [dd to = /dev/null >> shows substantial FreeBSD vs. Ubuntu differences for bs=3D1k (or 1K) = and >> bs=3D512] >>=20 >> On Aug 30, 2023, at 18:45, Mark Saad wrote: >>=20 >>> All >>> Why not take scp out of the picture and try iperf? Why , we could be = looking >> at rss by default in Linux . >=20 > Actually I did the iperf3 test as well and posted results a couple = days ago.=20 > Pasting here: >=20 > FreeBSD iperf3 to localhost, single stream: 30.9 Gb/s=20 > Linux iperf3 to localhost, single stream: 48.8 Gb/s I'll note that (48.8Gb/s) / (30.9Gb/s) is a much smaller ratio than then either of the originally reported Ubuntu/FreeBSD ratios: (550 MB/s) / (70 MB/s) or: (350 MB/s) / (70 MB/s) Suggesting one or more non-network issues are contributing, especially vs. the localhost test that had the 550/70 ratio. > Neither of them has any tcp retry.=20 >=20 > Both VMs run on Intel(R) Xeon(R) Platinum 8473C (2100.00-MHz K8-class = CPU).=20 > The FreeBSD VM is 16 vcpu, with 128 GB memory. > The Linux VM is 4 vcpu, with 32 GB memory. >=20 > Wei >=20 >=20 >>=20 >> . . . =3D=3D=3D Mark Millard marklmi at yahoo.com From nobody Thu Aug 31 07:55:48 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rbtl10NKkz4rsN6 for ; Thu, 31 Aug 2023 07:55:57 +0000 (UTC) (envelope-from weh@microsoft.com) Received: from HK2P15301CU002.outbound.protection.outlook.com (mail-eastasiaazon11020018.outbound.protection.outlook.com [52.101.128.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "DigiCert Cloud Services CA-1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rbtl01CKmz4T4f for ; Thu, 31 Aug 2023 07:55:56 +0000 (UTC) (envelope-from weh@microsoft.com) Authentication-Results: mx1.freebsd.org; none ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=a/B+ftNafQMsQLNaXvtfT67hSTeV4StavR59PYW1mt4G/3CfZpc4XFecMAN7Zj5+j2MyXrTYIcaYGNt2d18F9GjrF78DPfb6QYCAS2vzdiJuLI4wSiM13cDLKFrCDDGNJh+AWwQHrDLmPsMkdI++39ig9qcRNd3TKbRGYamgr2d2qOTxWDjD9QWPFjtepvlSTJbWiyznHFwxF6G46tV7xhfUb5J2YbV/zgz+5T1lDTq2kHMMs8arw3t4ZtNIVVSUVmBujrdxDc8C8JJaKpTN92ERXuOvwBcAUFhE0DCGbJSQGn4QRpcy7/EwQQr5k/0kgD/1M7eFxJ4bCvo/ibkimA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=onkzgMt7BMPT0DzjghAPN/j4XbjBPrku3rOti1Ynqj8=; b=iCai6wgevGw1GPQxg2BA2TG/9PRFa68rm2gxk8yhCLoIcM4z/st86hQuC24BCH/Mvj55JxXqROQNzZVd30+t8itdDMv8q8TmgVYInIyaoInQVc9sCeyiz3ueW1Xozb0Zpmsi5sfocy8jhp1Z2D4/QPaYfhwBXwBX/CsJwbUy2mKZHvj++WxKIyooLf/juApVFl1hurZDcm39HIMFL8SmgpaDZyZEKykGDmX+RfaWo72Q/SxFGGA8gRZGfy81aJZjZY1um5UEBlGBNL1kwHR2gG1vmhV4PYc7k9GYsSdljlApsaD0xGjtz3tHymi7Ih4cTwz2mbH962Hw136Gy869LA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=microsoft.com; dmarc=pass action=none header.from=microsoft.com; dkim=pass header.d=microsoft.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=onkzgMt7BMPT0DzjghAPN/j4XbjBPrku3rOti1Ynqj8=; b=RkjyOQ7ktds2eabqcEjHVHFhj+1OeUPlwwUIITsI3oRf2A+8T7idJsJSldsTaOxYkXXz+83NNMAoyH3DW5T+yftfyv1u3OJ52LEds3ul45ZsMjSQcU089QXKLUxUQboVIJrANx6h1TTU80teHKIs7Ck8c+hl5QbgatAZctfOIFY= Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM (2603:1096:4:fc::7) by PSAP153MB0472.APCP153.PROD.OUTLOOK.COM (2603:1096:301:93::9) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.6768.15; Thu, 31 Aug 2023 07:55:49 +0000 Received: from SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83]) by SI2P153MB0441.APCP153.PROD.OUTLOOK.COM ([fe80::7bae:a915:919b:6f83%7]) with mapi id 15.20.6768.012; Thu, 31 Aug 2023 07:55:49 +0000 From: Wei Hu To: Mark Millard , Mark Saad CC: FreeBSD Hackers Subject: RE: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] Thread-Topic: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] Thread-Index: AQHZ2597zvq5AgzMUkaYwDpn0euEM7ADomqAgAAJDgCAABvOYIAAJvIAgAAaK8A= Date: Thu, 31 Aug 2023 07:55:48 +0000 Message-ID: References: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> <543FBABC-C75A-4AB0-BFB6-1C7C15ECBA4E@longcount.org> <96A99928-4E8E-4AB1-BF3A-D026B280EC0E@yahoo.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: msip_labels: MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ActionId=37591bed-2f46-4824-85e6-dfff40cbf447;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_ContentBits=0;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Enabled=true;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Method=Standard;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_Name=Internal;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SetDate=2023-08-31T07:50:31Z;MSIP_Label_f42aa342-8706-4288-bd11-ebb85995028c_SiteId=72f988bf-86f1-41af-91ab-2d7cd011db47; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: SI2P153MB0441:EE_|PSAP153MB0472:EE_ x-ms-office365-filtering-correlation-id: e2c03832-7a89-4d3d-ef22-08dba9f7b079 x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: pOIsmdjJhjR7Y8XGJcRbRBp+WRtZBW6z/5uCJ86+D5qz23F/rXU+2fvkhaug+xNn0kJHfnipOzqWnT/5V6+HuUbByaGzBZ+BSFexOvUgJ8iR+9gdS8vchgyuSBMkJuc18xXJwDufRv+7NHoXzB8wdlq9PkT+z+XjSf09Kx1ZBWeDgOIOBKBoun2Hx2JYGfo5xd0tsbIARqgeSSJ/vCesmDeJwwQUgkRhp2qVUU6BQuUZguR28J4Woz3wwSZRaC6ejdrOYCdLnDWmXJsmh7lVk6aTF1BJMaj/uUObPn7uxTqYc0rj6nhTJFAOmnQRVJItJc/yiMB8UILI+tqoU9STjYHN1B7+FwV98vO04Po826Zktpm4e/4kJGaXNNVGHbAnahqVCfQXYgMHxKJPA4pn3sB+IUsNXG1/7xbAPBJCRuxnHDTaRoRF7U0eQkKj1QAiBbjv15FAvUO632lVYGqx+zPshO0l4LyigI2x43v68kMt6AoJwH/mjmwfKneDcisZowYCtbNsFFMZp288JLWRxbEi2V2L0BEHkdqID/7Gq7sf1RdV4Wmt25zQyuhjvT4FQsiaL4npm6jI3rSfb/22Tkey9n4G3sSNeuvJY2IttKWLyGbsHdCrcgRW4kw8BQRxy+cMAj0yu7xA3AU48EBVNeVHagUPCEIXWmbywtuGD6s= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:SI2P153MB0441.APCP153.PROD.OUTLOOK.COM;PTR:;CAT:NONE;SFS:(13230031)(39860400002)(396003)(346002)(366004)(376002)(136003)(186009)(1800799009)(451199024)(82960400001)(2906002)(83380400001)(55016003)(82950400001)(122000001)(8990500004)(38070700005)(38100700002)(53546011)(110136005)(7696005)(6506007)(316002)(76116006)(66946007)(64756008)(66446008)(66476007)(66556008)(71200400001)(9686003)(41300700001)(10290500003)(33656002)(478600001)(86362001)(26005)(5660300002)(4326008)(52536014)(8676002)(8936002);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?us-ascii?Q?LGjYPBQW+C7LKXBGNZJSWxt72gL5IiWh5Nb2QydZ/YBSw3MDaKeW4dawwm6q?= =?us-ascii?Q?1oXyGYGwgc/LNctpvOdKfpLzYuwr/1YZaEM9zxukbWxwpH08TcYzzV7LJ3lU?= =?us-ascii?Q?ySOTXBFQCTEOi8pS8TMGBETQ6Dl93+GvcgJVvrbZD4JVGNWpc4SO0/Cc+TXa?= =?us-ascii?Q?ct4J+BNhhiCiSJq2Nrp0K8DSp+bw4mYQBnto5F/QGE9IQXXx4SIUBCAmSQWI?= =?us-ascii?Q?RN3CGBSvvg9SLMIn9AcHdHnRNLsgj8TEwXPscwhv7x1Q7Igaw4RWzsIU45uk?= =?us-ascii?Q?Nc99GJPYqTLw7jKXkr4yUNOln8w0CR+2i6bcfsTxP+Iwmiq2ksX3A1rLek9s?= =?us-ascii?Q?NrgqgqqCLrucbC2qYpSlbanDU+K/kiskI0WBm1qr54yKechATO8raw0Wm+GA?= =?us-ascii?Q?3SJE4g5spVQ+ZbJeO2TbNFUB1fyC5h4Eyne+5XYbs5DeACezyRpFZFSja0X9?= =?us-ascii?Q?AP1BnaJNQS+oFP97lUuDYrwgpoRif0rDvuKxOHERTpxRHPVjUF6qs9FTwrSF?= =?us-ascii?Q?NIkSbaSCw6zdMVyrw+HAo3tzAOzNW+ubktVdIXj3U4sMzXAUDWpU/55ZLRR1?= =?us-ascii?Q?XljNXnTxcAXM2t1hBXELYHf0ewrYO7MUiyETGLWUu9iwrG8m6g6vcBxkP6cJ?= =?us-ascii?Q?lzSOf5hE2YVPZ14A4n9jpUNjpM8MMeTJV5x4cDFKzDTMnZikvGr7iujdRj9H?= =?us-ascii?Q?XWjJPwuBh0OyVr6nZXOY0PolobbiO764vIHJAFaDkoRV77GBHEYnDjqSdbzl?= =?us-ascii?Q?UqNGU2cujF1iD/J5oASdggHTVzuhS0JB7OC2KAwSmIPZmgXvl2VHDxAjjFIN?= =?us-ascii?Q?QO5kzpJxut4FRqMbGT8lViCD5rNuvJ9juSkvrrtRoT0GHoiRRHmQUmSTE8ya?= =?us-ascii?Q?pJl0kzbNlNSYxI9uQzNTV1gRL7JZgbPcoAlyugWCq4Srh88m2XggNwfebtT3?= =?us-ascii?Q?4iKxWX5Gkr79h4YdTjCcLhe2N2W10sU3xFwObtJTRduJLCsgvn3cTfMSiIYM?= =?us-ascii?Q?fVMMgRhkZFnDW+qpFa2w32GdX3wnxy6NNq+eUrrtcT0aKwRwaqG1GybDRUsO?= =?us-ascii?Q?CefY4L+m3Cm89wMa8NwJEAtTGWl1MurN6hA6Fl2268tvXe4o66DwYlbdQcdu?= =?us-ascii?Q?5L4f8TYSGbz04qS2nCuaUBXniGjno1Ssg7DVns0iqIR6sLYgrcCf0H2Pgrnr?= =?us-ascii?Q?RHghvWf5Z0CiAy2QqGtX0ilm+T4umzX0XbrBE+gVOR3pgezLdkezuZtx1uAf?= =?us-ascii?Q?ko+mom73OhZBVvg6oiBCm6e/agVPEZSXC2vswo0QhWgwoQBlYDjxG/gLGZJq?= =?us-ascii?Q?FiiMHiDl/Ou2ns4gHXekNSotCEpWxzoIXi7a+GDQLCxOJuvz1CTQHUL8wwtg?= =?us-ascii?Q?JiqMlD90XibZGDJQrBEnRi6tX1ilrW43ShV3NDRZ4RUwhoN6fVbJuSZuXbfs?= =?us-ascii?Q?3dOOiyCZiW+J4bG+Bi71iVmxtArz1HS/gv09YZpJnfvgMzEUJmND/eE5UIgD?= =?us-ascii?Q?8ffKqEArwqABdcCo41YVHOyqBqbtftp3+HPWBjdP8ig+B6f4QiSKxzao4bAD?= =?us-ascii?Q?asuhMGz62AiPxQP1vsE=3D?= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 X-OriginatorOrg: microsoft.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: SI2P153MB0441.APCP153.PROD.OUTLOOK.COM X-MS-Exchange-CrossTenant-Network-Message-Id: e2c03832-7a89-4d3d-ef22-08dba9f7b079 X-MS-Exchange-CrossTenant-originalarrivaltime: 31 Aug 2023 07:55:48.5851 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 72f988bf-86f1-41af-91ab-2d7cd011db47 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: j3Z1zOzDlnUU9w+/QMTqcyA283bz6mdibqEaP48iqmBkBpfFP8cJvpKDpEBaxVOnXaPkvV/sLqm9X0AK8vFrww== X-MS-Exchange-Transport-CrossTenantHeadersStamped: PSAP153MB0472 X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:8075, ipnet:52.96.0.0/12, country:US] X-Rspamd-Queue-Id: 4Rbtl01CKmz4T4f > -----Original Message----- > From: Mark Millard > Sent: Thursday, August 31, 2023 2:17 PM > To: Mark Saad > Cc: Wei Hu ; FreeBSD Hackers hackers@freebsd.org> > Subject: Re: Very slow scp performance comparing to Linux [dd to /dev/nul= l > shows substantial FreeBSD vs. Ubuntu differences for bs=3D1k (or 1K) and > bs=3D512] >=20 > On Aug 30, 2023, at 20:59, Wei Hu wrote: >=20 > >> -----Original Message----- > >> From: Mark Millard > >> Sent: Thursday, August 31, 2023 10:18 AM > >> To: Mark Saad > >> Cc: Wei Hu ; FreeBSD Hackers >> hackers@freebsd.org> > >> Subject: Re: Very slow scp performance comparing to Linux [dd to > >> /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=3D1k > >> (or 1K) and bs=3D512] > >> > >> On Aug 30, 2023, at 18:45, Mark Saad wrote: > >> > >>> All > >>> Why not take scp out of the picture and try iperf? Why , we could be > >>> looking > >> at rss by default in Linux . > > > > Actually I did the iperf3 test as well and posted results a couple days= ago. > > Pasting here: > > > > FreeBSD iperf3 to localhost, single stream: 30.9 Gb/s Linux iperf3 to > > localhost, single stream: 48.8 Gb/s >=20 > I'll note that (48.8Gb/s) / (30.9Gb/s) is a much smaller ratio than then = either > of the originally reported Ubuntu/FreeBSD ratios: >=20 > (550 MB/s) / (70 MB/s) > or: > (350 MB/s) / (70 MB/s) >=20 > Suggesting one or more non-network issues are contributing, especially vs= . the > localhost test that had the 550/70 ratio. >=20 Yes, the network perf gap is not as big as I originally thought. Seems diff= erent factors contribute to the slowness of scp on FreeBSD. Network stack is only= one of them. Wei From nobody Fri Sep 1 00:00:25 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RcJ7t0bV1z4rmYR; Fri, 1 Sep 2023 00:00:26 +0000 (UTC) (envelope-from salvadore@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RcJ7s3wGwz3XKs; Fri, 1 Sep 2023 00:00:25 +0000 (UTC) (envelope-from salvadore@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693526425; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc; bh=uxm8M3vDBP5+TYcFpoGJwp4tMFUO7XjSZkedYQJCFpg=; b=d1KUVDMGMzjr4TDiPREFxX8/EsQdZTGWNjTmAU6DdfJbzhl+cTWzZiQ4LDcQ+XivTf4keM 9JTGPoRN4TxpEcy/QMZ7/7OuyHZwq1jWKsNU7vHmf0TL93hAWhtuxIMQnjJ494LBUBYeTt WjpiAWVG9DVsG4+TB7AgM/BTL406sBN3/pPgdxepI7H9bcSuh7Gsrwl179lx+PhZ3YTcqj rO1CV3NFnLyhJjFl1fcmb4AiQShftH2iuGW5ceTB8wpozluGzgmq5CEhAQ7ODYKudZVYhk ksUmvieKJ39vCZQW31H/xz9QviylGaZfWrgpkrwuHgEEcfsOXR14lPNu3nIGeQ== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693526425; a=rsa-sha256; cv=none; b=bEtSucEn0IoF94zKbmCGD7TyYGujf/eYxXUULucgDjHyHXi8CZTqUUBnKXKnMyWN4LXEQ6 xINKF9zN5pApOvCR47pyZJI+2EoMvau0j9SCNhPC+rRRMTZY6zsO49dSKgbUzTQH1R+pHX /LYb67+THBe0ntKJz5UbqQyMos4zbhAsLFHFCRudEBzH3wx6Dc7osKRJtJQ+7K6iOduTiS 6TVqU+fL2i3bwREyA7bwHRcfsbMW8wNW2aLBg5OSvrHmKIuonVkW7nfMsihlNuiJBEwMvH qOkqfM7lmoSzHvWgtnb8SV0HEZHZfteRcejmzFa3jZzsv1gy63HhY6iO+CjTnA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693526425; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc; bh=uxm8M3vDBP5+TYcFpoGJwp4tMFUO7XjSZkedYQJCFpg=; b=Tzsr/HoIOAMNwAwYt6z9fiicl/OodxcskGHV0mB3Cm8JVDEO7KWoydgc6VBsEeJ3ChNyxK XAuqikVmlCp51VH5wYQlgHWS+5aQ+lSnVNIlSuBvWoZ9UTLllU/qIbUO/Tc1xSieCb9pqc BATzsHGth3VMDmUFDnvRKjaZgOrc9Gz7wCd1pdgSt/LA1TqI+epKzuiyZaZ3FJoCT23x4Y l5Omh4X1ThSmRzcd0/E7waIRM/asRq0ZEYYnXbvMEdXuWgfcjqBR9qOXHDdWbg5MhUqvgk FzmGFNR0ixZD5TzLB7aF01E3AZHQiI1jQefsZdwwRk/YKDiIaSPQ8fUDow1PLA== Received: by freefall.freebsd.org (Postfix, from userid 1472) id 4A0441EB23; Fri, 1 Sep 2023 00:00:25 +0000 (UTC) To: freebsd-status-calls@FreeBSD.org Subject: Call for 2023Q3 status reports Cc: freebsd-current@FreeBSD.org,freebsd-hackers@FreeBSD.org,devsummit@FreeBSD.org,soc-students@FreeBSD.org,soc-mentors@FreeBSD.org Message-Id: <20230901000025.4A0441EB23@freefall.freebsd.org> Date: Fri, 1 Sep 2023 00:00:25 +0000 (UTC) From: Lorenzo Salvadore List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Dear FreeBSD Community, The deadline for the next FreeBSD Status Report update is September, 30th 2023 for work done since the last round of quarterly reports: July 2023 - September 2023. I would like to remind you that reports are published on a quarterly basis and are usually collected during the last month of each quarter, You are also welcome to submit them even earlier if you want, and the earlier you submit them, the more time we have for reviewing. Status report submissions do not need to be very long. They may be about anything happening in the FreeBSD project and community, and they provide a great way to inform FreeBSD users and developers about work that is underway or has been completed. Report submissions are not limited to committers; anyone doing anything interesting and FreeBSD related can -- and should -- write one! The following methods are available to submit your reports: * submit a review on Phabricator and add the group "status" to the reviewers list. You should put your reports in the directory doc/website/content/en/status/report-2023-07-2023-09/ (create it if it is missing); * submit a pull request at . You should put your reports in the directory doc/website/content/en/status/report-2023-07-2023-09/ (create it if it is missing); * send an email to status-submissions@FreeBSD.org including your report. An AsciiDoc template is available at . We look forward to seeing your 2023Q3 reports! Thanks, Lorenzo Salvadore (on behalf of status@) From nobody Sat Sep 2 06:24:16 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rd4d11501z4s0nd for ; Sat, 2 Sep 2023 06:24:53 +0000 (UTC) (envelope-from c@bow.st) Received: from comms.drone (in.bow.st [71.19.146.166]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rd4cz2GSSz4dL8 for ; Sat, 2 Sep 2023 06:24:51 +0000 (UTC) (envelope-from c@bow.st) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of c@bow.st designates 71.19.146.166 as permitted sender) smtp.mailfrom=c@bow.st; dmarc=none Received: from homebase (unknown [IPv6:fe80::ff1d:976a:a7e4:ee6a]) by comms.drone (Postfix) with ESMTPSA id AFE85FE0C for ; Sat, 2 Sep 2023 06:24:46 +0000 (UTC) From: "Mathew\, Cherry G.*" To: freebsd-hackers@freebsd.org Subject: ARC model specified in spinroot/promela User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (berkeley-unix) Date: Sat, 02 Sep 2023 06:24:16 +0000 Message-ID: <85jzt96qjz.fsf@bow.st> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Spamd-Bar: / X-Spamd-Result: default: False [-0.80 / 15.00]; HFILTER_HELO_IP_A(1.00)[comms.drone]; NEURAL_HAM_MEDIUM(-0.85)[-0.852]; NEURAL_HAM_SHORT(-0.62)[-0.622]; NEURAL_HAM_LONG(-0.43)[-0.427]; HFILTER_HELO_NORES_A_OR_MX(0.30)[comms.drone]; R_SPF_ALLOW(-0.20)[+mx:c]; MIME_GOOD(-0.10)[multipart/mixed,text/plain,text/x-patch]; ONCE_RECEIVED(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ASN(0.00)[asn:47066, ipnet:71.19.146.0/24, country:US]; RCVD_TLS_ALL(0.00)[]; R_DKIM_NA(0.00)[]; BLOCKLISTDE_FAIL(0.00)[71.19.146.166:server fail]; DMARC_NA(0.00)[bow.st]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[] X-Rspamd-Queue-Id: 4Rd4cz2GSSz4dL8 --=-=-= Content-Type: text/plain Hello hackers, I'm writing to introduce a project I've been working on off-and-on for a while now - verifying parts of kernel code using a formal specifier[1] Please find attached a patch to try out a formal verification run of the Adaptive Replacement Cache by Megido et.al. [2] You can try it out by installing spin from your favourite package manager, and then running "make" in the current directory - it just needs the usual C toolchain, afaik. I'm hoping that someone can help me complete the current run, as I don't have the computing resources required to run the full model (about 16GB RAM). Meanwhile I'll keep finding ways to reduce the statespace required. The idea here is that once verified, the model can be written up as C code, and then re-exracted using the modex [3] tool I have some unpublished work that demonstrates this using the md(4) NetBSD driver, but I want to be able to do a clean-room implementation first, in order to get a better sense of a developer methodology I'm going to call "DDD" or "D3" (Design Driven Development). Bricks and boquets welcome. (Please Cc: me, as I am not subscribed to the ML) [1] https://spinroot.com/spin/whatispin.html [2] https://www.usenix.org/legacy/events/fast03/tech/full_papers/megiddo/megiddo.pdf Fig. 4 [3] http://spinroot.com/modex/MANUAL.html Best, -- ~cherry --=-=-= Content-Type: text/x-patch Content-Disposition: inline; filename=arc.diff diff -urN arc.null/arc.drv arc/arc.drv --- arc.null/arc.drv 1970-01-01 00:00:00.000000000 +0000 +++ arc/arc.drv 2023-09-02 05:31:42.000000000 +0000 @@ -0,0 +1,97 @@ +/* $NetBSD$ */ + +/* Spin model driver for NetBSD arc(9) arc.c written by cherry */ + +/* XXX: Move these into a set of library includes ? */ +/* XXX: Equivalence verification */ +/* Note: CAS implemented in an atomic {} block */ +#define mutex_enter(_mutex) \ + atomic { \ + (_mutex == 0) -> _mutex = 1; \ + } + +#define mutex_exit(_mutex) \ + atomic { \ + assert(_mutex == 1); \ + (_mutex == 1) -> _mutex = 0; \ + } + +#define N_ITEMS 6 /* Number of cache items to test with */ +#define IID_INVAL N_ITEMS /* Valid item.iid must be < N_ITEMS */ +#define C 5 /* Cache size - use judiciously - adds to statespace */ +#define ITEM_REPS 6 /* Max repeat item requests */ + +typedef arc_item { + byte iid; /* Unique identifier for item */ + bool cached; +}; + +/* Note that we use the arc_item.iid as the member lookup handle to reduce state space */ +typedef arc_list { + chan item_list = [C + 1] of { byte }; /* A list of page items */ +}; + +#define lengthof(_arc_list) len(_arc_list.item_list) +#define memberof(_arc_list, _arc_item) _arc_list.item_list??[eval(_arc_item.iid)] +#define addMRU(_arc_list, _arc_item) _arc_list.item_list!_arc_item.iid +#define readLRU(_arc_list, _arc_item) _arc_list.item_list?<_arc_item.iid> +#define delLRU(_arc_list) _arc_list.item_list?_ +#define delitem(_arc_list, _arc_item) _arc_list.item_list??eval(_arc_item.iid) +#define refreshitemto(_src_arc_list, _dest_arc_list, _arc_item) \ + d_step { \ + delitem(_src_arc_list, _arc_item); \ + addMRU(_dst_arc_list, _arc_item); \ + } +#define refreshitem(_arc_list, _arc_item) refreshitemto(_arc_list, _arc_list, _arc_item) + +#define cachefetch(_arc_item) _arc_item.cached = true +#define cacheremove(_arc_item) _arc_item.cached = false + +#define min(a, b) ((a < b) -> a : b) +#define max(a, b) ((a > b) -> a : b) + +/* Declare arc lists */ +arc_list B1, B2, T1, T2; + +#define init_arc_item(_arc_item, _iid, _cached) \ + d_step { \ + _arc_item.iid = _iid; \ + _arc_item.cached = _cached; \ + } + +hidden arc_item _x[N_ITEMS]; /* Input state is irrelevant from a verification PoV */ +hidden byte _x_iid = 0; +hidden byte _item_rep = 0; + +/* Temp variable to hold LRU item */ +hidden arc_item LRUitem; + +/* Adaptation "delta" variables */ +hidden byte d1, d2; +byte p = 0; + +bit sc_lock; + +/* Drive the procs */ +init { + + atomic { + _x_iid = 0; + do + :: _x_iid < N_ITEMS -> + init_arc_item(_x[_x_iid], _x_iid, false); + _item_rep = 0; + do + :: _item_rep < (_x_iid % ITEM_REPS) -> + run p_arc(_x[_x_iid]); + _item_rep++; + :: _item_rep >= (_x_iid % ITEM_REPS) -> + break; + od + _x_iid++; + :: _x_iid >= N_ITEMS -> + break + od + } + +} diff -urN arc.null/arc.inv arc/arc.inv --- arc.null/arc.inv 1970-01-01 00:00:00.000000000 +0000 +++ arc/arc.inv 2023-09-02 05:31:42.000000000 +0000 @@ -0,0 +1,19 @@ +/* $NetBSD$ */ + +/* These are Linear Temporal Logic invariants (and constraints) + * applied over the statespace created by the promela + * specification. Correctness is implied by Logical consistency. + */ +ltl +{ + /* Liveness - all thread finally end */ + eventually always (_nr_pr == 1) && + /* Might look obvious, but make this explicit */ + always ((lengthof(T1) <= C) && + (lengthof(B1) <= C) && + (lengthof(T2) <= C) && + (lengthof(B2) <= C) ) && + /* Not strictly true, but this forces a good driver algo */ + eventually (p > 0) + +} diff -urN arc.null/arc.pml arc/arc.pml --- arc.null/arc.pml 1970-01-01 00:00:00.000000000 +0000 +++ arc/arc.pml 2023-09-02 05:31:42.000000000 +0000 @@ -0,0 +1,194 @@ +/* $NetBSD$ */ + +/* Spin process model for NetBSD arc(9) arc.c written by cherry */ + +/* + * We implement the following algorithm from page 10, Figure 4. + * https://www.usenix.org/legacy/events/fast03/tech/full_papers/megiddo/megiddo.pdf + * + * + * ARC(c) + * + * INPUT: The request stream x1,x2,....,xt,.... + * INITIALIZATION: Set p = 0 and set the LRU lists T1, B1, T2, and B2 to empty. + * + * For every t>=1 and any xt, one and only one of the following four cases must occur. + * Case I: xt is in T1 or T2. A cache hit has occurred in ARC(c) and DBL(2c). + * Move xt to MRU position in T2. + * + * Case II: xt is in B1. A cache miss (resp. hit) has occurred in ARC(c) (resp. DBL(2c)). + * ADAPTATION: Update p = min { p + d1,c } + * where d1 = { 1 if |B1| >= |B2|, |B2|/|B1| otherwise + * + * REPLACE(xt, p). Move xt from B1 to the MRU position in T2 (also fetch xt to the cache). + * + * Case III: xt is in B2. A cache miss (resp. hit) has occurred in ARC(c) (resp. DBL(2c)). + * ADAPTATION: Update p = max { p - d2,0 } + * where d2 = { 1 if |B2| >= |B1|, |B1|/|B2| otherwise + * + * REPLACE(xt, p). Move xt from B2 to the MRU position in T2 (also fetch xt to the cache). + * + * Case IV: xt is not in T1 U B1 U T2 U B2. A cache miss has occurred in ARC(c) and DBL(2c). + * Case A: L1 = T1 U B1 has exactly c pages. + * If (|T1| < c) + * Delete LRU page in B1. REPLACE(xt,p). + * else + * Here B1 is empty. Delete LRU page in T1 (also remove it from the cache). + * endif + * Case B: L1 = T1 U B1 has less than c pages. + * If (|T1| + |T2| + |B1| + |B2| >= c) + * Delete LRU page in B2, if (|T1| + |T2| + |B1| + |B2| = 2c). + * REPLACE(xt, p). + * endif + * + * Finally, fetch xt to the cache and move it to MRU position in T1. + * + * Subroutine REPLACE(xt,p) + * If ( (|T1| is not empty) and ((|T1| exceeds the target p) or (xt is in B2 and |T1| = p)) ) + * Delete the LRU page in T1 (also remove it from the cache), and move it to MRU position in B1. + * else + * Delete the LRU page in T2 (also remove it from the cache), and move it to MRU position in B2. + * endif + */ + +inline REPLACE(/* arc_item */ x_t, /* byte */ p) +{ + /* + * Since LRUitem is declared in scope p_ARC, we expect it to be only accessible from there and REPLACE() + * as REPLACE() is only expected to be called from p_ARC. + * XXX: May need to revisit due to Modex related limitations. + */ + init_arc_item(LRUitem, IID_INVAL, false); + + if + :: + (lengthof(T1) != 0) && + ((lengthof(T1) > p) || (memberof(B2, x_t) && (lengthof(T1) == p))) + -> + d_step { + readLRU(T1, LRUitem); + delLRU(T1); + cacheremove(LRUitem); + addMRU(B1, LRUitem); + } + + :: + else + -> + d_step { + readLRU(T2, LRUitem); + delLRU(T2); + cacheremove(LRUitem); + addMRU(B2, LRUitem); + } + fi +} + +proctype p_arc(arc_item x_t) +{ + /* Serialise entry */ + + mutex_enter(sc_lock); + + if + :: /* Case I */ + memberof(T1, x_t) + -> + d_step { + delitem(T1, x_t); + addMRU(T2, x_t); + } + :: /* Case I */ + memberof(T2, x_t) + -> + d_step { + delitem(T2, x_t); + addMRU(T2, x_t); + } + :: /* Case II */ + memberof(B1, x_t) + -> + d1 = ((lengthof(B1) >= lengthof(B2)) -> 1 : (lengthof(B2)/lengthof(B1))); + p = min((p + d1), C); + + REPLACE(x_t, p); + d_step { + delitem(B1, x_t); + addMRU(T2, x_t); + cachefetch(x_t); + } + :: /* Case III */ + memberof(B2, x_t) + -> + d2 = ((lengthof(B2) >= lengthof(B1)) -> 1 : (lengthof(B1)/lengthof(B2))); + p = max(p - d2, 0); + + REPLACE(x_t, p); + d_step { + delitem(B2, x_t); + addMRU(T2, x_t); + cachefetch(x_t); + } + :: /* Case IV */ + !(memberof(T1, x_t) || + memberof(B1, x_t) || + memberof(T2, x_t) || + memberof(B2, x_t)) + -> + if + :: /* Case A */ + ((lengthof(T1) + lengthof(B1)) == C) + -> + if + :: + (lengthof(T1) < C) + -> + delLRU(B1); + REPLACE(x_t, p); + :: + else + -> + assert(lengthof(B1) == 0); + d_step { + readLRU(T1, LRUitem); + delLRU(T1); + cacheremove(LRUitem); + } + fi + :: /* Case B */ + ((lengthof(T1) + lengthof(B1)) < C) + -> + if + :: + ((lengthof(T1) + + lengthof(T2) + + lengthof(B1) + + lengthof(B2)) >= C) + -> + if + :: + ((lengthof(T1) + + lengthof(T2) + + lengthof(B1) + + lengthof(B2)) == (2 * C)) + -> + delLRU(B2); + REPLACE(x_t, p); + :: + else + -> + skip; + fi + :: + else + -> + skip; + fi + fi + cachefetch(x_t); + addMRU(T1, x_t); + fi + + mutex_exit(sc_lock); +} + diff -urN arc.null/Makefile arc/Makefile --- arc.null/Makefile 1970-01-01 00:00:00.000000000 +0000 +++ arc/Makefile 2023-09-02 05:31:42.000000000 +0000 @@ -0,0 +1,89 @@ +# This set of spinroot related files were written by cherry +# in the Gregorian Calendar year AD.2023, in the month +# of February that year. +# +# We have two specification files and a properties file +# +# The properties file contains "constraint" sections +# such as ltl or never claims (either or, not both). +# The specification is divided into two files: +# the file with suffix '.drv' is a "driver" which +# instantiates processes that will ultimately "drive" the +# models under test. +# The file with the suffix '.pml' contains the process +# model code, which, is intended to be the formal specification +# for the code we are interested in writing in C. +# +# We process these files in slightly different ways during +# the dev cycle, but broadly speaking, the idea is to create +# a file called 'spinmodel.pml' which contains the final +# model file that is fed to spin. +# +# Note that when we use the model extractor tool "modex" to +# extract the 'specification' from C code written to implement +# the model defined above. We use a 'harness' file (see file with +# suffix '.prx' below. +# +# Once the harness has been run, spinmodel.pml should be +# synthesised and processed as usual. +# +# The broad idea is that software dev starts by writing the spec +# first, validating the model, and then implementing the model in +# C, after which we come back to extract the model from the C file +# and cross check our implementation using spin. +# +# If things go well, the constraints specified in the '.ltl' file +# should hold exactly for both the handwritten model, and the +# extracted one. + +spin-gen: arc.pml arc.drv arc.inv + cp arc.pml model #mimic modex + cat arc.drv > spinmodel.pml;cat model >> spinmodel.pml;cat arc.inv >> spinmodel.pml; + spin -a spinmodel.pml + +spin-build: pan.* + cc -o pan pan.c + +spin-run: spin-build #XXX depend on pan + ./pan -a #Generate arc.pml.trail on error + +# You run the trace only if the spin run above failed and created a trail +spin-trace: spinmodel.pml.trail + spin -t spinmodel.pml -p -g # -p (statements) -g (globals) -l (locals) -s (send) -r (recv) + ./pan -r spinmodel.pml.trail -g + +# Modex Extracts from arc.c to 'model' - see arc.prx +# Unfortunately there doesn't seem to be a way to specify a filename +# to generate +SRC = $HOME/work/NetBSD-src + +modex-gen: arc.prx arc.c + #modex -v -w -D_KERNEL -I$obj/home/antix/work/NetBSD-src/destdir.amd64/usr/include -I$$HOME/work/NetBSD-src/obj/home/antix/work/NetBSD-src/sys/arch/amd64/compile/GENERIC/ -I$$HOME/work/NetBSD-src/sys arc.prx + #cat model > spinmodel.pml + touch ioconf.h # Pretend we have an Kern conf + modex -v -w arc.prx + cat arc.drv > spinmodel.pml;cat model >> spinmodel.pml;cat arc.inv >> spinmodel.pml; + spin -a spinmodel.pml #Sanity check + +modex-gen-clean: + rm -f ioconf.h # Temp - see above. + rm -f spinmodel.pml # Our consolidated model file + rm -f _spin_nvr.tmp # Never claim file + rm -f model # modex generated intermediate "model" file + rm -f pan.* # Spin generated source files + rm -f _modex* # modex generated script files + rm -f *.I *.M +spin-gen-clean: + rm -f spinmodel.pml # Our consolidated model file + rm -f _spin_nvr.tmp # Never claim file + rm -f model # Intermediate "model" file + rm -f pan.* # Spin generated source files + +spin-build-clean: + rm -f pan + +spin-run-clean: + rm -f spinmodel.pml.trail + +clean: modex-gen-clean spin-gen-clean spin-build-clean spin-run-clean +# rm -f *~ --=-=-=-- From nobody Sat Sep 2 07:05:40 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rd5Xw6Gmzz4s7SN for ; Sat, 2 Sep 2023 07:06:24 +0000 (UTC) (envelope-from c@bow.st) Received: from comms.drone (in.bow.st [71.19.146.166]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rd5Xt30xvz3HxX for ; Sat, 2 Sep 2023 07:06:22 +0000 (UTC) (envelope-from c@bow.st) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of c@bow.st designates 71.19.146.166 as permitted sender) smtp.mailfrom=c@bow.st; dmarc=none Received: from homebase (unknown [IPv6:fe80::ff1d:976a:a7e4:ee6a]) by comms.drone (Postfix) with ESMTPSA id 91D7EFE0C for ; Sat, 2 Sep 2023 07:06:11 +0000 (UTC) From: "Mathew\, Cherry G.*" To: freebsd-hackers@freebsd.org Subject: [patch] ARC model specified in spinroot/promela User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (berkeley-unix) Date: Sat, 02 Sep 2023 07:05:40 +0000 Message-ID: <85y1hp5a2j.fsf@bow.st> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain X-Spamd-Bar: - X-Spamd-Result: default: False [-1.18 / 15.00]; HFILTER_HELO_IP_A(1.00)[comms.drone]; NEURAL_HAM_SHORT(-1.00)[-0.996]; NEURAL_HAM_MEDIUM(-0.90)[-0.902]; NEURAL_HAM_LONG(-0.38)[-0.380]; HFILTER_HELO_NORES_A_OR_MX(0.30)[comms.drone]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; TO_DN_NONE(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[bow.st]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; ASN(0.00)[asn:47066, ipnet:71.19.146.0/24, country:US]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4Rd5Xt30xvz3HxX Hello hackers, I'm writing to introduce a project I've been working on off-and-on for a while now - verifying parts of kernel code using a formal specifier[1] Please find attached a patch to try out a formal verification run of the Adaptive Replacement Cache by Megido et.al. [2] You can try it out by installing spin from your favourite package manager, and then running "make" in the current directory - it just needs the usual C toolchain, afaik. I'm hoping that someone can help me complete the current run, as I don't have the computing resources required to run the full model (about 16GB RAM). Meanwhile I'll keep finding ways to reduce the statespace required. The idea here is that once verified, the model can be written up as C code, and then re-exracted using the modex [3] tool I have some unpublished work that demonstrates this using the md(4) NetBSD driver, but I want to be able to do a clean-room implementation first, in order to get a better sense of a developer methodology I'm going to call "DDD" or "D3" (Design Driven Development). Bricks and boquets welcome. (Please Cc: me, as I am not subscribed to the ML) [1] https://spinroot.com/spin/whatispin.html [2] https://www.usenix.org/legacy/events/fast03/tech/full_papers/megiddo/megiddo.pdf Fig. 4 [3] http://spinroot.com/modex/MANUAL.html Best, -- ~cherry diff -urN arc.null/arc.drv arc/arc.drv --- arc.null/arc.drv 1970-01-01 00:00:00.000000000 +0000 +++ arc/arc.drv 2023-09-02 05:31:42.000000000 +0000 @@ -0,0 +1,97 @@ +/* $NetBSD$ */ + +/* Spin model driver for NetBSD arc(9) arc.c written by cherry */ + +/* XXX: Move these into a set of library includes ? */ +/* XXX: Equivalence verification */ +/* Note: CAS implemented in an atomic {} block */ +#define mutex_enter(_mutex) \ + atomic { \ + (_mutex == 0) -> _mutex = 1; \ + } + +#define mutex_exit(_mutex) \ + atomic { \ + assert(_mutex == 1); \ + (_mutex == 1) -> _mutex = 0; \ + } + +#define N_ITEMS 6 /* Number of cache items to test with */ +#define IID_INVAL N_ITEMS /* Valid item.iid must be < N_ITEMS */ +#define C 5 /* Cache size - use judiciously - adds to statespace */ +#define ITEM_REPS 6 /* Max repeat item requests */ + +typedef arc_item { + byte iid; /* Unique identifier for item */ + bool cached; +}; + +/* Note that we use the arc_item.iid as the member lookup handle to reduce state space */ +typedef arc_list { + chan item_list = [C + 1] of { byte }; /* A list of page items */ +}; + +#define lengthof(_arc_list) len(_arc_list.item_list) +#define memberof(_arc_list, _arc_item) _arc_list.item_list??[eval(_arc_item.iid)] +#define addMRU(_arc_list, _arc_item) _arc_list.item_list!_arc_item.iid +#define readLRU(_arc_list, _arc_item) _arc_list.item_list?<_arc_item.iid> +#define delLRU(_arc_list) _arc_list.item_list?_ +#define delitem(_arc_list, _arc_item) _arc_list.item_list??eval(_arc_item.iid) +#define refreshitemto(_src_arc_list, _dest_arc_list, _arc_item) \ + d_step { \ + delitem(_src_arc_list, _arc_item); \ + addMRU(_dst_arc_list, _arc_item); \ + } +#define refreshitem(_arc_list, _arc_item) refreshitemto(_arc_list, _arc_list, _arc_item) + +#define cachefetch(_arc_item) _arc_item.cached = true +#define cacheremove(_arc_item) _arc_item.cached = false + +#define min(a, b) ((a < b) -> a : b) +#define max(a, b) ((a > b) -> a : b) + +/* Declare arc lists */ +arc_list B1, B2, T1, T2; + +#define init_arc_item(_arc_item, _iid, _cached) \ + d_step { \ + _arc_item.iid = _iid; \ + _arc_item.cached = _cached; \ + } + +hidden arc_item _x[N_ITEMS]; /* Input state is irrelevant from a verification PoV */ +hidden byte _x_iid = 0; +hidden byte _item_rep = 0; + +/* Temp variable to hold LRU item */ +hidden arc_item LRUitem; + +/* Adaptation "delta" variables */ +hidden byte d1, d2; +byte p = 0; + +bit sc_lock; + +/* Drive the procs */ +init { + + atomic { + _x_iid = 0; + do + :: _x_iid < N_ITEMS -> + init_arc_item(_x[_x_iid], _x_iid, false); + _item_rep = 0; + do + :: _item_rep < (_x_iid % ITEM_REPS) -> + run p_arc(_x[_x_iid]); + _item_rep++; + :: _item_rep >= (_x_iid % ITEM_REPS) -> + break; + od + _x_iid++; + :: _x_iid >= N_ITEMS -> + break + od + } + +} diff -urN arc.null/arc.inv arc/arc.inv --- arc.null/arc.inv 1970-01-01 00:00:00.000000000 +0000 +++ arc/arc.inv 2023-09-02 05:31:42.000000000 +0000 @@ -0,0 +1,19 @@ +/* $NetBSD$ */ + +/* These are Linear Temporal Logic invariants (and constraints) + * applied over the statespace created by the promela + * specification. Correctness is implied by Logical consistency. + */ +ltl +{ + /* Liveness - all thread finally end */ + eventually always (_nr_pr == 1) && + /* Might look obvious, but make this explicit */ + always ((lengthof(T1) <= C) && + (lengthof(B1) <= C) && + (lengthof(T2) <= C) && + (lengthof(B2) <= C) ) && + /* Not strictly true, but this forces a good driver algo */ + eventually (p > 0) + +} diff -urN arc.null/arc.pml arc/arc.pml --- arc.null/arc.pml 1970-01-01 00:00:00.000000000 +0000 +++ arc/arc.pml 2023-09-02 05:31:42.000000000 +0000 @@ -0,0 +1,194 @@ +/* $NetBSD$ */ + +/* Spin process model for NetBSD arc(9) arc.c written by cherry */ + +/* + * We implement the following algorithm from page 10, Figure 4. + * https://www.usenix.org/legacy/events/fast03/tech/full_papers/megiddo/megiddo.pdf + * + * + * ARC(c) + * + * INPUT: The request stream x1,x2,....,xt,.... + * INITIALIZATION: Set p = 0 and set the LRU lists T1, B1, T2, and B2 to empty. + * + * For every t>=1 and any xt, one and only one of the following four cases must occur. + * Case I: xt is in T1 or T2. A cache hit has occurred in ARC(c) and DBL(2c). + * Move xt to MRU position in T2. + * + * Case II: xt is in B1. A cache miss (resp. hit) has occurred in ARC(c) (resp. DBL(2c)). + * ADAPTATION: Update p = min { p + d1,c } + * where d1 = { 1 if |B1| >= |B2|, |B2|/|B1| otherwise + * + * REPLACE(xt, p). Move xt from B1 to the MRU position in T2 (also fetch xt to the cache). + * + * Case III: xt is in B2. A cache miss (resp. hit) has occurred in ARC(c) (resp. DBL(2c)). + * ADAPTATION: Update p = max { p - d2,0 } + * where d2 = { 1 if |B2| >= |B1|, |B1|/|B2| otherwise + * + * REPLACE(xt, p). Move xt from B2 to the MRU position in T2 (also fetch xt to the cache). + * + * Case IV: xt is not in T1 U B1 U T2 U B2. A cache miss has occurred in ARC(c) and DBL(2c). + * Case A: L1 = T1 U B1 has exactly c pages. + * If (|T1| < c) + * Delete LRU page in B1. REPLACE(xt,p). + * else + * Here B1 is empty. Delete LRU page in T1 (also remove it from the cache). + * endif + * Case B: L1 = T1 U B1 has less than c pages. + * If (|T1| + |T2| + |B1| + |B2| >= c) + * Delete LRU page in B2, if (|T1| + |T2| + |B1| + |B2| = 2c). + * REPLACE(xt, p). + * endif + * + * Finally, fetch xt to the cache and move it to MRU position in T1. + * + * Subroutine REPLACE(xt,p) + * If ( (|T1| is not empty) and ((|T1| exceeds the target p) or (xt is in B2 and |T1| = p)) ) + * Delete the LRU page in T1 (also remove it from the cache), and move it to MRU position in B1. + * else + * Delete the LRU page in T2 (also remove it from the cache), and move it to MRU position in B2. + * endif + */ + +inline REPLACE(/* arc_item */ x_t, /* byte */ p) +{ + /* + * Since LRUitem is declared in scope p_ARC, we expect it to be only accessible from there and REPLACE() + * as REPLACE() is only expected to be called from p_ARC. + * XXX: May need to revisit due to Modex related limitations. + */ + init_arc_item(LRUitem, IID_INVAL, false); + + if + :: + (lengthof(T1) != 0) && + ((lengthof(T1) > p) || (memberof(B2, x_t) && (lengthof(T1) == p))) + -> + d_step { + readLRU(T1, LRUitem); + delLRU(T1); + cacheremove(LRUitem); + addMRU(B1, LRUitem); + } + + :: + else + -> + d_step { + readLRU(T2, LRUitem); + delLRU(T2); + cacheremove(LRUitem); + addMRU(B2, LRUitem); + } + fi +} + +proctype p_arc(arc_item x_t) +{ + /* Serialise entry */ + + mutex_enter(sc_lock); + + if + :: /* Case I */ + memberof(T1, x_t) + -> + d_step { + delitem(T1, x_t); + addMRU(T2, x_t); + } + :: /* Case I */ + memberof(T2, x_t) + -> + d_step { + delitem(T2, x_t); + addMRU(T2, x_t); + } + :: /* Case II */ + memberof(B1, x_t) + -> + d1 = ((lengthof(B1) >= lengthof(B2)) -> 1 : (lengthof(B2)/lengthof(B1))); + p = min((p + d1), C); + + REPLACE(x_t, p); + d_step { + delitem(B1, x_t); + addMRU(T2, x_t); + cachefetch(x_t); + } + :: /* Case III */ + memberof(B2, x_t) + -> + d2 = ((lengthof(B2) >= lengthof(B1)) -> 1 : (lengthof(B1)/lengthof(B2))); + p = max(p - d2, 0); + + REPLACE(x_t, p); + d_step { + delitem(B2, x_t); + addMRU(T2, x_t); + cachefetch(x_t); + } + :: /* Case IV */ + !(memberof(T1, x_t) || + memberof(B1, x_t) || + memberof(T2, x_t) || + memberof(B2, x_t)) + -> + if + :: /* Case A */ + ((lengthof(T1) + lengthof(B1)) == C) + -> + if + :: + (lengthof(T1) < C) + -> + delLRU(B1); + REPLACE(x_t, p); + :: + else + -> + assert(lengthof(B1) == 0); + d_step { + readLRU(T1, LRUitem); + delLRU(T1); + cacheremove(LRUitem); + } + fi + :: /* Case B */ + ((lengthof(T1) + lengthof(B1)) < C) + -> + if + :: + ((lengthof(T1) + + lengthof(T2) + + lengthof(B1) + + lengthof(B2)) >= C) + -> + if + :: + ((lengthof(T1) + + lengthof(T2) + + lengthof(B1) + + lengthof(B2)) == (2 * C)) + -> + delLRU(B2); + REPLACE(x_t, p); + :: + else + -> + skip; + fi + :: + else + -> + skip; + fi + fi + cachefetch(x_t); + addMRU(T1, x_t); + fi + + mutex_exit(sc_lock); +} + diff -urN arc.null/Makefile arc/Makefile --- arc.null/Makefile 1970-01-01 00:00:00.000000000 +0000 +++ arc/Makefile 2023-09-02 05:31:42.000000000 +0000 @@ -0,0 +1,89 @@ +# This set of spinroot related files were written by cherry +# in the Gregorian Calendar year AD.2023, in the month +# of February that year. +# +# We have two specification files and a properties file +# +# The properties file contains "constraint" sections +# such as ltl or never claims (either or, not both). +# The specification is divided into two files: +# the file with suffix '.drv' is a "driver" which +# instantiates processes that will ultimately "drive" the +# models under test. +# The file with the suffix '.pml' contains the process +# model code, which, is intended to be the formal specification +# for the code we are interested in writing in C. +# +# We process these files in slightly different ways during +# the dev cycle, but broadly speaking, the idea is to create +# a file called 'spinmodel.pml' which contains the final +# model file that is fed to spin. +# +# Note that when we use the model extractor tool "modex" to +# extract the 'specification' from C code written to implement +# the model defined above. We use a 'harness' file (see file with +# suffix '.prx' below. +# +# Once the harness has been run, spinmodel.pml should be +# synthesised and processed as usual. +# +# The broad idea is that software dev starts by writing the spec +# first, validating the model, and then implementing the model in +# C, after which we come back to extract the model from the C file +# and cross check our implementation using spin. +# +# If things go well, the constraints specified in the '.ltl' file +# should hold exactly for both the handwritten model, and the +# extracted one. + +spin-gen: arc.pml arc.drv arc.inv + cp arc.pml model #mimic modex + cat arc.drv > spinmodel.pml;cat model >> spinmodel.pml;cat arc.inv >> spinmodel.pml; + spin -a spinmodel.pml + +spin-build: pan.* + cc -o pan pan.c + +spin-run: spin-build #XXX depend on pan + ./pan -a #Generate arc.pml.trail on error + +# You run the trace only if the spin run above failed and created a trail +spin-trace: spinmodel.pml.trail + spin -t spinmodel.pml -p -g # -p (statements) -g (globals) -l (locals) -s (send) -r (recv) + ./pan -r spinmodel.pml.trail -g + +# Modex Extracts from arc.c to 'model' - see arc.prx +# Unfortunately there doesn't seem to be a way to specify a filename +# to generate +SRC = $HOME/work/NetBSD-src + +modex-gen: arc.prx arc.c + #modex -v -w -D_KERNEL -I$obj/home/antix/work/NetBSD-src/destdir.amd64/usr/include -I$$HOME/work/NetBSD-src/obj/home/antix/work/NetBSD-src/sys/arch/amd64/compile/GENERIC/ -I$$HOME/work/NetBSD-src/sys arc.prx + #cat model > spinmodel.pml + touch ioconf.h # Pretend we have an Kern conf + modex -v -w arc.prx + cat arc.drv > spinmodel.pml;cat model >> spinmodel.pml;cat arc.inv >> spinmodel.pml; + spin -a spinmodel.pml #Sanity check + +modex-gen-clean: + rm -f ioconf.h # Temp - see above. + rm -f spinmodel.pml # Our consolidated model file + rm -f _spin_nvr.tmp # Never claim file + rm -f model # modex generated intermediate "model" file + rm -f pan.* # Spin generated source files + rm -f _modex* # modex generated script files + rm -f *.I *.M +spin-gen-clean: + rm -f spinmodel.pml # Our consolidated model file + rm -f _spin_nvr.tmp # Never claim file + rm -f model # Intermediate "model" file + rm -f pan.* # Spin generated source files + +spin-build-clean: + rm -f pan + +spin-run-clean: + rm -f spinmodel.pml.trail + +clean: modex-gen-clean spin-gen-clean spin-build-clean spin-run-clean +# rm -f *~ From nobody Sat Sep 2 08:23:43 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Rd7Gj1pYwz4sMdm for ; Sat, 2 Sep 2023 08:24:13 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Rd7Gh1KrCz3VrC for ; Sat, 2 Sep 2023 08:24:12 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from webmail2.leidinger.net (roundcube.Leidinger.net [192.168.1.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: Alexander@Leidinger.net) by outgoing.leidinger.net (Postfix) with ESMTPSA id 204A7E944; Sat, 2 Sep 2023 10:23:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1693643033; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GGXxohyLideG07ncKmZ+imAhqlSArWSZdg1aiKA/aRE=; b=0qpoAE+Q/9AIqRnVOzZBsjkvhoMxYu4BYnTBg6qWM1FF8IOzr7aB4sV7DN5Ha9mIeFrEK5 VWvO5JYeGfYl2jh6Fr7KOH3A/tjJA8ynod6s3qubC5G9gWcnW81FjB6fS/g9J/eYY4++zz BneZWQbiC7vB8d179FL8XnFxxl1BPQb1cBIrk1EWYB2JLmfaNDgTm4ffp3qqKT3Q5rHuph 2BxrNCBse8qe6vJiz9NAqH6hiEb2aNgP/+YR+RZkRIKO+k1sWpUx7cxgN90QvUoB0CV5X/ r5b8ZN4cPZM3td/idm3wZqoM7PLGLPVukHq+et0Ig2ri3X8Tws0LTDkYn5D0uw== List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Date: Sat, 02 Sep 2023 10:23:43 +0200 From: Alexander Leidinger To: "Mathew, Cherry G.*" Cc: freebsd-hackers@freebsd.org Subject: Re: ARC model specified in spinroot/promela In-Reply-To: <85jzt96qjz.fsf@bow.st> References: <85jzt96qjz.fsf@bow.st> Message-ID: <9c424a574cdd39fc879c9ed9192556c0@Leidinger.net> X-Sender: Alexander@Leidinger.net Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE] X-Rspamd-Queue-Id: 4Rd7Gh1KrCz3VrC Am 2023-09-02 08:24, schrieb Mathew, Cherry G.*: > Hello hackers, > > I'm writing to introduce a project I've been working on off-and-on for > a > while now - verifying parts of kernel code using a formal specifier[1] > > Please find attached a patch to try out a formal verification run of > the > Adaptive Replacement Cache by Megido et.al. [2] > > You can try it out by installing spin from your favourite package > manager, and then running "make" in the current directory - it just > needs the usual C toolchain, afaik. > > I'm hoping that someone can help me complete the current run, as I > don't > have the computing resources required to run the full model (about 16GB > RAM). Meanwhile I'll keep finding ways to reduce the statespace > required. How long is this supposed to take? For me it took about 2 seconds to finish. Note, the Makefile specifies a NetBSD src directory which I don't have on this FreeBSD system... Samstag, 02. September 2023, 10:18:12 (235) root@ttypts/3 # make cp arc.pml model #mimic modex cat arc.drv > spinmodel.pml;cat model >> spinmodel.pml;cat arc.inv >> spinmodel.pml; spin -a spinmodel.pml ltl ltl_0: ((<> ([] ((_nr_pr==1)))) && ([] (((((len(T1.item_list)<=5)) && ((len(B1.item_list)<=5))) && ((len(T2.item_list)<=5))) && ((len(B2.item_list)<=5))))) && (<> ((p>0))) Samstag, 02. September 2023, 10:18:14 (236) root@ttypts/3 # ll total 275 -rw-r--r-- 1 root wheel 1,1K 2 Sep. 10:18 _spin_nvr.tmp -rw-r--r-- 1 root wheel 2,6K 2 Sep. 10:15 arc.drv -rw-r--r-- 1 root wheel 555B 2 Sep. 10:15 arc.inv -rw-r--r-- 1 root wheel 4,6K 2 Sep. 10:15 arc.pml -rw-r--r-- 1 root wheel 3,3K 2 Sep. 10:15 Makefile -rw-r--r-- 1 root wheel 4,6K 2 Sep. 10:18 model -rw-r--r-- 1 root wheel 2,9K 2 Sep. 10:18 pan.b -rw-r--r-- 1 root wheel 335K 2 Sep. 10:18 pan.c -rw-r--r-- 1 root wheel 18K 2 Sep. 10:18 pan.h -rw-r--r-- 1 root wheel 42K 2 Sep. 10:18 pan.m -rw-r--r-- 1 root wheel 55K 2 Sep. 10:18 pan.p -rw-r--r-- 1 root wheel 30K 2 Sep. 10:18 pan.t -rw-r--r-- 1 root wheel 7,8K 2 Sep. 10:18 spinmodel.pml Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From nobody Sat Sep 2 10:33:18 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdB7h3DDnz4rYqn for ; Sat, 2 Sep 2023 10:33:20 +0000 (UTC) (envelope-from adridg@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdB7h0mcMz4P30; Sat, 2 Sep 2023 10:33:20 +0000 (UTC) (envelope-from adridg@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693650800; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=0CfSNXAt3vjcY9KjY/8Ggq++DDtjfglIYGQ94moLWfM=; b=FEsWi2L+fccdQd7ua/W06skyEVy/2sF6OI1OhEXMMwVBmKNbpsgCDnYDy/kM1rz7MEQEb3 zBMxLqnTRcTgOX621IM7sRTAYGeXNjuo5mZhnVkEuqUmXnCiZ7MqCVUANFGdqFcLa6k6Xo 8eECfMoowdIb+WTrIx5fy0gWI7RObDSG252jl20GMR7bHs4iyFbPwUIYN0T8+TzJ+kPrSu rENNGBtlN7ALPu9/mjHxwcBpQryN+XCRAP2tfUP+jLnwVNyglgXdm80mGbiYC4xxoEX1+0 aQ/xr3Msw+aHVC2tseKwmRf3WUNbrB4idQMAgKbnqF4BwQTJVIq62co35SPSLw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1693650800; a=rsa-sha256; cv=none; b=gWp0n/7CX+K8rW2HFMIIFTpuWFWi8zTT3YhyFhyeeqTfY4odax79VTVzKIlyZqmQmA0CxT 82p/Fj4/71wSIaI5v8W3xU91fBSWGCf42BViABTrdnQtRWiSGtC1vQQi7ssn2fu32MCYVm LGlt/NvpMY9rOXkiGOh72nvGFTXcNHY8gR2CheHD53xy74EBYXJRMp2yi4cBReYAjcv7F6 0GzWXHFaWrdE8aUe+2rqWIoze4wUXByyTWyY/IL7sZVtl1Z4PgkBFtV5iv5XSWT9j7VcA9 c9xbJpp5nQqRyr7V+lPJ6ntHw83HweBMQynfP7nWMU0rAf2uhr5iNYpn7W587w== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1693650800; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=0CfSNXAt3vjcY9KjY/8Ggq++DDtjfglIYGQ94moLWfM=; b=cPAknFXFbNlDqd5Z3ZfHyt/VQeiIMzQngTEOnaR4sI7d/Fn7hiZDaE4ISpdvOHt6BEQjwG YFltjQzKXZbQ6qG/ILdRpihvxyZhEVF/8vYN9+mzPxP3aIaQa8h7+VkoyHR6tvOfb6Ybvi yLu86DqyZQ/zNJHxUi2BeZLquCDTt27RDXZMfJbWyMQl5A1+G4TKjWZdKjamGvUL0/Gqpn EkRxTOF13uhUnHrtQtCLDSgg8lBo11uxAm9IUMTmEVnXVFcZKMLk7Z2WZNR8S3Hi92wRJS qr2+m89480EAP535cMs9285w3S9PjLX63KF9CZu8+j7XWk8kdvLEVP7lguXEuQ== Received: from beastie.bionicmutton.org (2a02-a46a-34af-1-2ef0-5dff-fe62-ecc.fixed6.kpn.net [IPv6:2a02:a46a:34af:1:2ef0:5dff:fe62:ecc]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: adridg) by smtp.freebsd.org (Postfix) with ESMTPSA id 4RdB7g4XZ7zqrS; Sat, 2 Sep 2023 10:33:19 +0000 (UTC) (envelope-from adridg@freebsd.org) From: Adriaan de Groot To: hackers@freebsd.org, Alexander Leidinger , c@bow.st Subject: Re: ARC model specified in spinroot/promela Date: Sat, 02 Sep 2023 12:33:18 +0200 Message-ID: <5846941.Zv9zXsTiuT@beastie.bionicmutton.org> Organization: FreeBSD List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart2673165.TYJnH3iKXO"; micalg="pgp-sha256"; protocol="application/pgp-signature" --nextPart2673165.TYJnH3iKXO Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="utf-8"; protected-headers="v1" From: Adriaan de Groot Subject: Re: ARC model specified in spinroot/promela Date: Sat, 02 Sep 2023 12:33:18 +0200 Message-ID: <5846941.Zv9zXsTiuT@beastie.bionicmutton.org> Organization: FreeBSD MIME-Version: 1.0 For what it's worth, spin is available from ports (devel/spin), which I've just adopted and updated to 6.5.2, so it is quite straightforward to get this running on any recent FreeBSD system. I tested only with the ancient Peterson's mutual exclusion, which resolves instantly. Mailing-list archives don't preserve attachments, so, Cherry, if you could send it me directly that would be lovely. I spotted A. Mader's PLC Controller in the SPIN documents, that is one I am familiar with, and then realised that academic papers from the 2000s don't come with source code :( [ade] --nextPart2673165.TYJnH3iKXO Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part. Content-Transfer-Encoding: 7Bit -----BEGIN PGP SIGNATURE----- iQGzBAABCAAdFiEESUdADzdGoDiQC7F4Mo10LYgHpDUFAmTzD24ACgkQMo10LYgH pDX11AwA6Qr75mAYOtj9HWaFhE05Haxx8xrHh82JEgLikc/6PBSuwuVoeoWNNFK8 03OssTuTUtPZBjb6y6LSbBEP1lhEhVeSoSD8PASHZswC+4CyfWYbbF5hJPf0icpY tGDbMcJmSCge8jMF2z6DWXHhp+E28AQa7NUpvLL6uxbWhzY86Vp5tcnv79GerICG rtTKcSyYMpMAyam9exqTrn9usDeP7TGkjyX1H+JO/dW+5Pv6PrP+ZQYiljoSLWV6 oD/Xml2qc2NEJrl87u8AhaXw/auhSBXG0hYHMRB95yX/whOTkKt3vgWNj20ovbcN 3IRqjHkei0+Aqg9FNMPl1gZ4dp8K811lgs22dGVzpiqKghLl5cCA//dlIrEbUube 2vF0Xy4E8mR+RY1unu9KsnGobCN01SnRLVF0g3ITZygYLvmDa8y0MWO6+khAH9BS 8fDPCTw0YqSLoldMzCZ+/xE3Assn8HHbRxuv0fOodFrMs6ovvikAPWVNqr5YIyc/ 5mD0bpNI =IZXN -----END PGP SIGNATURE----- --nextPart2673165.TYJnH3iKXO-- From nobody Sat Sep 2 10:47:03 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdBSG3mG1z4rcj4 for ; Sat, 2 Sep 2023 10:47:42 +0000 (UTC) (envelope-from c@bow.st) Received: from comms.drone (in.bow.st [71.19.146.166]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdBSD0B9Kz4R5B; Sat, 2 Sep 2023 10:47:39 +0000 (UTC) (envelope-from c@bow.st) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of c@bow.st designates 71.19.146.166 as permitted sender) smtp.mailfrom=c@bow.st; dmarc=none Received: from homebase (unknown [IPv6:fe80::ff1d:976a:a7e4:ee6a]) by comms.drone (Postfix) with ESMTPSA id 81175FCFE; Sat, 2 Sep 2023 10:47:34 +0000 (UTC) From: "Mathew\, Cherry G.*" To: Alexander Leidinger , adridg@freebsd.org Cc: freebsd-hackers@freebsd.org Subject: Re: ARC model specified in spinroot/promela References: <85jzt96qjz.fsf@bow.st> <9c424a574cdd39fc879c9ed9192556c0@Leidinger.net> Date: Sat, 02 Sep 2023 10:47:03 +0000 In-Reply-To: <9c424a574cdd39fc879c9ed9192556c0@Leidinger.net> (Alexander Leidinger's message of "Sat, 02 Sep 2023 10:23:43 +0200") Message-ID: <858r9o6ee0.fsf@bow.st> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (berkeley-unix) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain X-Spamd-Bar: - X-Spamd-Result: default: False [-1.87 / 15.00]; HFILTER_HELO_IP_A(1.00)[comms.drone]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; NEURAL_HAM_LONG(-0.99)[-0.989]; NEURAL_HAM_SHORT(-0.99)[-0.988]; HFILTER_HELO_NORES_A_OR_MX(0.30)[comms.drone]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; TO_DN_SOME(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; ASN(0.00)[asn:47066, ipnet:71.19.146.0/24, country:US]; FROM_HAS_DN(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; RCVD_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[bow.st]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4RdBSD0B9Kz4R5B Hi Alexander, >>>>> "AL" == Alexander Leidinger writes: [...] AL> How long is this supposed to take? For me it took about 2 AL> seconds to finish. Apologies, I should have given more detailed instructions. I've organised the process in three steps: 1) Generate the model from spec: make spin-gen 2) Build the model: make spin-build 3) Run the model: make spin-run This is the heavy duty part, which takes up quite a bit of vmem (my process dies at about 8GB due to lack of swap etc. - makes no sense to thrash it beyond that without RAM - it slows down a lot). If all goes well with step 3) , then you will see a summary of the run on console. However, if there was an inconsistency or error detected in the run, then a tracefile is generated (spinmodel.pml.trail). I've included a target to dump a human friendly version of this trace. 4) Print trace from debug trail: make spin-trace AL> Note, the Makefile specifies a NetBSD src directory which I AL> don't have on this FreeBSD system... Sorry, I should have mentioned - you can safely ignore that - no specific external source layout is currently assumed - in fact I run these tests on a Linux VM at the moment. The source files become relevant only when the "model extraction" is done - I haven't got there yet for this project. >>>>> "ade" == Adriaan de Groot writes: ade> For what it's worth, spin is available from ports (devel/spin), ade> which I've just adopted and updated to 6.5.2, so it is quite ade> straightforward to get this running on any recent FreeBSD ade> system. Hi ade, Nice! I'd be curious to know performance comparisons viz Linux etc. esp because there seems to be a threaded version of spin (not sure if the FreeBSD package build enables this). ade> I tested only with the ancient Peterson's mutual exclusion, ade> which resolves instantly. Mailing-list archives don't preserve ade> attachments, so, Cherry, if you could send it me directly that ade> would be lovely. I spotted A. Mader's PLC Controller in the ade> SPIN documents, that is one I am familiar with, and then ade> realised that academic papers from the 2000s don't come with ade> source code :( Thanks, I noticed that after checking the archives, and did send a second mail with an inline patch. Will also send you a private one after this. Thanks so much for your interest - mainly, I'd like to understand how bad the statespace explosion is for this model. This will help me get a sense of what complexity of models we can attempt to verify like this, with current "commodity" hardware. Many Thanks, -- ~cherry From nobody Sat Sep 2 11:03:50 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdBpz164xz4rgwC for ; Sat, 2 Sep 2023 11:03:55 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Received: from mail-ej1-x62d.google.com (mail-ej1-x62d.google.com [IPv6:2a00:1450:4864:20::62d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdBpy6Pkjz4VbT; Sat, 2 Sep 2023 11:03:54 +0000 (UTC) (envelope-from grahamperrin@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x62d.google.com with SMTP id a640c23a62f3a-99c0cb7285fso334013466b.0; Sat, 02 Sep 2023 04:03:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693652633; x=1694257433; darn=freebsd.org; h=content-transfer-encoding:in-reply-to:subject:from:cc:references:to :content-language:user-agent:mime-version:date:message-id:from:to:cc :subject:date:message-id:reply-to; bh=WxNXcJDjapZVQzNuELIZ0zqazc2r9/sh5JQ5pZ70q0g=; b=FWdyf+drUFGLxxb71KoA1DWKwo3dGd+gW7pctLmy1HjeYgkqj2TQW5gsd/wCzExbnK POfwG/1WwOUsqYVAEXRFC1IwZd6/r8lwvFj9aPxDIri+db12umuJ4qdBrmdTRu/XID85 dS7FMET92DtJXp7e6Oi4mQD25qZd0r0nZ0VgHxEdhxuRArimucQHCvHgplOlYasBxAqm 6LfctUwOpuD1AD8ysi4DcewE568tfc7pnkD9vaNllQ5CzpMynkEDv1eoCw60ReGQRnTy OaCtMjRlsuFXt7BT++2IRJV7PiAE4wmYcZGzbI0Rz8XIE660A3+ljlDPliIssAu7zEDb DwmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693652633; x=1694257433; h=content-transfer-encoding:in-reply-to:subject:from:cc:references:to :content-language:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=WxNXcJDjapZVQzNuELIZ0zqazc2r9/sh5JQ5pZ70q0g=; b=av/pRWmEQpCe9a6WXZlmj4+6bjoPQB1i4bLz77p/zZ9QWxdzZJJ/vEcEca+elws8RL bKzhcsS6ka46swUkFZmnuhHJiNs1qqIINoMagFPF4sUxudVEpY3ERTZ/QrslC2wwkSYQ QP1HOPcveMtZkqEFqysvG4ORCqrOTRz5r/k7KalxfnMbJxMV8RAd1f7tpOXd3j5iOs3g gz+Rq1H5miB8XuOTRe1haF7gzFznb5UFYS69vZQYOSvmZDUZ9L+dXSVq2ubrLBhTy2Se Cb/Kq9Rq4s0aasP2EJbeaXikqZ9GiKfTcTEEfsK0lw/g5w0V3fwyhD95IbomT6EV5UU5 3TXA== X-Gm-Message-State: AOJu0YwB3FujfrC2g5M/VIW3TuXiZYWfQp+D5XkzeeK3s/LtifiKe0+7 0TI+ljn9E6mg/qgbUN0ZGE0gR4J0UElnkA== X-Google-Smtp-Source: AGHT+IEpGbWyZelZscO/0nmA7mea1e9wKLjzI4BSZDQl6vra1cDlCl3mTCIxv1LDN6py9hL5JvFXmA== X-Received: by 2002:a17:907:a0c6:b0:9a2:26d8:f184 with SMTP id hw6-20020a170907a0c600b009a226d8f184mr3460182ejc.51.1693652632874; Sat, 02 Sep 2023 04:03:52 -0700 (PDT) Received: from [192.168.1.10] (80-42-66-93.dynamic.dsl.as9105.com. [80.42.66.93]) by smtp.gmail.com with ESMTPSA id z15-20020a17090655cf00b0098dfec235ccsm3273884ejp.47.2023.09.02.04.03.52 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 02 Sep 2023 04:03:52 -0700 (PDT) Message-ID: <92116a4b-b46d-1745-d0fb-00703651f789@gmail.com> Date: Sat, 2 Sep 2023 12:03:50 +0100 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Content-Language: en-US To: hackers@freebsd.org References: <5846941.Zv9zXsTiuT@beastie.bionicmutton.org> X-Priority: 4 (Low) Cc: "Cherry G. Mathew" , Adriaan de Groot , Alexander Leidinger From: Graham Perrin Subject: Attachments at lists.freebsd.org (was: ARC model specified in spinroot/promela) In-Reply-To: <5846941.Zv9zXsTiuT@beastie.bionicmutton.org> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Queue-Id: 4RdBpy6Pkjz4VbT On 02/09/2023 11:33, Adriaan de Groot wrote: > … Mailing-list archives don't preserve attachments, … OT, but maybe of interest. If an email does reach a FreeBSD-managed list, then (in my experience): attachments /are/ preserved, albeit not represented in a way that's immediately useful. , for example. HTH From nobody Sat Sep 2 12:03:06 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdD7Y3pJJz4rttt for ; Sat, 2 Sep 2023 12:03:21 +0000 (UTC) (envelope-from jhs@berklix.com) Received: from land.berklix.org (land.berklix.org [144.76.10.75]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "land.berklix.org", Issuer "land.berklix.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdD7W6Fnsz4dfW for ; Sat, 2 Sep 2023 12:03:19 +0000 (UTC) (envelope-from jhs@berklix.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of jhs@berklix.com has no SPF policy when checking 144.76.10.75) smtp.mailfrom=jhs@berklix.com; dmarc=none Received: from dell.no.berklix.net (p4fe6d686.dip0.t-ipconnect.de [79.230.214.134]) (authenticated bits=128) by land.berklix.org (8.16.1/8.16.1) with ESMTPSA id 382C3Bk6035975 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=FAIL); Sat, 2 Sep 2023 12:03:12 GMT (envelope-from jhs@berklix.com) Received: from dell.no.berklix.net (localhost [127.0.0.1]) by dell.no.berklix.net (8.16.1/8.16.1) with ESMTPS id 382C369I032322 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 2 Sep 2023 14:03:06 +0200 (CEST) (envelope-from jhs@dell.no.berklix.net) Received: (from jhs@localhost) by dell.no.berklix.net (8.16.1/8.16.1/Submit) id 382C36L4032321; Sat, 2 Sep 2023 14:03:06 +0200 (CEST) (envelope-from jhs) Message-Id: <202309021203.382C36L4032321@dell.no.berklix.net> To: "FreeBSD Hackers" cc: sfd-discuss@mailman3.com Subject: Software Freedom Day From: "Julian H. Stacey" Organization: http://berklix.com/jhs/ User-agent: EXMH on FreeBSD http://www.berklix.eu/free/ X-From: http://www.berklix.eu/~jhs/ List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <32319.1693656186.1@dell.no.berklix.net> Content-Transfer-Encoding: quoted-printable Date: Sat, 02 Sep 2023 14:03:06 +0200 X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.10 / 15.00]; AUTH_NA(1.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; MIME_GOOD(-0.10)[text/plain]; RCPT_COUNT_TWO(0.00)[2]; MLMMJ_DEST(0.00)[hackers@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_SPF_NA(0.00)[no SPF record]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; HAS_ORG_HEADER(0.00)[]; ASN(0.00)[asn:24940, ipnet:144.76.0.0/16, country:DE]; FREEFALL_USER(0.00)[jhs]; ARC_NA(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[berklix.com]; TO_DN_SOME(0.00)[]; RCVD_TLS_LAST(0.00)[] X-Rspamd-Queue-Id: 4RdD7W6Fnsz4dfW Hi FreeBSD Hackers cc sfd-discuss@mailman3.com The 20th annual Software Freedom Day globaly is Sat. 16 Sept. 2023 https://www.softwarefreedomday.org/ It's fun to attend &/or organise a local event. eg a 2011 example from Munich Germany http://www.berklix.org/sfd/ Forwarded from: "Julian H. Stacey" http://berklix.eu/~jh= s/ ------- Forwarded Message From: LM Date: Fri, 1 Sep 2023 09:39:30 -0400 To: sfd-discuss@mailman3.com Precedence: list Reply-To: Open discussions about SFD Subject: [SFD-discuss]Software Freedom Day - virtual event options List-Id: Open discussions about SFD Archived-At: = Archived-At: List-Archive: = List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: We're at 16 days and counting until Software Freedom Day, the 20th edition= . We're up to 24 teams! Unfortunately, there are a couple of continents we haven't covered this year and I do know of people in those areas who would love to attend an event. If there's a virtual part to your Software Freedom Day celebration or you're hosting any hybrid events, please let me know. I'd like to pass the information on to people who can't attend a local SFD event. I'd need time and time zone, how to join and language of choice (human not programming). Would appreciate your help in finding events for those not fortunate enough to have one in their area. Also, would appreciate hearing if you or anyone you know is in North America and might want to get together either to attend or volunteer or help organize an event in that time zone. If you're having trouble finding a local SFD event, please post to this list as well and let us know. If you're considering a virtual or hybrid event, you can find some resources and software links on the Software Freedom Day's video conferencing page: https://wiki.softwarefreedomday.org/VideoConferencing Thanks to everyone involved with Software Freedom Day this year. _______________________________________________ SFD-discuss mailing list -- sfd-discuss@mailman3.com To unsubscribe send an email to sfd-discuss-leave@mailman3.com ------- End of Forwarded Message From nobody Sat Sep 2 17:12:24 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdM0v4qjVz4sKvZ for ; Sat, 2 Sep 2023 17:13:03 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdM0t6Hxwz4b0N; Sat, 2 Sep 2023 17:13:02 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from webmail2.leidinger.net (roundcube.Leidinger.net [192.168.1.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: Alexander@Leidinger.net) by outgoing.leidinger.net (Postfix) with ESMTPSA id 335E4428F; Sat, 2 Sep 2023 19:12:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1693674776; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=JFaym4kbhF1nd4D2xPhqBnJVrpJ5b1Nh1mG6Mxm7ix4=; b=bwQSaYgdONVY0UmgJzUl7N+9drRpM0PXD/3SjoVyzrQP0BjcDZce3LgqmDTO1lxke7UHMt x1cZimQr6a8yv1e1ffOWywM/i9V1fKNcwBDgkRE9U/3YxAn0IbgrONTs2HY9iXTee3PmXe npnijqb3yDBZSp4MdUecoxsGdIMsiUhehofH8setKMgBHAlBkJuD6/Y+elc/in2/I+qKRB A1glYfN7LHN+fXm1ejQ7mOm6rrdfmJAzhpqpjc+/Dh3kwp7/SGJjlD2C9NivYsjs8e4+Si jWZpIMjKVPUlkqQlONP0AOZKXQ0C2JOOPgUoB7TQ/vJ4qU780nSKFgXOR3h5WA== List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Date: Sat, 02 Sep 2023 19:12:24 +0200 From: Alexander Leidinger To: "Mathew, Cherry G.*" Cc: adridg@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: ARC model specified in spinroot/promela In-Reply-To: <858r9o6ee0.fsf@bow.st> References: <85jzt96qjz.fsf@bow.st> <9c424a574cdd39fc879c9ed9192556c0@Leidinger.net> <858r9o6ee0.fsf@bow.st> Message-ID: X-Sender: Alexander@Leidinger.net Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] X-Rspamd-Queue-Id: 4RdM0t6Hxwz4b0N Am 2023-09-02 12:47, schrieb Mathew, Cherry G.*: > Hi Alexander, > >>>>>> "AL" == Alexander Leidinger writes: > > > [...] > > > AL> How long is this supposed to take? For me it took about 2 > AL> seconds to finish. > > Apologies, I should have given more detailed instructions. > > I've organised the process in three steps: > > 1) Generate the model from spec: make spin-gen > 2) Build the model: make spin-build > 3) Run the model: make spin-run > > This is the heavy duty part, which takes up quite a bit of vmem (my > process dies at about 8GB due to lack of swap etc. - makes no sense to > thrash it beyond that without RAM - it slows down a lot). Seems to be single threaded (Adrian, this is with spin 6.5.0 from an about 2 days old ports tree). Takes only 1 CPU. After 20min it is at 25G (20G RES). The system has 64G of RAM and 100G of swap, we will see if this is enough. I will report back when it is finished. Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From nobody Sat Sep 2 17:15:29 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdM406m2nz4sLBH for ; Sat, 2 Sep 2023 17:15:44 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-256) client-digest SHA256) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdM403mRTz4cFc; Sat, 2 Sep 2023 17:15:44 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from webmail2.leidinger.net (roundcube.Leidinger.net [192.168.1.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: Alexander@Leidinger.net) by outgoing.leidinger.net (Postfix) with ESMTPSA id C0B444781; Sat, 2 Sep 2023 19:15:29 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1693674932; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=5004KhCJFKI7unae0yugPQutFVcpvDGQKKd6Y3clMPY=; b=r7+SDvoLRQGn2Wzjwq0Lkiw+IHWP+0A8Ltp+I0VL4tBJXAJ9SmtZSz2vOiYy5oOVp8e/ku muM4LxMQIQ1XptKeW7UgkcLgvJnrYf+st2L0tXDcjB5peIVLQebBzTuLbHtm7mGrDUtbLJ QRlM8/Y+GbA7CYyOI/GWab3/em28v2LaPBZNHHvHPP490pVl1zAmaHF/QXDP06yATV4BLy S7ktjNkfJ3pLKTbRNahjd0Om6zGplABsRuDJuLdUO/0VJb6wt2NwJTFUuaOukMWBknhqAz PuCkUocazmsnXQUyh2yuugcyxQ0am4dZuOHoXwZj6+FFGmEdQwCHo5ZEBCDDFg== List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Date: Sat, 02 Sep 2023 19:15:29 +0200 From: Alexander Leidinger To: "Mathew, Cherry G.*" Cc: adridg@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: ARC model specified in spinroot/promela In-Reply-To: <858r9o6ee0.fsf@bow.st> References: <85jzt96qjz.fsf@bow.st> <9c424a574cdd39fc879c9ed9192556c0@Leidinger.net> <858r9o6ee0.fsf@bow.st> Message-ID: <0b73ec6dd871c8e0c646f31614570058@Leidinger.net> X-Sender: Alexander@Leidinger.net Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE] X-Rspamd-Queue-Id: 4RdM403mRTz4cFc Am 2023-09-02 12:47, schrieb Mathew, Cherry G.*: > Hi Alexander, > >>>>>> "AL" == Alexander Leidinger writes: > > > [...] > > > AL> How long is this supposed to take? For me it took about 2 > AL> seconds to finish. > > Apologies, I should have given more detailed instructions. > > I've organised the process in three steps: > > 1) Generate the model from spec: make spin-gen > 2) Build the model: make spin-build > 3) Run the model: make spin-run > > This is the heavy duty part, which takes up quite a bit of vmem (my > process dies at about 8GB due to lack of swap etc. - makes no sense to > thrash it beyond that without RAM - it slows down a lot). > > If all goes well with step 3) , then you will see a summary of the run > on console. However, if there was an inconsistency or error detected in > the run, then a tracefile is generated (spinmodel.pml.trail). I've > included a target to dump a human friendly version of this trace. And just when I've send the mail, it finished... :) ---snip--- Samstag, 02. September 2023, 18:51:21 (203) root@ttypts/1 # time make spin-gen cp arc.pml model #mimic modex cat arc.drv > spinmodel.pml;cat model >> spinmodel.pml;cat arc.inv >> spinmodel.pml; spin -a spinmodel.pml ltl ltl_0: ((<> ([] ((_nr_pr==1)))) && ([] (((((len(T1.item_list)<=5)) && ((len(B1.item_list)<=5))) && ((len(T2.item_list)<=5))) && ((len(B2.item_list)<=5))))) && (<> ((p>0))) make spin-gen 0,04s user 0,11s system 24% cpu 0,585 total Samstag, 02. September 2023, 18:51:29 (204) root@ttypts/1 # time make spin-build cc -o pan pan.c make spin-build 0,47s user 0,26s system 56% cpu 1,286 total Samstag, 02. September 2023, 18:51:37 (205) root@ttypts/1 # time make spin-run cc -o pan pan.c ./pan -a #Generate arc.pml.trail on error Depth= 271 States= 1e+06 Transitions= 2.04e+06 Memory= 194.550 t= 4.47 R= 2e+05 Depth= 271 States= 2e+06 Transitions= 4.09e+06 Memory= 264.081 t= 9.16 R= 2e+05 Depth= 271 States= 3e+06 Transitions= 6.16e+06 Memory= 334.101 t= 13.5 R= 2e+05 Depth= 271 States= 4e+06 Transitions= 8.25e+06 Memory= 410.370 t= 18.4 R= 2e+05 Depth= 271 States= 5e+06 Transitions= 1.04e+07 Memory= 485.272 t= 23.4 R= 2e+05 Depth= 271 States= 6e+06 Transitions= 1.25e+07 Memory= 560.175 t= 28 R= 2e+05 Depth= 271 States= 7e+06 Transitions= 1.46e+07 Memory= 635.077 t= 33.1 R= 2e+05 Depth= 271 States= 8e+06 Transitions= 1.67e+07 Memory= 710.077 t= 38 R= 2e+05 Depth= 271 States= 9e+06 Transitions= 1.88e+07 Memory= 787.812 t= 43.2 R= 2e+05 Depth= 271 States= 1e+07 Transitions= 2.09e+07 Memory= 867.987 t= 48.5 R= 2e+05 Depth= 271 States= 1.1e+07 Transitions= 2.3e+07 Memory= 948.163 t= 53.5 R= 2e+05 Depth= 271 States= 1.2e+07 Transitions= 2.51e+07 Memory= 1028.339 t= 58.7 R= 2e+05 Depth= 271 States= 1.3e+07 Transitions= 2.72e+07 Memory= 1108.515 t= 63.8 R= 2e+05 Depth= 271 States= 1.4e+07 Transitions= 2.93e+07 Memory= 1188.202 t= 69.1 R= 2e+05 Depth= 271 States= 1.5e+07 Transitions= 3.15e+07 Memory= 1267.401 t= 74.5 R= 2e+05 Depth= 271 States= 1.6e+07 Transitions= 3.36e+07 Memory= 1346.503 t= 79.9 R= 2e+05 Depth= 271 States= 1.7e+07 Transitions= 3.58e+07 Memory= 1425.605 t= 85.2 R= 2e+05 [...] Depth= 271 States= 2.07e+08 Transitions= 4.44e+08 Memory= 20175.351 t= 1.22e+03 R= 2e+05 Depth= 271 States= 2.08e+08 Transitions= 4.46e+08 Memory= 20262.558 t= 1.22e+03 R= 2e+05 Depth= 271 States= 2.09e+08 Transitions= 4.49e+08 Memory= 20349.667 t= 1.23e+03 R= 2e+05 pan:1: acceptance cycle (at depth 270) pan: wrote spinmodel.pml.trail (Spin Version 6.5.0 -- 1 July 2019) Warning: Search not completed + Partial Order Reduction Full statespace search for: never claim + (ltl_0) assertion violations + (if within scope of claim) acceptance cycles + (fairness disabled) invalid end states - (disabled by never claim) State-vector 184 byte, depth reached 271, errors: 1 1.0465042e+08 states, stored (2.09301e+08 visited) 2.4007432e+08 states, matched 4.4937506e+08 transitions (= visited+matched) 44667618 atomic steps hash conflicts: 63844226 (resolved) Stats on memory usage (in Megabytes): 21158.112 equivalent memory usage for states (stored*(State-vector + overhead)) 18344.457 actual memory usage for states (compression: 86.70%) state-vector as stored = 156 byte + 28 byte overhead 2048.000 memory used for hash table (-w28) 0.534 memory used for DFS stack (-m10000) 17.170 memory lost to fragmentation 20375.839 total actual memory usage pan: elapsed time 1.23e+03 seconds pan: rate 169922.52 states/second make spin-run 1199,05s user 34,46s system 99% cpu 20:33,94 total ---snip--- Hope this helps, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From nobody Sat Sep 2 17:17:52 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdM7B30rHz4sLsc for ; Sat, 2 Sep 2023 17:18:30 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Received: from mailgate.Leidinger.net (bastille.leidinger.net [89.238.82.207]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) client-signature ECDSA (P-256)) (Client CN "mailgate.leidinger.net", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdM7B0j7Bz4cms; Sat, 2 Sep 2023 17:18:30 +0000 (UTC) (envelope-from Alexander@Leidinger.net) Authentication-Results: mx1.freebsd.org; none Received: from webmail2.leidinger.net (roundcube.Leidinger.net [192.168.1.123]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: Alexander@Leidinger.net) by outgoing.leidinger.net (Postfix) with ESMTPSA id 46513440C; Sat, 2 Sep 2023 19:17:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1693675104; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=gLCt2XjBP4dH47CUlsURCYfRppsJxWEhUcVMVW31e5Y=; b=kDweZdYdAx7QL+NP3QmBrsGLPi4yWyztFJO7aRIsnU4ezoUCIGIxGaelo+eanv0gV1CqVd VXHDHGnyP0p/A/yK2L3zA+v2X18a7J/5VHwypLIayHt7FYwfrT2bmbxJJRH3A7W7h1QIxI KCx3qZMIaYW893eQ6MsVlNQWQG7d7KR22EuUkUEBnP4Kum3PgA/O/t86sQMcNQAD5lbshI /thaympHlfUJVPYregktEFGxCnow/LnQNSxgQ/oUmRy3HUw+prUzsGRaQ6naylgPDqXPLK 1cA7XxgQ+/f50BWzDco3vW8ymDBqfylBL8GwsFXWNySetFqKv2AgJZr6/46C/w== List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Date: Sat, 02 Sep 2023 19:17:52 +0200 From: Alexander Leidinger To: "Mathew, Cherry G.*" Cc: adridg@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: ARC model specified in spinroot/promela In-Reply-To: <858r9o6ee0.fsf@bow.st> References: <85jzt96qjz.fsf@bow.st> <9c424a574cdd39fc879c9ed9192556c0@Leidinger.net> <858r9o6ee0.fsf@bow.st> Message-ID: <39eb9c6f36d743a3cf71632bdc36a18e@Leidinger.net> X-Sender: Alexander@Leidinger.net Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:34240, ipnet:89.238.64.0/18, country:DE] X-Rspamd-Queue-Id: 4RdM7B0j7Bz4cms Am 2023-09-02 12:47, schrieb Mathew, Cherry G.*: > 4) Print trace from debug trail: make spin-trace (207) root@ttypts/1 # make spin-trace spin -t spinmodel.pml -p -g # -p (statements) -g (globals) -l (locals) -s (send) -r (recv) ltl ltl_0: ((<> ([] ((_nr_pr==1)))) && ([] (((((len(T1.item_list)<=5)) && ((len(B1.item_list)<=5))) && ((len(T2.item_list)<=5))) && ((len(B2.item_list)<=5))))) && (<> ((p>0))) Never claim moves to line 5 [(!((p>0)))] Never claim moves to line 15 [(!((p>0)))] <<<<>>>> spin: trail ends after 272 steps #processes: 0 queue 2 (item_list): queue 4 (item_list): queue 1 (item_list): [1] queue 3 (item_list): [5][4][3][2] _x[0].iid = 0 _x[0].cached = 0 _x[1].iid = 1 _x[1].cached = 0 _x[2].iid = 2 _x[2].cached = 0 _x[3].iid = 3 _x[3].cached = 0 _x[4].iid = 4 _x[4].cached = 0 _x[5].iid = 5 _x[5].cached = 0 _x_iid = 6 _item_rep = 5 LRUitem.iid = 0 LRUitem.cached = 0 d1 = 0 d2 = 0 p = 0 sc_lock = 0 272: proc - (ltl_0:1) _spin_nvr.tmp:14 (state 22) 16 processes created ./pan -r spinmodel.pml.trail -g MSC: ~G 3 1: proc 0 (ltl_0) spinmodel.pml:3 (state 12) [(!((p>0)))] 2: proc 1 (:init:) spinmodel.pml:79 (state 21) [x_iid = 0] 3: proc 1 (:init:) spinmodel.pml:81 (state 18) [((x_iid<6))] 4: proc 1 (:init:) spinmodel.pml:82 (state 5) [D_STEP82] 5: proc 1 (:init:) spinmodel.pml:83 (state 6) [item_rep = 0] 6: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep>=(x_iid%6)))] 7: proc 1 (:init:) spinmodel.pml:81 (state 18) [((x_iid<6))] 8: proc 1 (:init:) spinmodel.pml:82 (state 5) [D_STEP82] 9: proc 1 (:init:) spinmodel.pml:83 (state 6) [item_rep = 0] 10: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 11: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 12: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 13: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep>=(x_iid%6)))] 14: proc 1 (:init:) spinmodel.pml:81 (state 18) [((x_iid<6))] 15: proc 1 (:init:) spinmodel.pml:82 (state 5) [D_STEP82] 16: proc 1 (:init:) spinmodel.pml:83 (state 6) [item_rep = 0] 17: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 18: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 19: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 20: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 21: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 22: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 23: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep>=(x_iid%6)))] 24: proc 1 (:init:) spinmodel.pml:81 (state 18) [((x_iid<6))] 25: proc 1 (:init:) spinmodel.pml:82 (state 5) [D_STEP82] 26: proc 1 (:init:) spinmodel.pml:83 (state 6) [item_rep = 0] 27: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 28: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 29: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 30: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 31: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 32: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 33: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 34: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 35: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 36: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep>=(x_iid%6)))] 37: proc 1 (:init:) spinmodel.pml:81 (state 18) [((x_iid<6))] 38: proc 1 (:init:) spinmodel.pml:82 (state 5) [D_STEP82] 39: proc 1 (:init:) spinmodel.pml:83 (state 6) [item_rep = 0] 40: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 41: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 42: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 43: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 44: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 45: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 46: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 47: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 48: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 49: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 50: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 51: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 52: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep>=(x_iid%6)))] 53: proc 1 (:init:) spinmodel.pml:81 (state 18) [((x_iid<6))] 54: proc 1 (:init:) spinmodel.pml:82 (state 5) [D_STEP82] 55: proc 1 (:init:) spinmodel.pml:83 (state 6) [item_rep = 0] 56: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 57: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 58: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 59: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 60: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 61: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 62: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 63: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 64: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 65: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 66: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 67: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 68: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep<(x_iid%6)))] 69: proc 1 (:init:) spinmodel.pml:86 (state 8) [(run p_arc(x[x_iid].iid,x[x_iid].cached))] 70: proc 1 (:init:) spinmodel.pml:87 (state 9) [item_rep = (item_rep+1)] 71: proc 1 (:init:) spinmodel.pml:85 (state 12) [((item_rep>=(x_iid%6)))] 72: proc 1 (:init:) spinmodel.pml:81 (state 18) [((x_iid>=6))] 73: proc 1 (:init:) spinmodel.pml:80 (state 20) [break] MSC: ~G 15 74: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 75: proc 16 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 76: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 77: proc 16 (p_arc) spinmodel.pml:192 (state 126) [(!((((T1.item_list??[eval(x_t.iid)]||B1.item_list??[eval(x_t.iid)])||T2.item_list??[eval(x_t.iid)])||B2.item_list??[eval(x_t.iid)])))] 78: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 79: proc 16 (p_arc) spinmodel.pml:237 (state 122) [(((len(T1.item_list)+len(B1.item_list))<5))] 80: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 81: proc 16 (p_arc) spinmodel.pml:263 (state 120) [else] 82: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 83: proc 16 (p_arc) spinmodel.pml:282 (state 119) [(1)] 84: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 85: proc 16 (p_arc) spinmodel.pml:286 (state 125) [values: 3!5] 85: proc 16 (p_arc) spinmodel.pml:286 (state 125) [T1.item_list!x_t.iid] 86: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 87: proc 16 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 88: proc 16 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 89: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 90: proc 16 (p_arc) -:0 (state 0) [-end-] 91: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 92: proc 15 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 93: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 94: proc 15 (p_arc) spinmodel.pml:192 (state 126) [(T1.item_list??[eval(x_t.iid)])] 95: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 96: proc 15 (p_arc) spinmodel.pml:194 (state 7) [values: 4!5] 96: proc 15 (p_arc) spinmodel.pml:194 (state 7) [D_STEP194] 97: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 98: proc 15 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 99: proc 15 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 100: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 101: proc 15 (p_arc) -:0 (state 0) [-end-] 102: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 103: proc 14 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 104: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 105: proc 14 (p_arc) spinmodel.pml:192 (state 126) [(T2.item_list??[eval(x_t.iid)])] 106: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 107: proc 14 (p_arc) spinmodel.pml:201 (state 11) [values: 4!5] 107: proc 14 (p_arc) spinmodel.pml:201 (state 11) [D_STEP201] 108: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 109: proc 14 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 110: proc 14 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 111: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 112: proc 14 (p_arc) -:0 (state 0) [-end-] 113: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 114: proc 13 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 115: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 116: proc 13 (p_arc) spinmodel.pml:192 (state 126) [(T2.item_list??[eval(x_t.iid)])] 117: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 118: proc 13 (p_arc) spinmodel.pml:201 (state 11) [values: 4!5] 118: proc 13 (p_arc) spinmodel.pml:201 (state 11) [D_STEP201] 119: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 120: proc 13 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 121: proc 13 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 122: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 123: proc 13 (p_arc) -:0 (state 0) [-end-] 124: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 125: proc 12 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 126: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 127: proc 12 (p_arc) spinmodel.pml:192 (state 126) [(T2.item_list??[eval(x_t.iid)])] 128: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 129: proc 12 (p_arc) spinmodel.pml:201 (state 11) [values: 4!5] 129: proc 12 (p_arc) spinmodel.pml:201 (state 11) [D_STEP201] 130: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 131: proc 12 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 132: proc 12 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 133: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 134: proc 12 (p_arc) -:0 (state 0) [-end-] 135: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 136: proc 11 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 137: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 138: proc 11 (p_arc) spinmodel.pml:192 (state 126) [(!((((T1.item_list??[eval(x_t.iid)]||B1.item_list??[eval(x_t.iid)])||T2.item_list??[eval(x_t.iid)])||B2.item_list??[eval(x_t.iid)])))] 139: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 140: proc 11 (p_arc) spinmodel.pml:237 (state 122) [(((len(T1.item_list)+len(B1.item_list))<5))] 141: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 142: proc 11 (p_arc) spinmodel.pml:263 (state 120) [else] 143: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 144: proc 11 (p_arc) spinmodel.pml:282 (state 119) [(1)] 145: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 146: proc 11 (p_arc) spinmodel.pml:286 (state 125) [values: 3!4] 146: proc 11 (p_arc) spinmodel.pml:286 (state 125) [T1.item_list!x_t.iid] 147: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 148: proc 11 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 149: proc 11 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 150: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 151: proc 11 (p_arc) -:0 (state 0) [-end-] 152: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 153: proc 10 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 154: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 155: proc 10 (p_arc) spinmodel.pml:192 (state 126) [(T1.item_list??[eval(x_t.iid)])] 156: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 157: proc 10 (p_arc) spinmodel.pml:194 (state 7) [values: 4!4] 157: proc 10 (p_arc) spinmodel.pml:194 (state 7) [D_STEP194] 158: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 159: proc 10 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 160: proc 10 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 161: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 162: proc 10 (p_arc) -:0 (state 0) [-end-] 163: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 164: proc 9 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 165: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 166: proc 9 (p_arc) spinmodel.pml:192 (state 126) [(T2.item_list??[eval(x_t.iid)])] 167: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 168: proc 9 (p_arc) spinmodel.pml:201 (state 11) [values: 4!4] 168: proc 9 (p_arc) spinmodel.pml:201 (state 11) [D_STEP201] 169: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 170: proc 9 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 171: proc 9 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 172: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 173: proc 9 (p_arc) -:0 (state 0) [-end-] 174: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 175: proc 8 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 176: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 177: proc 8 (p_arc) spinmodel.pml:192 (state 126) [(T2.item_list??[eval(x_t.iid)])] 178: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 179: proc 8 (p_arc) spinmodel.pml:201 (state 11) [values: 4!4] 179: proc 8 (p_arc) spinmodel.pml:201 (state 11) [D_STEP201] 180: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 181: proc 8 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 182: proc 8 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 183: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 184: proc 8 (p_arc) -:0 (state 0) [-end-] 185: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 186: proc 7 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 187: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 188: proc 7 (p_arc) spinmodel.pml:192 (state 126) [(!((((T1.item_list??[eval(x_t.iid)]||B1.item_list??[eval(x_t.iid)])||T2.item_list??[eval(x_t.iid)])||B2.item_list??[eval(x_t.iid)])))] 189: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 190: proc 7 (p_arc) spinmodel.pml:237 (state 122) [(((len(T1.item_list)+len(B1.item_list))<5))] 191: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 192: proc 7 (p_arc) spinmodel.pml:263 (state 120) [else] 193: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 194: proc 7 (p_arc) spinmodel.pml:282 (state 119) [(1)] 195: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 196: proc 7 (p_arc) spinmodel.pml:286 (state 125) [values: 3!3] 196: proc 7 (p_arc) spinmodel.pml:286 (state 125) [T1.item_list!x_t.iid] 197: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 198: proc 7 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 199: proc 7 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 200: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 201: proc 7 (p_arc) -:0 (state 0) [-end-] 202: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 203: proc 6 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 204: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 205: proc 6 (p_arc) spinmodel.pml:192 (state 126) [(T1.item_list??[eval(x_t.iid)])] 206: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 207: proc 6 (p_arc) spinmodel.pml:194 (state 7) [values: 4!3] 207: proc 6 (p_arc) spinmodel.pml:194 (state 7) [D_STEP194] 208: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 209: proc 6 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 210: proc 6 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 211: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 212: proc 6 (p_arc) -:0 (state 0) [-end-] 213: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 214: proc 5 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 215: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 216: proc 5 (p_arc) spinmodel.pml:192 (state 126) [(T2.item_list??[eval(x_t.iid)])] 217: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 218: proc 5 (p_arc) spinmodel.pml:201 (state 11) [values: 4!3] 218: proc 5 (p_arc) spinmodel.pml:201 (state 11) [D_STEP201] 219: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 220: proc 5 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 221: proc 5 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 222: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 223: proc 5 (p_arc) -:0 (state 0) [-end-] 224: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 225: proc 4 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 226: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 227: proc 4 (p_arc) spinmodel.pml:192 (state 126) [(!((((T1.item_list??[eval(x_t.iid)]||B1.item_list??[eval(x_t.iid)])||T2.item_list??[eval(x_t.iid)])||B2.item_list??[eval(x_t.iid)])))] 228: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 229: proc 4 (p_arc) spinmodel.pml:237 (state 122) [(((len(T1.item_list)+len(B1.item_list))<5))] 230: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 231: proc 4 (p_arc) spinmodel.pml:263 (state 120) [else] 232: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 233: proc 4 (p_arc) spinmodel.pml:282 (state 119) [(1)] 234: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 235: proc 4 (p_arc) spinmodel.pml:286 (state 125) [values: 3!2] 235: proc 4 (p_arc) spinmodel.pml:286 (state 125) [T1.item_list!x_t.iid] 236: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 237: proc 4 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 238: proc 4 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 239: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 240: proc 4 (p_arc) -:0 (state 0) [-end-] 241: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 242: proc 3 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 243: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 244: proc 3 (p_arc) spinmodel.pml:192 (state 126) [(T1.item_list??[eval(x_t.iid)])] 245: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 246: proc 3 (p_arc) spinmodel.pml:194 (state 7) [values: 4!2] 246: proc 3 (p_arc) spinmodel.pml:194 (state 7) [D_STEP194] 247: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 248: proc 3 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 249: proc 3 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 250: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 251: proc 3 (p_arc) -:0 (state 0) [-end-] 252: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 253: proc 2 (p_arc) spinmodel.pml:188 (state 3) [((sc_lock==0))] 254: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 255: proc 2 (p_arc) spinmodel.pml:192 (state 126) [(!((((T1.item_list??[eval(x_t.iid)]||B1.item_list??[eval(x_t.iid)])||T2.item_list??[eval(x_t.iid)])||B2.item_list??[eval(x_t.iid)])))] 256: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 257: proc 2 (p_arc) spinmodel.pml:237 (state 122) [(((len(T1.item_list)+len(B1.item_list))<5))] 258: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 259: proc 2 (p_arc) spinmodel.pml:263 (state 120) [else] 260: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 261: proc 2 (p_arc) spinmodel.pml:282 (state 119) [(1)] 262: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 263: proc 2 (p_arc) spinmodel.pml:286 (state 125) [values: 3!1] 263: proc 2 (p_arc) spinmodel.pml:286 (state 125) [T1.item_list!x_t.iid] 264: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 265: proc 2 (p_arc) spinmodel.pml:289 (state 131) [assert((sc_lock==1))] 266: proc 2 (p_arc) spinmodel.pml:289 (state 129) [((sc_lock==1))] 267: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 268: proc 2 (p_arc) -:0 (state 0) [-end-] 269: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 270: proc 1 (:init:) -:0 (state 0) [-end-] <<<<>>>> 271: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] 272: proc 0 (ltl_0) spinmodel.pml:15 (state 22) [(!((p>0)))] spin: trail ends after 272 steps #processes 1: 272: proc 0 (ltl_0) spinmodel.pml:15 (state 22) (invalid end state) (!((p>0))) global vars: (struct B1) chan item_list (=1): len 0: (struct B2) chan item_list (=2): len 0: (struct T1) chan item_list (=3): len 1: [1,], (struct T2) chan item_list (=4): len 4: [5,], [4,], [3,], [2,], byte p: 0 bit sc_lock: 0 Bye, Alexander. -- http://www.Leidinger.net Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF From nobody Sat Sep 2 18:38:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdNwS6WC7z4rWsH for ; Sat, 2 Sep 2023 18:39:20 +0000 (UTC) (envelope-from c@bow.st) Received: from comms.drone (in.bow.st [71.19.146.166]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdNwR22lxz3NhG; Sat, 2 Sep 2023 18:39:19 +0000 (UTC) (envelope-from c@bow.st) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of c@bow.st designates 71.19.146.166 as permitted sender) smtp.mailfrom=c@bow.st; dmarc=none Received: from homebase (unknown [IPv6:fe80::ff1d:976a:a7e4:ee6a]) by comms.drone (Postfix) with ESMTPSA id 72AF5FCFE; Sat, 2 Sep 2023 18:39:13 +0000 (UTC) From: "Mathew\, Cherry G.*" To: Alexander Leidinger Cc: adridg@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: ARC model specified in spinroot/promela References: <85jzt96qjz.fsf@bow.st> <9c424a574cdd39fc879c9ed9192556c0@Leidinger.net> <858r9o6ee0.fsf@bow.st> Date: Sat, 02 Sep 2023 18:38:41 +0000 In-Reply-To: (Alexander Leidinger's message of "Sat, 02 Sep 2023 19:12:24 +0200") Message-ID: <85pm304dzi.fsf@bow.st> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/26.3 (berkeley-unix) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain X-Spamd-Bar: - X-Spamd-Result: default: False [-1.88 / 15.00]; HFILTER_HELO_IP_A(1.00)[comms.drone]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; NEURAL_HAM_SHORT(-1.00)[-0.995]; NEURAL_HAM_LONG(-0.99)[-0.991]; HFILTER_HELO_NORES_A_OR_MX(0.30)[comms.drone]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; ONCE_RECEIVED(0.10)[]; RCPT_COUNT_THREE(0.00)[3]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_ALL(0.00)[]; ASN(0.00)[asn:47066, ipnet:71.19.146.0/24, country:US]; TO_DN_SOME(0.00)[]; RCVD_COUNT_ONE(0.00)[1]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[bow.st]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 4RdNwR22lxz3NhG >>>>> Alexander Leidinger writes: [...] > Seems to be single threaded (Adrian, this is with spin 6.5.0 from > an about 2 days old ports tree). Takes only 1 CPU. After 20min it > is at 25G (20G RES). The system has 64G of RAM and 100G of swap, > we will see if this is enough. Thank you so much for this, and the trace output in the subsequent email! So there's room for improvement in my Makefile to: 1) Explore multithreaded/CPU 2) Understand state space explosion scale - it is exponential to the number of processes, I've fixed it to run in a loop - will report back once I have good progress also with the model extraction part. 3) My model has errors - I'm glad to see that the error reporting is the same in my new optimised loop, as in the RAM hungry version you ran. Thanks once again - this is very helpful. I will report back once I have more progress. -- ~cherry From nobody Sat Sep 2 20:05:37 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdQrD4L4vz4ryWj; Sat, 2 Sep 2023 20:05:48 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdQrB6fT9z3d0t; Sat, 2 Sep 2023 20:05:46 +0000 (UTC) (envelope-from wojtek@puchar.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=puchar.net header.s=default header.b=PTQBi48U; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net; dmarc=none Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.17.1) with ESMTPS id 382K5cnd002631 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 2 Sep 2023 22:05:39 +0200 (CEST) (envelope-from wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1693685139; bh=WBzFC9kJ3eEQu3jg/DcjA+Yb+r7p+ldIvfgK5fvxVjY=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=PTQBi48Uzph2P1EnDa0kPRnzg4JWEQbwPflsXjs7E30z7tc4VIcW18qzNGHtcNO99 C5OJrEvl9pgAgaSKGINKfAYRupGJF6c8fegQOUT8kuwlxu0yvZjOOp2Y8viYn1nETw HQTbgWvsLUW73ScNsJR4exWk83XORplvrpKVA5GU= Received: from wojtek.intra (localhost [127.0.0.1]) by wojtek.intra (8.16.1/8.16.1) with ESMTP id 382K5bBd050813; Sat, 2 Sep 2023 22:05:37 +0200 (CEST) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by wojtek.intra (8.16.1/8.16.1/Submit) with ESMTP id 382K5bQg050810; Sat, 2 Sep 2023 22:05:37 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: wojtek.intra: wojtek owned process doing -bs Date: Sat, 2 Sep 2023 22:05:37 +0200 (CEST) From: Wojciech Puchar To: Wei Hu cc: "freebsd-hackers@FreeBSD.org" , "freebsd-net@FreeBSD.org" Subject: Re: Very slow scp performance comparing to Linux In-Reply-To: Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.997]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-net@freebsd.org,freebsd-hackers@freebsd.org]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; RCVD_COUNT_TWO(0.00)[2]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[puchar.net:+]; TO_DN_EQ_ADDR_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_SOME(0.00)[]; HAS_XAW(0.00)[]; FROM_HAS_DN(0.00)[]; DMARC_NA(0.00)[puchar.net]; RWL_MAILSPIKE_POSSIBLE(0.00)[194.1.144.90:from] X-Rspamd-Queue-Id: 4RdQrB6fT9z3d0t tested on really today's low end laptop CPU: Intel(R) Pentium(R) CPU N3700 @ 1.60GHz (1600.19-MHz K8-class CPU) got 73MB/s from tmpfs to tmpfs, most of it was user time of ssh got 69MB/s from SSD to SSD, same. Test on server computer CPU: Intel(R) Pentium(R) CPU N3700 @ 1.60GHz (1600.19-MHz K8-class CPU) 337MB/s tmpfs to tmpfs 317MB/s ssd to ssd too mostly user time (ssh and sshd, encryption/decryption). I don't see anything wrong. What to check: a) linux ssh may have encryption off somehome b) you tested in on slow storage explanation of b - linux doesn't write to disk as long as it doesn't run of of RAM. FreeBSD writes to disk just as large block is ready, then write hell lot of data to disk, often blocking everything. So Linux looks great in stupid benchmarks. FreeBSD gives you good all-round performance with many processes working. From nobody Sat Sep 2 20:11:42 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdQzB66dBz4s0kN for ; Sat, 2 Sep 2023 20:11:50 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdQz64Mrcz3fTv for ; Sat, 2 Sep 2023 20:11:46 +0000 (UTC) (envelope-from wojtek@puchar.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=puchar.net header.s=default header.b=eeDnbsGW; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net; dmarc=none Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.17.1) with ESMTPS id 382KBhqv005001 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 2 Sep 2023 22:11:44 +0200 (CEST) (envelope-from wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1693685504; bh=qKtSOpQ2JqUrUTAeRrOnFHtrbLLxksUsamYd0pvxYiE=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=eeDnbsGW/BgpmBZ3KWivcRLDUcI+wxLH1fxXrEbgp5QHKQnyU7ipMzz4zLhZAXfc7 rk1qwxLsaqVowAe2Z+KbuAp/6doRlJzn5Y6Ckm8UoWGmwJM2hxcwRV2hwdiMtzmsDJ kdhpkk3vKPBLdUCdU3Uc+VNLw7ZilfQ8ngFkVp/Y= Received: from wojtek.intra (localhost [127.0.0.1]) by wojtek.intra (8.16.1/8.16.1) with ESMTP id 382KBhZU050851; Sat, 2 Sep 2023 22:11:43 +0200 (CEST) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by wojtek.intra (8.16.1/8.16.1/Submit) with ESMTP id 382KBgTj050848; Sat, 2 Sep 2023 22:11:42 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: wojtek.intra: wojtek owned process doing -bs Date: Sat, 2 Sep 2023 22:11:42 +0200 (CEST) From: Wojciech Puchar To: Wei Hu cc: Guido Falsi , Mikhail Zakharov , "freebsd-hackers@freebsd.org" Subject: RE: Very slow scp performance comparing to Linux In-Reply-To: Message-ID: References: <6952b7fd-f90f-2677-1997-d0c708cb559e@yahoo.com> <455dc69a-dd15-18d9-8e93-91bbff3997e4@madpilot.net> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_ALLOW(-0.20)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_CC(0.00)[madpilot.net,yahoo.com,freebsd.org]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; DKIM_TRACE(0.00)[puchar.net:+]; TO_DN_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[194.1.144.90:from]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; HAS_XAW(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[puchar.net]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RdQz64Mrcz3fTv > The throughput on FreeBSD went up significantly from 50 MB/s to 630 MB/s with > NIC interface. Linux went up modestly form 550 MB/s to 660 MB/s. > > So, looks in the non-tmpfs case, the Linux ext4 filesystem does cache large amounts > of data in memory, much larger than FreeBSD ufs. > linux filesystem caching is just stupid, optimized to show up good in benchmarks. It just accepts writes as long as there is available memory, without writing anything to disk, then if it cannot do it anymore or some time passed, schedules huge amount of writes, often stalling whole system. This is how it was almost 20 years ago when i was still using linux, seems like didn't change much. Just think how much data loss and inconsistency would occur if there will be crash or power loss after "writing" 100000 files of 10GB which can easily be done in short time as linux just doesn't block writing process at all and fill memory. Since i started using FreeBSD UFS, many machines, lighter of heavier workload, always mixed workload, i never ever have more that a few files loss on power outage. From nobody Sat Sep 2 20:13:15 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdR0z0t12z4s1Q2 for ; Sat, 2 Sep 2023 20:13:23 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdR0y2ffbz3gVC for ; Sat, 2 Sep 2023 20:13:22 +0000 (UTC) (envelope-from wojtek@puchar.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=puchar.net header.s=default header.b=JpceOdKw; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net; dmarc=none Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.17.1) with ESMTPS id 382KDGA1006600 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 2 Sep 2023 22:13:19 +0200 (CEST) (envelope-from wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1693685600; bh=5Q7sOqBgmUdPRThSHhLxgL7F4elr75oMnKOkdcPQNU0=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=JpceOdKwmtjq6RMlJh/SotJCy54nEbgmuz+5DiG7VvosJq/fm1iy/vEfngidYBdEI fJBiB7ehUFUuB01w7DOUYosWubA2yGWVP4w3I/t5jkvtDsXP21TRQIbVv0GwVcS0x3 9icQyB6C+LI6fEyTPZksLo81oNb6nyPgbt2Ine+8= Received: from wojtek.intra (localhost [127.0.0.1]) by wojtek.intra (8.16.1/8.16.1) with ESMTP id 382KDF0b050864; Sat, 2 Sep 2023 22:13:15 +0200 (CEST) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by wojtek.intra (8.16.1/8.16.1/Submit) with ESMTP id 382KDFUl050861; Sat, 2 Sep 2023 22:13:15 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: wojtek.intra: wojtek owned process doing -bs Date: Sat, 2 Sep 2023 22:13:15 +0200 (CEST) From: Wojciech Puchar To: Wei Hu cc: Mark Millard , FreeBSD Hackers Subject: RE: Very slow scp performance comparing to Linux In-Reply-To: Message-ID: <59b9639e-80c7-38c5-a55d-7fa11b431bad@puchar.net> References: <948CAEBD-EB60-46B9-96EE-FE41CA6C64A1@yahoo.com> <07C2C9E3-7317-43AF-A60C-393ADF90079D@yahoo.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; R_DKIM_ALLOW(-0.20)[puchar.net:s=default]; R_SPF_ALLOW(-0.20)[+mx:c]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_CC(0.00)[yahoo.com,freebsd.org]; DKIM_TRACE(0.00)[puchar.net:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[194.1.144.90:from]; TO_DN_ALL(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[puchar.net]; HAS_XAW(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RdR0y2ffbz3gVC > 1. ext4 caches a lot more than ufs? > 2. there is a tcp performance gap in the network stack between FreeBSD and Ubuntu? > > Would you also try run scp on ufs on your bare metal arm host? I am curious to now how different between ufs and zfs. > run on the same machine with FreeBSD directly. Compare to your VM host. Personally i never run FreeBSD on VM except for tests. Exactly reverse since bhyve exist. From nobody Sat Sep 2 20:16:15 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdR4Q3SsGz4s2Fr for ; Sat, 2 Sep 2023 20:16:22 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdR4P5GThz4Dc4 for ; Sat, 2 Sep 2023 20:16:21 +0000 (UTC) (envelope-from wojtek@puchar.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=puchar.net header.s=default header.b=X8I3KKdY; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net; dmarc=none Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.17.1) with ESMTPS id 382KGHeW007143 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 2 Sep 2023 22:16:19 +0200 (CEST) (envelope-from wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1693685779; bh=CTfEOuu+STV06nHTR7cY77IWdo2+LUtArYyoLjbK2jQ=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=X8I3KKdYV+GZenE5jdTTq/+q8lAdKdeeY554FbsSqNY6+M7ucsPicrcjg18NqLMmA SjxSY/NLCyYqlObZd+mzjoZht2bXqmxkBPFCsEIG9KHMzVsiWYe+S0CcRhb941kmjh QLxW6s8uIbMqtn530InHTByca7ojKnu5us4g3cDA= Received: from wojtek.intra (localhost [127.0.0.1]) by wojtek.intra (8.16.1/8.16.1) with ESMTP id 382KGG08050881; Sat, 2 Sep 2023 22:16:16 +0200 (CEST) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by wojtek.intra (8.16.1/8.16.1/Submit) with ESMTP id 382KGFjp050878; Sat, 2 Sep 2023 22:16:15 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: wojtek.intra: wojtek owned process doing -bs Date: Sat, 2 Sep 2023 22:16:15 +0200 (CEST) From: Wojciech Puchar To: Wei Hu cc: Mark Millard , Mark Saad , FreeBSD Hackers Subject: RE: Very slow scp performance comparing to Linux [dd to /dev/null shows substantial FreeBSD vs. Ubuntu differences for bs=1k (or 1K) and bs=512] In-Reply-To: Message-ID: References: <2D466F3C-527C-4EE1-8C3D-3E8CDD8D547F@yahoo.com> <543FBABC-C75A-4AB0-BFB6-1C7C15ECBA4E@longcount.org> <96A99928-4E8E-4AB1-BF3A-D026B280EC0E@yahoo.com> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.50 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; R_DKIM_ALLOW(-0.20)[puchar.net:s=default]; R_SPF_ALLOW(-0.20)[+mx:c]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_CC(0.00)[yahoo.com,longcount.org,freebsd.org]; DKIM_TRACE(0.00)[puchar.net:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[194.1.144.90:from]; TO_DN_ALL(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[puchar.net]; HAS_XAW(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RdR4P5GThz4Dc4 >>> All >>> Why not take scp out of the picture and try iperf? Why , we could be looking >> at rss by default in Linux . > > Actually I did the iperf3 test as well and posted results a couple days ago. > Pasting here: > > FreeBSD iperf3 to localhost, single stream: 30.9 Gb/s > Linux iperf3 to localhost, single stream: 48.8 Gb/s > Yes linux have lower system call latency, and somewhat more optimized networking for special cases (like localhost). For multiple processes using network results will be different. But probably still be somewhat slower than linux. Just not 40% slower i think From nobody Sat Sep 2 20:20:22 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdR9D1PQ7z4s39C for ; Sat, 2 Sep 2023 20:20:32 +0000 (UTC) (envelope-from wojtek@puchar.net) Received: from puchar.net (puchar.net [194.1.144.90]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdR9B36R2z4Fph for ; Sat, 2 Sep 2023 20:20:30 +0000 (UTC) (envelope-from wojtek@puchar.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=puchar.net header.s=default header.b=ldf+7lWR; spf=pass (mx1.freebsd.org: domain of wojtek@puchar.net designates 194.1.144.90 as permitted sender) smtp.mailfrom=wojtek@puchar.net; dmarc=none Received: Received: from 127.0.0.1 (localhost [127.0.0.1]) by puchar.net (8.15.2/8.17.1) with ESMTPS id 382KKNse019893 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 2 Sep 2023 22:20:24 +0200 (CEST) (envelope-from wojtek@puchar.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=puchar.net; s=default; t=1693686025; bh=py5xFXmJ4NKE73EFn9ggcoZmM36vGxOZtzrprBp/O0U=; h=Date:From:To:cc:Subject:In-Reply-To:References; b=ldf+7lWRMi8fxiZ9SvzlS3Qmfk1K74oOz30G48IlWKwmGnSmlbV18b7AeS4q9eKqz 5eERsImghcd0zEpHXw2Ir1uzvHW2tzx5kbuCFcMS/GdG5duAcs4hWt7rlBeKO/eNcI qPkT9/6H4jE507v1ir4ioQDUNuSThh3aP9vAihho= Received: from wojtek.intra (localhost [127.0.0.1]) by wojtek.intra (8.16.1/8.16.1) with ESMTP id 382KKMCn050914; Sat, 2 Sep 2023 22:20:22 +0200 (CEST) (envelope-from wojtek@puchar.net) Received: from localhost (wojtek@localhost) by wojtek.intra (8.16.1/8.16.1/Submit) with ESMTP id 382KKMDF050911; Sat, 2 Sep 2023 22:20:22 +0200 (CEST) (envelope-from wojtek@puchar.net) X-Authentication-Warning: wojtek.intra: wojtek owned process doing -bs Date: Sat, 2 Sep 2023 22:20:22 +0200 (CEST) From: Wojciech Puchar To: void cc: freebsd-hackers@freebsd.org Subject: Re: can sftp be made multi-threaded? In-Reply-To: Message-ID: References: List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.49 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_LONG(-1.00)[-0.996]; R_SPF_ALLOW(-0.20)[+mx:c]; R_DKIM_ALLOW(-0.20)[puchar.net:s=default]; MIME_GOOD(-0.10)[text/plain]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_TO(0.00)[f-m.fm]; ASN(0.00)[asn:43476, ipnet:194.1.144.0/24, country:PL]; MIME_TRACE(0.00)[0:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_TWO(0.00)[2]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[puchar.net:+]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; DMARC_NA(0.00)[puchar.net]; HAS_XAW(0.00)[]; RCVD_TLS_LAST(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[194.1.144.90:from] X-Rspamd-Queue-Id: 4RdR9B36R2z4Fph > > Can sftp be made multi-threaded? Would rsync-over-ssh accomplish the same > thing? > > The issue I'm having is with large backups taking ages to transfer > even with both source and destination being on gigabit fibre. simply divide it to smaller parts and run them in parallel > > It looks like sftp is single-threaded and the bandwidth per thread is > something like 5-50 Mbps and it's nowhere near saturating the bandwidth. Mb or MB? From nobody Sat Sep 2 23:22:41 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RdWCT3cVBz4s0hT for ; Sat, 2 Sep 2023 23:22:45 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-qk1-x734.google.com (mail-qk1-x734.google.com [IPv6:2607:f8b0:4864:20::734]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RdWCS3DDsz4hT7 for ; Sat, 2 Sep 2023 23:22:44 +0000 (UTC) (envelope-from bacon4000@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=AktSRcWE; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::734 as permitted sender) smtp.mailfrom=bacon4000@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-qk1-x734.google.com with SMTP id af79cd13be357-76ef935abaeso16390685a.2 for ; Sat, 02 Sep 2023 16:22:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1693696963; x=1694301763; darn=freebsd.org; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=PDVL0unmWOMKjhp1Y5zU4fjX3zjxQqRI8KJgLMof4js=; b=AktSRcWEg6qBduzVGI0rEbO6x94cpHgequg2eHWHS2tTIdjm/iLjpo9bKHIsaWL4Jj wKnfJHCKmpSxqGTbAcAcETf9AjfSsSODptx8UMCjAT8KdPDgVPUuLzcnLIjP6ElsfOWX C3d59AcO+MmiTUDcromWRQ4YCgMSSvbIvd0x24ivCX1og6Ea7hydOenqg2HN665aP11b vCk8a/SQw+9I6VwwXRILbT3Eetfsg+DnPQh0r3L/t1uLdm4u41LCs5gi4ZiBkHJZBDDy RnuawmzXu2Br9zaHQ5SLJMJ/giXUnn2qcvLnzX7wHEh/Wouf41lcJyLuAyVPK49qEUGN UrGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1693696963; x=1694301763; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=PDVL0unmWOMKjhp1Y5zU4fjX3zjxQqRI8KJgLMof4js=; b=YNDtE/O1mZkppzQAXGGRA5HQUyBCQSsSOS7Nz6O3WRwf5x2FoMwa8qoyLoDZaubITW xhZ+VQDJtfI91RTyrqQpOvzOhqee2FNkM/P1bwl6/5ltau4H9fKpIXtYxNriagJXplWs wNs78a5W59tuZk3CBu5ea5UBARcytpu80/ZBrUkHq3P4zhR9I542OW8V9CH3QNpnKgVd av2M5rOvgl5jkvfzvA9+LNfjGWIg+ae3ctQS4E1pkoanl0OvETy7MgWZBejxOtDygNBb PfECWL8L2ShPvPZKlsJ+7FSv/qYfAldFN3em43m9cwCNgu/oyOKJxTT1ZWNwE0yWWmLJ qOAQ== X-Gm-Message-State: AOJu0YwoO8OgL/R3VGWdPlVYzprNU1+93ctbj2vNXMAvkWClHTXlrxRT 7UobXhduQ/5+ki358C93c4ZvItH6W+E= X-Google-Smtp-Source: AGHT+IGu5z7eCPylv3Rx3jYqz3q6ckQNrvsqnA1bca0qKPPeUm9RuuyHyGXm4iuHbFVypN1ktzghuA== X-Received: by 2002:a05:620a:290b:b0:76f:1eac:e72d with SMTP id m11-20020a05620a290b00b0076f1eace72dmr8905402qkp.38.1693696963106; Sat, 02 Sep 2023 16:22:43 -0700 (PDT) Received: from ?IPV6:2603:6000:a401:3a00:6e88:14ff:fea7:590c? (2603-6000-a401-3a00-6e88-14ff-fea7-590c.res6.spectrum.com. [2603:6000:a401:3a00:6e88:14ff:fea7:590c]) by smtp.gmail.com with ESMTPSA id m15-20020ae9e00f000000b0076f124abe4dsm2367156qkk.77.2023.09.02.16.22.42 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sat, 02 Sep 2023 16:22:42 -0700 (PDT) Message-ID: Date: Sat, 2 Sep 2023 18:22:41 -0500 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.15.0 Subject: Re: can sftp be made multi-threaded? Content-Language: en-US To: freebsd-hackers@freebsd.org References: From: Jason Bacon In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Bar: -- X-Spamd-Result: default: False [-3.00 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::734:from]; ARC_NA(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; DKIM_TRACE(0.00)[gmail.com:+]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2] X-Rspamd-Queue-Id: 4RdWCS3DDsz4hT7 On 9/2/23 15:20, Wojciech Puchar wrote: >> >> Can sftp be made multi-threaded? Would rsync-over-ssh accomplish the >> same thing? >> >> The issue I'm having is with large backups taking ages to transfer >> even with both source and destination being on gigabit fibre. > > simply divide it to smaller parts and run them in parallel How many network hops between source and destination? "traceroute" will provide some idea. If this is poor bandwidth utilization due to overhead in a long network route, parallelizing could work well. If the backup server is on the same local network, or only a few hops away, something else needs probably to be fixed. -- Life is a game. Play hard. Play fair. Have fun.