From nobody Fri Aug 11 14:10:29 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RMm0W2y3fz4TrcT for ; Fri, 11 Aug 2023 14:10:35 +0000 (UTC) (envelope-from void@f-m.fm) Received: from wout4-smtp.messagingengine.com (wout4-smtp.messagingengine.com [64.147.123.20]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4RMm0V35xyz4QQT for ; Fri, 11 Aug 2023 14:10:34 +0000 (UTC) (envelope-from void@f-m.fm) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=f-m.fm header.s=fm3 header.b=Qd1Xdw+q; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=EDExSBba; spf=pass (mx1.freebsd.org: domain of void@f-m.fm designates 64.147.123.20 as permitted sender) smtp.mailfrom=void@f-m.fm; dmarc=pass (policy=none) header.from=f-m.fm Received: from compute5.internal (compute5.nyi.internal [10.202.2.45]) by mailout.west.internal (Postfix) with ESMTP id 679EA3200939 for ; Fri, 11 Aug 2023 10:10:32 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute5.internal (MEProxy); Fri, 11 Aug 2023 10:10:32 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=f-m.fm; h=cc :content-type:content-type:date:date:from:from:in-reply-to :in-reply-to:message-id:mime-version:references:reply-to:sender :subject:subject:to:to; s=fm3; t=1691763031; x=1691849431; bh=1N XmXoXAIDYfM1WRFtzhPvJwl8X+21Zz7tlnYrl73M8=; b=Qd1Xdw+q6yesuPZP0u 5pXvVrzxi2ZxgnNsByMzmdF1XaJQHOJRSFh3mUQWjYduwpuiADs7PaesdHRjXaEH XfZxkFt91XbNNXYyEWmNkGZSZlAXKgtIuJ5WHwm7QPIpDlTbxokDrOyi3yY6X4JL MM/uSLDfhgQX88fr0bnJ9EMO3t0KOxMpm53jLKTBu5rCJSJp7CJQfjPXSJjtuK+A glNr1DsFCD2Og3/sCxcqIfcMxnRmJRlAstuurgkZi0UDOZ+khNnVpftNqIz3gxAD /1c+oYxqFlIbBx9KBoK+iXlMNClsXEXYMKWTO81R3ObJg9bpgsVRROEhN6Yil9Ds RR1A== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:content-type:date:date :feedback-id:feedback-id:from:from:in-reply-to:in-reply-to :message-id:mime-version:references:reply-to:sender:subject :subject:to:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm3; t=1691763031; x=1691849431; bh=1NXmXoXAIDYfM 1WRFtzhPvJwl8X+21Zz7tlnYrl73M8=; b=EDExSBbaJqhNWn0U6ILdPfcy0A/93 8z05n0f2pCOHpvnb+Ko8cFuf4xF8H/gVZVaEe2whpp/TK5gzgPKYeqs8EGNvfCqy 6VBYBOckjQbQHc3uUGoGbXmiMDyD/qdBeN9/rK2j8G3ATk9YJVm32W01jqmc5nEJ EilVhTiJP+SxnwcJEIzM38Uas8vGGs5EV4EVqcsItPLGf6/FZ16yo2N3SM4+VhXK XYfKETH6lKOtj5gzLx9fcXpNzx0Sivhbe+yk2uKwl3u4+m7vxsC/pmaKHQvIm79X gplePwdmWE8GyqfIN58Udrrw0pxact3qXywX3q7VShN4DZnPkmJ7mAmUQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedviedrleekgdejudcutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehttdertd dttddvnecuhfhrohhmpehvohhiugcuoehvohhiugesfhdqmhdrfhhmqeenucggtffrrght thgvrhhnpeekleduvdelhfeileefgffghfffkedtheellefgudfgvdegkeejjedutdehhe fgueenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehv ohhiugesfhdqmhdrfhhm X-ME-Proxy: Feedback-ID: i2541463c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Fri, 11 Aug 2023 10:10:31 -0400 (EDT) Date: Fri, 11 Aug 2023 15:10:29 +0100 From: void To: freebsd-hackers@freebsd.org Subject: Re: can sftp be made multi-threaded? Message-ID: Mail-Followup-To: freebsd-hackers@freebsd.org References: <202308111121.37BBLX0J064263@donotpassgo.dyslexicfish.net> List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline In-Reply-To: <202308111121.37BBLX0J064263@donotpassgo.dyslexicfish.net> X-Spamd-Result: default: False [-3.36 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; DWL_DNSWL_LOW(-1.00)[messagingengine.com:dkim]; NEURAL_HAM_MEDIUM(-1.00)[-0.999]; NEURAL_HAM_LONG(-0.98)[-0.981]; MID_RHS_NOT_FQDN(0.50)[]; DMARC_POLICY_ALLOW(-0.50)[f-m.fm,none]; RWL_MAILSPIKE_EXCELLENT(-0.40)[64.147.123.20:from]; NEURAL_HAM_SHORT(-0.38)[-0.382]; R_DKIM_ALLOW(-0.20)[f-m.fm:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.20]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.20:from]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[f-m.fm]; MIME_TRACE(0.00)[0:+]; DKIM_TRACE(0.00)[f-m.fm:+,messagingengine.com:+]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[f-m.fm]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Spamd-Bar: --- X-Rspamd-Queue-Id: 4RMm0V35xyz4QQT On Fri, Aug 11, 2023 at 12:21:33PM +0100, Jamie Landeg-Jones wrote: >rsync just spawns an ssh command, so would probably behave similarly. I'm hoping that rsync will spawn many ssh. Need to look at max sessions on both ends of the connection. Since encountering the described problem, the person at the other end is away for the week so have not been able to test thoroughly. What I have been able to test shows that there is spiky latency in the connection, as well as slow speed, single-threaded. >Another thing, scp transfers from my test Rpi2 are much slower than the network >can handle due to the CPU use, which hits 100% on one cpu whilst it's running. >So, check that CPU isn't the bottleneck too. Yup. That won't be happening here. Dual xenon with 56 cores at remote end and same (but with 32 cores) at this end >As for the speed, I just tested sftp to transfer a file of random data, 2 GB in >size from one FreeBSD box in London to another in France: > >The final result was: > > 100% 2000MB 43.5MB/s 00:46 (Note, that's MegaBYTES/s) I ran a similar test. Sending system is on synchronous gigabit fibre on US east coast, receiving system is near London on 110/21 fibre (so, gigabit in the sending direction): 100% 2000MB 7.2MB/s 04:36 using rsync -azP : 2,097,152,000 100% 6.81MB/s 0:04:53 (xfr#1, to-chk=0/1) the speed fluctulates a lot. Both systems are quiet in a network and OS sense for the duration of the test. >The London box is pretty old, and is a virtual host scheduled to be decomissioned. >It is running an old openssl 1.X, openssh 8.8 and is a single core 2.4Ghz amd64 box. > >The France box is a 4 core bare metal 3.1Ghz and64 running openssh 9.2 and openssl 1.1.1 both ends here are running very recent -current, so ssl/ssh is OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023 >Anything more I can tell you that may help? Thanks very much for your input. I'm certain it's not a freebsd problem. --