From nobody Fri Aug 11 16:53:44 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RMqd14Zj8z4m8p4 for ; Fri, 11 Aug 2023 16:53:57 +0000 (UTC) (envelope-from joesuf4@gmail.com) Received: from mail-wm1-x335.google.com (mail-wm1-x335.google.com [IPv6:2a00:1450:4864:20::335]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RMqd04LZxz3D0K for ; Fri, 11 Aug 2023 16:53:56 +0000 (UTC) (envelope-from joesuf4@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20221208 header.b=suOyNSpQ; spf=pass (mx1.freebsd.org: domain of joesuf4@gmail.com designates 2a00:1450:4864:20::335 as permitted sender) smtp.mailfrom=joesuf4@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-3fe8a158fcbso5603015e9.2 for ; Fri, 11 Aug 2023 09:53:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1691772835; x=1692377635; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=aRR9+UzdGQ/bTD/iuZx3730jjbbieaXHuGn6F7fJC8s=; b=suOyNSpQ9z/MU7JNCtGzihbDif67situWFHcFzy9uQYy4i4Nb3A/dorv/8R0bUZSiJ 2p5b5mipnya+9SqYLtI2jxWJyRitxctuZfJnvK2h7F1PYa9Biv9X7/zXqEKF0rFNWBPn /QRccaT6lxDGvN9IDa7ZleUZGK+2eIC9nX6zLLFB+hYJ7zQ270iOSfruQaReNDX0IBhv 1GIp83A0trHRVzRyppKWrFUQ2QPh1nmn/Gifz/CKLjbZtOxf1jJrt+ODeT7l+LDWXgKN gNwb8iq6BXHbQYBsYrNqqeYXRU0zjy82AOD4PKbe/rJlaYCQp47WEc4dDNantPfDwsBm OEVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1691772835; x=1692377635; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=aRR9+UzdGQ/bTD/iuZx3730jjbbieaXHuGn6F7fJC8s=; b=STjILU0r9+gUVTdiErAGnnnmumnbeAZz1/73gZk1bPpt8O2Nx0RlA5PFCLATeGcP2+ 76DMPKOZIb3fvnxxlHv29Iis+JKJ1c/Dagj94QDc6ac4GhZ5USjoo59FXUDv5gKNFE7Z djuYuR7HuysYfdxIULrtxrL0XbPsaeosSU/Acyo+2d1WY8WImR5K3O9dH5QtN5Z0Iq0q b6dwOPHLRAMytc0sPi63S2rcbYTwTHh0cRwUaA/bU1spxUzygbCOPDC3JUYM/rC5oOQ8 s2xhf3WGCZIvsvPmQpCCZa5FrzOPj35MIAxTaE0TDzJoZMnKEv8iTyavxwP7XGu/9LWq KwVA== X-Gm-Message-State: AOJu0Yw/p3i0yu3bQya23N+5AciOkOoZD+drH8T5Y5oyRvQ1710ZyjCO plUuElrLwz1I2BDj2kZxf6T3f4bkTgsZLyHYNzZ3jKZw X-Google-Smtp-Source: AGHT+IFfzBzythsmXAO2dFpiNJl4NVehgmC1yBtzF9sgoj4PDRN58794HexOyozHFrojmmGy0FOM/7uBO/H0xFhojIM= X-Received: by 2002:adf:db0a:0:b0:314:2e95:1ec9 with SMTP id s10-20020adfdb0a000000b003142e951ec9mr1884434wri.10.1691772834760; Fri, 11 Aug 2023 09:53:54 -0700 (PDT) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <202308111121.37BBLX0J064263@donotpassgo.dyslexicfish.net> In-Reply-To: From: Joe Schaefer Date: Fri, 11 Aug 2023 12:53:44 -0400 Message-ID: Subject: Re: can sftp be made multi-threaded? To: freebsd-hackers@freebsd.org Content-Type: multipart/alternative; boundary="000000000000db4d3d0602a88e19" X-Spamd-Result: default: False [-2.90 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; NEURAL_HAM_SHORT(-0.97)[-0.971]; NEURAL_HAM_LONG(-0.97)[-0.969]; NEURAL_HAM_MEDIUM(-0.96)[-0.962]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20221208]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_ONE(0.00)[1]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ARC_NA(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::335:from]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; TO_DN_NONE(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Spamd-Bar: -- X-Rspamd-Queue-Id: 4RMqd04LZxz3D0K --000000000000db4d3d0602a88e19 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Put the split files on tmpfs for full effect On Fri, Aug 11, 2023 at 12:51 PM Joe Schaefer wrote: > % split -n $(nproc) foo.pdf; ls x* | xargs -P 0 -J % scp % > user@bar.example.com:%; rm x*; ssh user@bar.example.com sh -c =E2=80=9Cc= at x* > > foo.pdf; rm x*=E2=80=9D > > On Fri, Aug 11, 2023 at 12:27 PM Joe Schaefer wrote: > >> If it=E2=80=99s just a single file, split it into chunks. >> >> On Fri, Aug 11, 2023 at 12:25 PM Joe Schaefer wrote: >> >>> Why don=E2=80=99t you just use xargs -P until you=E2=80=99ve exhausted = your CPU capacity? >>> >>> On Fri, Aug 11, 2023 at 10:10 AM void wrote: >>> >>>> On Fri, Aug 11, 2023 at 12:21:33PM +0100, Jamie Landeg-Jones wrote: >>>> >>>> >rsync just spawns an ssh command, so would probably behave similarly. >>>> >>>> I'm hoping that rsync will spawn many ssh. Need to look at max session= s >>>> on both ends of the connection. >>>> >>>> Since encountering the described problem, the person at the other >>>> end is away for the week so have not been able to test thoroughly. >>>> What I have been able to test shows that there is spiky latency >>>> in the connection, as well as slow speed, single-threaded. >>>> >>>> >Another thing, scp transfers from my test Rpi2 are much slower than >>>> the network >>>> >can handle due to the CPU use, which hits 100% on one cpu whilst it's >>>> running. >>>> >So, check that CPU isn't the bottleneck too. >>>> >>>> Yup. That won't be happening here. Dual xenon with 56 cores at remote >>>> end and same (but with 32 cores) at this end >>>> >>>> >As for the speed, I just tested sftp to transfer a file of random >>>> data, 2 GB in >>>> >size from one FreeBSD box in London to another in France: >>>> > >>>> >The final result was: >>>> > >>>> > 100% 2000MB 43.5MB/s 00:46 (Note, that's MegaBYTES/s) >>>> >>>> I ran a similar test. >>>> Sending system is on synchronous gigabit fibre on US east coast, >>>> receiving system is near London on 110/21 fibre (so, gigabit in the >>>> sending >>>> direction): >>>> >>>> 100% 2000MB 7.2MB/s 04:36 >>>> >>>> using rsync -azP : 2,097,152,000 100% 6.81MB/s 0:04:53 (xfr#1, >>>> to-chk=3D0/1) >>>> >>>> the speed fluctulates a lot. Both systems are quiet in a network and O= S >>>> sense >>>> for the duration of the test. >>>> >>>> >The London box is pretty old, and is a virtual host scheduled to be >>>> decomissioned. >>>> >It is running an old openssl 1.X, openssh 8.8 and is a single core >>>> 2.4Ghz amd64 box. >>>> > >>>> >The France box is a 4 core bare metal 3.1Ghz and64 running openssh 9.= 2 >>>> and openssl 1.1.1 >>>> >>>> both ends here are running very recent -current, so ssl/ssh is >>>> OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023 >>>> >>>> >Anything more I can tell you that may help? >>>> >>>> Thanks very much for your input. I'm certain it's not a freebsd proble= m. >>>> >>>> -- >>>> >>>> --000000000000db4d3d0602a88e19 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Put the split files on tmpfs for full effect=C2=A0
<= div>
On= Fri, Aug 11, 2023 at 12:51 PM Joe Schaefer <joesuf4@gmail.com> wrote:
% split -n $(nproc) foo.pdf; ls x* | xargs -P 0 -= J % scp % =C2=A0user@bar.example.com:%; rm x*; ssh user@bar.example.com sh -c =E2=80=9Cc= at x* > foo.pdf; rm x*=E2=80=9D

On Fri, Aug 11, 2023 at 12:27 PM Joe= Schaefer <joesuf= 4@gmail.com> wrote:
If i= t=E2=80=99s just a single file, split it into chunks.

On Fri, Aug 11, 2= 023 at 12:25 PM Joe Schaefer <joesuf4@gmail.com> wrote:
Why don=E2=80=99t you just use xargs -P until you=E2=80=99ve= exhausted your CPU capacity?

On Fri, Aug 11, 2= 023 at 10:10 AM void <v= oid@f-m.fm> wrote:
On Fri, Aug 11, 2023 a= t 12:21:33PM +0100, Jamie Landeg-Jones wrote:

>rsync just spawns an ssh command, so would probably behave similarly.
I'm hoping that rsync will spawn many ssh. Need to look at max sessions=
on both ends of the connection.

Since encountering the described problem, the person at the other
end is away for the week so have not been able to test thoroughly.
What I have been able to test shows that there is spiky latency
in the connection, as well as slow speed, single-threaded.

>Another thing, scp transfers from my test Rpi2 are much slower than the= network
>can handle due to the CPU use, which hits 100% on one cpu whilst it'= ;s running.
>So, check that CPU isn't the bottleneck too.

Yup. That won't be happening here. Dual xenon with 56 cores at remote end and same (but with 32 cores) at this end

>As for the speed, I just tested sftp to transfer a file of random data,= 2 GB in
>size from one FreeBSD box in London to another in France:
>
>The final result was:
>
> 100% 2000MB=C2=A0 43.5MB/s=C2=A0 =C2=A000:46=C2=A0 (Note, that's M= egaBYTES/s)

I ran a similar test.
Sending system is on synchronous gigabit fibre on US east coast,
receiving system is near London on 110/21 fibre (so, gigabit in the sending=
direction):

100% 2000MB=C2=A0 =C2=A07.2MB/s=C2=A0 =C2=A004:36

using rsync -azP : 2,097,152,000 100%=C2=A0 =C2=A0 6.81MB/s=C2=A0 =C2=A0 0:= 04:53 (xfr#1, to-chk=3D0/1)

the speed fluctulates a lot. Both systems are quiet in a network and OS sen= se
for the duration of the test.

>The London box is pretty old, and is a virtual host scheduled to be dec= omissioned.
>It is running an old openssl 1.X, openssh 8.8 and is a single core 2.4G= hz amd64 box.
>
>The France box is a 4 core bare metal 3.1Ghz and64 running openssh 9.2 = and openssl 1.1.1

both ends here are running very recent -current, so ssl/ssh is
OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023

>Anything more I can tell you that may help?

Thanks very much for your input. I'm certain it's not a freebsd pro= blem.

--

--000000000000db4d3d0602a88e19--