Date: Fri, 11 Aug 2023 12:53:44 -0400 From: Joe Schaefer <joesuf4@gmail.com> To: freebsd-hackers@freebsd.org Subject: Re: can sftp be made multi-threaded? Message-ID: <CAOzHqcJVKf=Ds%2B=C7UrC%2B1dp%2Brm=jQ6mK%2BGMCkfW=GEcsrFe_w@mail.gmail.com> In-Reply-To: <CAOzHqcL=ngJYpQ165uUPsrnm2vaBGo%2B__ACJa9mZXx5dfjkkNQ@mail.gmail.com> References: <ZNOJUfIHGtY9lLpg@int21h> <202308111121.37BBLX0J064263@donotpassgo.dyslexicfish.net> <ZNZBVf1CuI35PyBf@int21h> <CAOzHqcK5DUj0-b%2BP_sqPyC7NknHKaS1_fNcnTWKjde-SYn9QSA@mail.gmail.com> <CAOzHqc%2BGYsS=gCGfHC%2BxLK42hnK95tea_7qiKgUDJO=bF3Ys9A@mail.gmail.com> <CAOzHqcL=ngJYpQ165uUPsrnm2vaBGo%2B__ACJa9mZXx5dfjkkNQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Put the split files on tmpfs for full effect On Fri, Aug 11, 2023 at 12:51 PM Joe Schaefer <joesuf4@gmail.com> wrote: > % split -n $(nproc) foo.pdf; ls x* | xargs -P 0 -J % scp % > user@bar.example.com:%; rm x*; ssh user@bar.example.com sh -c “cat x* > > foo.pdf; rm x*” > > On Fri, Aug 11, 2023 at 12:27 PM Joe Schaefer <joesuf4@gmail.com> wrote: > >> If it’s just a single file, split it into chunks. >> >> On Fri, Aug 11, 2023 at 12:25 PM Joe Schaefer <joesuf4@gmail.com> wrote: >> >>> Why don’t you just use xargs -P until you’ve exhausted your CPU capacity? >>> >>> On Fri, Aug 11, 2023 at 10:10 AM void <void@f-m.fm> wrote: >>> >>>> On Fri, Aug 11, 2023 at 12:21:33PM +0100, Jamie Landeg-Jones wrote: >>>> >>>> >rsync just spawns an ssh command, so would probably behave similarly. >>>> >>>> I'm hoping that rsync will spawn many ssh. Need to look at max sessions >>>> on both ends of the connection. >>>> >>>> Since encountering the described problem, the person at the other >>>> end is away for the week so have not been able to test thoroughly. >>>> What I have been able to test shows that there is spiky latency >>>> in the connection, as well as slow speed, single-threaded. >>>> >>>> >Another thing, scp transfers from my test Rpi2 are much slower than >>>> the network >>>> >can handle due to the CPU use, which hits 100% on one cpu whilst it's >>>> running. >>>> >So, check that CPU isn't the bottleneck too. >>>> >>>> Yup. That won't be happening here. Dual xenon with 56 cores at remote >>>> end and same (but with 32 cores) at this end >>>> >>>> >As for the speed, I just tested sftp to transfer a file of random >>>> data, 2 GB in >>>> >size from one FreeBSD box in London to another in France: >>>> > >>>> >The final result was: >>>> > >>>> > 100% 2000MB 43.5MB/s 00:46 (Note, that's MegaBYTES/s) >>>> >>>> I ran a similar test. >>>> Sending system is on synchronous gigabit fibre on US east coast, >>>> receiving system is near London on 110/21 fibre (so, gigabit in the >>>> sending >>>> direction): >>>> >>>> 100% 2000MB 7.2MB/s 04:36 >>>> >>>> using rsync -azP : 2,097,152,000 100% 6.81MB/s 0:04:53 (xfr#1, >>>> to-chk=0/1) >>>> >>>> the speed fluctulates a lot. Both systems are quiet in a network and OS >>>> sense >>>> for the duration of the test. >>>> >>>> >The London box is pretty old, and is a virtual host scheduled to be >>>> decomissioned. >>>> >It is running an old openssl 1.X, openssh 8.8 and is a single core >>>> 2.4Ghz amd64 box. >>>> > >>>> >The France box is a 4 core bare metal 3.1Ghz and64 running openssh 9.2 >>>> and openssl 1.1.1 >>>> >>>> both ends here are running very recent -current, so ssl/ssh is >>>> OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023 >>>> >>>> >Anything more I can tell you that may help? >>>> >>>> Thanks very much for your input. I'm certain it's not a freebsd problem. >>>> >>>> -- >>>> >>>> [-- Attachment #2 --] <div dir="auto">Put the split files on tmpfs for full effect </div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 12:51 PM Joe Schaefer <<a href="mailto:joesuf4@gmail.com">joesuf4@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto">% split -n $(nproc) foo.pdf; ls x* | xargs -P 0 -J % scp % user@bar.example.com:%; rm x*; ssh <a href="mailto:user@bar.example.com" target="_blank">user@bar.example.com</a> sh -c “cat x* > foo.pdf; rm x*”</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 12:27 PM Joe Schaefer <<a href="mailto:joesuf4@gmail.com" target="_blank">joesuf4@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="auto">If it’s just a single file, split it into chunks.</div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 12:25 PM Joe Schaefer <<a href="mailto:joesuf4@gmail.com" target="_blank">joesuf4@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)"><div dir="auto">Why don’t you just use xargs -P until you’ve exhausted your CPU capacity?</div><div dir="auto"><br></div><div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 11, 2023 at 10:10 AM void <<a href="mailto:void@f-m.fm" target="_blank">void@f-m.fm</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">On Fri, Aug 11, 2023 at 12:21:33PM +0100, Jamie Landeg-Jones wrote:<br> <br> >rsync just spawns an ssh command, so would probably behave similarly.<br> <br> I'm hoping that rsync will spawn many ssh. Need to look at max sessions <br> on both ends of the connection.<br> <br> Since encountering the described problem, the person at the other<br> end is away for the week so have not been able to test thoroughly.<br> What I have been able to test shows that there is spiky latency<br> in the connection, as well as slow speed, single-threaded.<br> <br> >Another thing, scp transfers from my test Rpi2 are much slower than the network<br> >can handle due to the CPU use, which hits 100% on one cpu whilst it's running.<br> >So, check that CPU isn't the bottleneck too.<br> <br> Yup. That won't be happening here. Dual xenon with 56 cores at remote<br> end and same (but with 32 cores) at this end<br> <br> >As for the speed, I just tested sftp to transfer a file of random data, 2 GB in<br> >size from one FreeBSD box in London to another in France:<br> ><br> >The final result was:<br> ><br> > 100% 2000MB 43.5MB/s 00:46 (Note, that's MegaBYTES/s)<br> <br> I ran a similar test.<br> Sending system is on synchronous gigabit fibre on US east coast,<br> receiving system is near London on 110/21 fibre (so, gigabit in the sending<br> direction):<br> <br> 100% 2000MB 7.2MB/s 04:36<br> <br> using rsync -azP : 2,097,152,000 100% 6.81MB/s 0:04:53 (xfr#1, to-chk=0/1)<br> <br> the speed fluctulates a lot. Both systems are quiet in a network and OS sense<br> for the duration of the test.<br> <br> >The London box is pretty old, and is a virtual host scheduled to be decomissioned.<br> >It is running an old openssl 1.X, openssh 8.8 and is a single core 2.4Ghz amd64 box.<br> ><br> >The France box is a 4 core bare metal 3.1Ghz and64 running openssh 9.2 and openssl 1.1.1<br> <br> both ends here are running very recent -current, so ssl/ssh is <br> OpenSSH_9.3p1, OpenSSL 3.0.9 30 May 2023<br> <br> >Anything more I can tell you that may help?<br> <br> Thanks very much for your input. I'm certain it's not a freebsd problem.<br> <br> -- <br> <br> </blockquote></div></div> </blockquote></div></div> </blockquote></div></div> </blockquote></div></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOzHqcJVKf=Ds%2B=C7UrC%2B1dp%2Brm=jQ6mK%2BGMCkfW=GEcsrFe_w>