Date: Wed, 22 Feb 2023 17:22:32 -0500 From: Charles Sprickman <spork@bway.net> To: Freddie Cash <fjwcash@gmail.com> Cc: Miroslav Lachman <000.fbsd@quip.cz>, mike tancsa <mike@sentex.net>, Alan Somers <asomers@freebsd.org>, freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: speeding up zfs send | recv (update) Message-ID: <0171E506-3899-42B2-B7DC-4145BAA595D7@bway.net> In-Reply-To: <CAOjFWZ7k7ANwcGyNCoYMg%2BLUBzAz2VyNfxQo5rKcrYj8XFgG3Q@mail.gmail.com> References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <CAOtMX2gifUmgqwSKpRGcfzCm_=BX_szNF1AF8WTMfAmbrJ5UWA@mail.gmail.com> <f6ea3387-faf8-4c63-d1e7-906fa397b00b@sentex.net> <a38578c6-b633-249d-90f0-0652377d76c0@quip.cz> <c229a502-fb76-ec6a-a56b-934d3b56e474@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <CAOjFWZ7k7ANwcGyNCoYMg%2BLUBzAz2VyNfxQo5rKcrYj8XFgG3Q@mail.gmail.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] > On Feb 22, 2023, at 4:43 PM, Freddie Cash <fjwcash@gmail.com> wrote: > > [Sorry for top part, GMail sucks for replies.] > > If this is a LAN or private WAN where you trust the network, piping the send stream through netcat will remove ssh from the equation. > > That's what we switched to using once it became almost impossible to get the "none" cipher working with ssh on FreeBSD. > > We use ssh to connect to the remote server and enable a netcat listener on port X, then pipe the send through netcat to the remote system on port X. That way it's logged and uses ssh for authentication. > > We easily saturate gigabit links between our ZFS systems using netcat. This is kind of tangential, but is there any ssh client/server that is able to make use of multiple CPU cores or is that just not easily possible? The first set of hosts I worked with that had 10Gb/s internal network ports kind of showed me how much of a bottleneck trying to encrypt with a single core is. If using netcat or similar to avoid the ssh overhead, can IPSEC or a VPN option (wireguard?) be a bit of a workaround? Do any VPN implementations on FreeBSD put multiple cores to use? Thanks, Charles > > > > Cheers, > Freddie > > Typos due to smartphone keyboard. > > On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz <mailto:000.fbsd@quip.cz>> wrote: > On 22/02/2023 22:08, mike tancsa wrote: > > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: > >> Interresting numbers. I think I am the only one who get best speed > >> with chacha20-poly1305@openssh.com <mailto:chacha20-poly1305@openssh.com> > >> > >> > >> It seems the speed of SSH is limited by single core performance which > >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). > >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. > > > > The CPU I have has > > aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard > > > > which probably helps. > > That explains it > aesni0: No AES or SHA support. > > >> I know there were some HPN patches to ssh, beside that is there any > >> option I can try to use less CPU? > >> > >> I will play with cpuset to pin ssh on one core and everything else on > >> the other core. > > > > It looks like you are running into a CPU bottleneck TBH > > Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but > without some tweaks on ssh I will not gain more speed :( > > Thank you for your help! > > Miroslav Lachman > > [-- Attachment #2 --] <html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 22, 2023, at 4:43 PM, Freddie Cash <<a href="mailto:fjwcash@gmail.com" class="">fjwcash@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="auto" class="">[Sorry for top part, GMail sucks for replies.]<div dir="auto" class=""><br class=""></div><div dir="auto" class="">If this is a LAN or private WAN where you trust the network, piping the send stream through netcat will remove ssh from the equation.<div dir="auto" class=""><br class=""></div><div dir="auto" class="">That's what we switched to using once it became almost impossible to get the "none" cipher working with ssh on FreeBSD.</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">We use ssh to connect to the remote server and enable a netcat listener on port X, then pipe the send through netcat to the remote system on port X. That way it's logged and uses ssh for authentication.</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">We easily saturate gigabit links between our ZFS systems using netcat.<br class=""></div></div></div></div></blockquote><div><br class=""></div><div>This is kind of tangential, but is there any ssh client/server that is able to make use of multiple CPU cores or is that just not easily possible?</div><div><br class=""></div><div>The first set of hosts I worked with that had 10Gb/s internal network ports kind of showed me how much of a bottleneck trying to encrypt with a single core is.</div><div><br class=""></div><div>If using netcat or similar to avoid the ssh overhead, can IPSEC or a VPN option (wireguard?) be a bit of a workaround? Do any VPN implementations on FreeBSD put multiple cores to use?</div><div><br class=""></div><div>Thanks,</div><div><br class=""></div><div>Charles</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="auto" class=""><div dir="auto" class=""><div dir="auto" class=""><br class=""></div><div dir="auto" class=""><br class=""><br class=""><div data-smartmail="gmail_signature" dir="auto" class="">Cheers,<br class="">Freddie<br class=""><br class="">Typos due to smartphone keyboard.</div></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <<a href="mailto:000.fbsd@quip.cz" class="">000.fbsd@quip.cz</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 22/02/2023 22:08, mike tancsa wrote:<br class=""> > On 2/22/2023 4:03 PM, Miroslav Lachman wrote:<br class=""> >> Interresting numbers. I think I am the only one who get best speed <br class=""> >> with <a href="mailto:chacha20-poly1305@openssh.com" target="_blank" rel="noreferrer" class="">chacha20-poly1305@openssh.com</a><br class=""> >><br class=""> >><br class=""> >> It seems the speed of SSH is limited by single core performance which <br class=""> >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). <br class=""> >> Even if CPU has 50% idle, ssh runs on 99.8% of single core.<br class=""> > <br class=""> > The CPU I have has<br class=""> > aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard<br class=""> > <br class=""> > which probably helps.<br class=""> <br class=""> That explains it<br class=""> aesni0: No AES or SHA support.<br class=""> <br class=""> >> I know there were some HPN patches to ssh, beside that is there any <br class=""> >> option I can try to use less CPU?<br class=""> >><br class=""> >> I will play with cpuset to pin ssh on one core and everything else on <br class=""> >> the other core.<br class=""> > <br class=""> > It looks like you are running into a CPU bottleneck TBH<br class=""> <br class=""> Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but <br class=""> without some tweaks on ssh I will not gain more speed :(<br class=""> <br class=""> Thank you for your help!<br class=""> <br class=""> Miroslav Lachman<br class=""> <br class=""> <br class=""> </blockquote></div> </div></blockquote></div><br class=""></body></html>home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0171E506-3899-42B2-B7DC-4145BAA595D7>