Date: Thu, 23 Feb 2023 13:15:32 -0600 From: Chris Watson <bsdunix44@gmail.com> To: Sysadmin Lists <sysadmin.lists@mailfence.com> Cc: Freddie Cash <fjwcash@gmail.com>, freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: speeding up zfs send | recv (update) Message-ID: <CAHnbxSScPFD5_JyeL0=nBA-jjRzBsWZYZ_T%2BQf3gY7_nnvaWbA@mail.gmail.com> In-Reply-To: <741387429.91447.1677122934622@ichabod.co-bxl> References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <CAOtMX2gifUmgqwSKpRGcfzCm_=BX_szNF1AF8WTMfAmbrJ5UWA@mail.gmail.com> <f6ea3387-faf8-4c63-d1e7-906fa397b00b@sentex.net> <a38578c6-b633-249d-90f0-0652377d76c0@quip.cz> <c229a502-fb76-ec6a-a56b-934d3b56e474@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <CAOjFWZ7k7ANwcGyNCoYMg%2BLUBzAz2VyNfxQo5rKcrYj8XFgG3Q@mail.gmail.com> <741387429.91447.1677122934622@ichabod.co-bxl>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000d0c06905f562d633 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable [Sorry miroslav, I hit send without checking the To: this was meant to be public] I=E2=80=99m a bit late, but I mentioned this to someone on this thread priv= ately, I=E2=80=99m curious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been mentio= ned in this thread. I=E2=80=99ve seen everything from VPN=E2=80=99s to nc. VPNs would be, imo, grossly unwarranted/massively overly complex/hard to secure just to simply have a secure pipe for doing ZFS send|recv. Simply configuring an spiped PtP pipe between A and B seems the simplest, most secure, performant option here. At least considering all the other options tossed out in this thread. No one=E2=80=99s using spiped? O.o Thoughts? Has anyone compared ssh to spiped regarding overhead and throughput in this scenario? Chris On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists <sysadmin.lists@mailfence.co= m> wrote: > > On Feb 22, 2023 at 1:43 PM, Freddie Cash <fjwcash@gmail.com> wrote: > > [Sorry for top part, GMail sucks for replies.] > > If this is a LAN or private WAN where you trust the network, piping the > send stream through netcat will remove ssh from the equation. > > That's what we switched to using once it became almost impossible to get > the "none" cipher working with ssh on FreeBSD. > > We use ssh to connect to the remote server and enable a netcat listener o= n > port X, then pipe the send through netcat to the remote system on port X. > That way it's logged and uses ssh for authentication. > > We easily saturate gigabit links between our ZFS systems using netcat. > > > > Cheers, > Freddie > > Typos due to smartphone keyboard. > > On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> > wrote: > > On 22/02/2023 22:08, mike tancsa wrote: > > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: > >> Interresting numbers. I think I am the only one who get best speed > >> with chacha20-poly1305@openssh.com > >> > >> > >> It seems the speed of SSH is limited by single core performance which > >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). > >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. > > > > The CPU I have has > > aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard > > > > which probably helps. > > That explains it > aesni0: No AES or SHA support. > > >> I know there were some HPN patches to ssh, beside that is there any > >> option I can try to use less CPU? > >> > >> I will play with cpuset to pin ssh on one core and everything else on > >> the other core. > > > > It looks like you are running into a CPU bottleneck TBH > > Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but > without some tweaks on ssh I will not gain more speed :( > > Thank you for your help! > > Miroslav Lachman > > > > You could pipe the stream through an encrypting program before piping to > netcat, then decrypt on the recieving end. > > $ zfs send | crypt | netcat ipaddr 2222 > $ netcat -vl 2222 | crypt | zfs recv > > I don't know if zfs can handle that, but worth a try. > > $ man crypt > The enigma utility, also known as crypt is a very simple encryption > program, working on a =E2=80=9Csecret-key=E2=80=9D basis. It operat= es as a filter, > i.e., > it encrypts or decrypts a stream of data from standard input, and > writes > the result to standard output. Since its operation is fully > symmetrical, > feeding the encrypted data stream again through the engine (using th= e > same secret key) will decrypt it. > > > -- Sent with https://mailfence.com Secure and private email --000000000000d0c06905f562d633 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div><div dir=3D"auto" style=3D"font-size:1rem;word-spacing:1px;border-colo= r:rgb(49,49,49);color:rgb(49,49,49)">[Sorry miroslav, I hit send without ch= ecking the To: this was meant to be public]=C2=A0</div><div dir=3D"auto" st= yle=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:rgb= (49,49,49)"><br></div><div dir=3D"auto" style=3D"font-size:1rem;word-spacin= g:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">I=E2=80=99m a bit lat= e, but I mentioned this to someone on this thread privately, I=E2=80=99m cu= rious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been mentioned in this th= read. I=E2=80=99ve seen everything from VPN=E2=80=99s to nc. VPNs would be,= imo, grossly unwarranted/massively overly complex/hard to secure just to s= imply have a secure pipe for doing ZFS send|recv.=C2=A0</div><div dir=3D"au= to" style=3D"word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49= )"><br></div><div dir=3D"auto" style=3D"font-size:1rem;word-spacing:1px;bor= der-color:rgb(49,49,49);color:rgb(49,49,49)">Simply configuring an spiped P= tP pipe between A and B seems the simplest, most secure, performant option = here. At least considering all the other options tossed out in this thread.= =C2=A0</div><div dir=3D"auto" style=3D"word-spacing:1px;border-color:rgb(49= ,49,49);color:rgb(49,49,49)"><br></div><div dir=3D"auto" style=3D"font-size= :1rem;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">No o= ne=E2=80=99s using spiped? O.o</div><div dir=3D"auto" style=3D"word-spacing= :1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"><br></div><div dir=3D"= auto" style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);c= olor:rgb(49,49,49)">Thoughts?=C2=A0</div><div dir=3D"auto" style=3D"word-sp= acing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"><br></div><div di= r=3D"auto" style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,= 49);color:rgb(49,49,49)">Has anyone compared ssh to spiped regarding overhe= ad and throughput in this scenario?</div><div dir=3D"auto" style=3D"font-si= ze:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"><b= r></div><div dir=3D"auto" style=3D"font-size:1rem;word-spacing:1px;border-c= olor:rgb(49,49,49);color:rgb(49,49,49)">Chris</div></div><div><br><div clas= s=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Feb 22, 202= 3 at 9:29 PM Sysadmin Lists <<a href=3D"mailto:sysadmin.lists@mailfence.= com">sysadmin.lists@mailfence.com</a>> wrote:<br></div><blockquote class= =3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;bo= rder-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">= <div style=3D"font-family:"Times New Roman";font-size:18px;color:= rgb(0,0,0)"><br><div style=3D"font-family:"Times New Roman""></di= v></div><div style=3D"font-family:"Times New Roman";font-size:18p= x;color:rgb(0,0,0)"><div style=3D"font-family:"Times New Roman"">= On Feb 22, 2023 at 1:43 PM, Freddie Cash <<a href=3D"mailto:fjwcash@gmai= l.com" target=3D"_blank" style=3D"font-family:"Times New Roman"">= fjwcash@gmail.com</a>> wrote:<blockquote type=3D"cite" style=3D"font-fam= ily:"Times New Roman""><div dir=3D"auto" style=3D"font-family:&qu= ot;Times New Roman"">[Sorry for top part, GMail sucks for replies.]<di= v dir=3D"auto" style=3D"font-family:"Times New Roman""><br></div>= <div dir=3D"auto" style=3D"font-family:"Times New Roman"">If this= is a LAN or private WAN where you trust the network, piping the send strea= m through netcat will remove ssh from the equation.<div dir=3D"auto" style= =3D"font-family:"Times New Roman""><br></div><div dir=3D"auto" st= yle=3D"font-family:"Times New Roman"">That's what we switched= to using once it became almost impossible to get the "none" ciph= er working with ssh on FreeBSD.</div><div dir=3D"auto" style=3D"font-family= :"Times New Roman""><br></div><div dir=3D"auto" style=3D"font-fam= ily:"Times New Roman"">We use ssh to connect to the remote server= and enable a netcat listener on port X, then pipe the send through netcat = to the remote system on port X. That way it's logged and uses ssh for a= uthentication.</div><div dir=3D"auto" style=3D"font-family:"Times New = Roman""><br></div><div dir=3D"auto" style=3D"font-family:"Times N= ew Roman"">We easily saturate gigabit links between our ZFS systems us= ing netcat.<br></div><div dir=3D"auto" style=3D"font-family:"Times New= Roman""><br></div><div dir=3D"auto" style=3D"font-family:"Times = New Roman""><br><br><div data-smartmail=3D"gmail_signature" dir=3D"aut= o" style=3D"font-family:"Times New Roman"">Cheers,<br>Freddie<br>= <br>Typos due to smartphone keyboard.</div></div></div></div><br><div style= =3D"font-family:"Times New Roman""><div dir=3D"ltr" style=3D"font= -family:"Times New Roman"">On Wed., Feb. 22, 2023, 1:31 p.m. Miro= slav Lachman, <<a href=3D"mailto:000.fbsd@quip.cz" target=3D"_blank" sty= le=3D"font-family:"Times New Roman"">000.fbsd@quip.cz</a>> wro= te:<br></div><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left-widt= h:1px;border-left-style:solid;padding-left:1ex;font-family:"Times New = Roman";border-left-color:rgb(204,204,204)">On 22/02/2023 22:08, mike t= ancsa wrote:<br> > On 2/22/2023 4:03 PM, Miroslav Lachman wrote:<br> >> Interresting numbers. I think I am the only one who get best speed= <br> >> with <a href=3D"mailto:chacha20-poly1305@openssh.com" rel=3D"noref= errer" target=3D"_blank" style=3D"font-family:"Times New Roman"">= chacha20-poly1305@openssh.com</a><br> >><br> >><br> >> It seems the speed of SSH is limited by single core performance wh= ich <br> >> is very poor on this machine (Intel(R) Pentium(R) Dual=C2=A0 CPU E= 2160). <br> >> Even if CPU has 50% idle, ssh runs on 99.8% of single core.<br> > <br> > The CPU I have has<br> > aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard= <br> > <br> > which probably helps.<br> <br> That explains it<br> aesni0: No AES or SHA support.<br> <br> >> I know there were some HPN patches to ssh, beside that is there an= y <br> >> option I can try to use less CPU?<br> >><br> >> I will play with cpuset to pin ssh on one core and everything else= on <br> >> the other core.<br> > <br> > It looks like you are running into a CPU bottleneck TBH<br> <br> Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but <br> without some tweaks on ssh I will not gain more speed :(<br> <br> Thank you for your help!<br> <br> Miroslav Lachman<br> <br> <br></blockquote></div></blockquote><div style=3D"font-family:"Times N= ew Roman""><br></div></div></div><div style=3D"font-family:"Times= New Roman";font-size:18px;color:rgb(0,0,0)"><div style=3D"font-family= :"Times New Roman""><div style=3D"font-family:"Times New Rom= an""><div style=3D"font-family:"Times New Roman"">You could = pipe the stream through an encrypting program before piping to</div><div st= yle=3D"font-family:"Times New Roman"">netcat, then decrypt on the= recieving end.</div><div style=3D"font-family:"Times New Roman""= ><br></div><div style=3D"font-family:"Times New Roman"">$ zfs sen= d | crypt | netcat ipaddr 2222</div><div style=3D"font-family:"Times N= ew Roman"">$ netcat -vl 2222 | crypt | zfs recv</div><div style=3D"fon= t-family:"Times New Roman""><br></div><div style=3D"font-family:&= quot;Times New Roman"">I don't know if zfs can handle that, but wo= rth a try.</div></div><div style=3D"font-family:"Times New Roman"= "><br></div><div style=3D"font-family:"Times New Roman"">$ man cr= ypt</div><div style=3D"font-family:"Times New Roman""><div style= =3D"font-family:"Times New Roman"">=C2=A0 =C2=A0 The enigma utili= ty, also known as crypt is a very simple encryption</div><div style=3D"font= -family:"Times New Roman"">=C2=A0 =C2=A0 =C2=A0program, working o= n a =E2=80=9Csecret-key=E2=80=9D basis.=C2=A0 It operates as a filter, i.e.= ,</div><div style=3D"font-family:"Times New Roman"">=C2=A0 =C2=A0= =C2=A0it encrypts or decrypts a stream of data from standard input, and wr= ites</div><div style=3D"font-family:"Times New Roman"">=C2=A0 =C2= =A0 =C2=A0the result to standard output.=C2=A0 Since its operation is fully= symmetrical,</div><div style=3D"font-family:"Times New Roman"">= =C2=A0 =C2=A0 =C2=A0feeding the encrypted data stream again through the eng= ine (using the</div><div style=3D"font-family:"Times New Roman"">= =C2=A0 =C2=A0 =C2=A0same secret key) will decrypt it.</div></div><div style= =3D"font-family:"Times New Roman""><br></div><div style=3D"font-f= amily:"Times New Roman""><br></div></div></div> --=20 Sent with <a href=3D"https://mailfence.com" target=3D"_blank">https://mailf= ence.com</a> =20 Secure and private email </blockquote></div></div> --000000000000d0c06905f562d633--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHnbxSScPFD5_JyeL0=nBA-jjRzBsWZYZ_T%2BQf3gY7_nnvaWbA>