Date: Fri, 24 Feb 2023 23:45:15 +0100 (CET) From: Sysadmin Lists <sysadmin.lists@mailfence.com> To: freebsd-fs <freebsd-fs@freebsd.org> Cc: Chris Watson <bsdunix44@gmail.com> Subject: Re: speeding up zfs send | recv (update) Message-ID: <1290947438.348129.1677278715319@fidget.co-bxl> In-Reply-To: <CAHnbxSScPFD5_JyeL0=nBA-jjRzBsWZYZ_T%2BQf3gY7_nnvaWbA@mail.gmail.com> References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <CAOtMX2gifUmgqwSKpRGcfzCm_=BX_szNF1AF8WTMfAmbrJ5UWA@mail.gmail.com> <f6ea3387-faf8-4c63-d1e7-906fa397b00b@sentex.net> <a38578c6-b633-249d-90f0-0652377d76c0@quip.cz> <c229a502-fb76-ec6a-a56b-934d3b56e474@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <CAOjFWZ7k7ANwcGyNCoYMg%2BLUBzAz2VyNfxQo5rKcrYj8XFgG3Q@mail.gmail.com> <741387429.91447.1677122934622@ichabod.co-bxl> <CAHnbxSScPFD5_JyeL0=nBA-jjRzBsWZYZ_T%2BQf3gY7_nnvaWbA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
------=_Part_348126_1685684535.1677278715318 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Feb 23, 2023 at 11:15 AM, Chris Watson <bsdunix44@gmail.com> wrote: [Sorry miroslav, I hit send without checking the To: this was meant to be p= ublic]=C2=A0 I=E2=80=99m a bit late, but I mentioned this to someone on this thread priv= ately, I=E2=80=99m curious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been= mentioned in this thread. I=E2=80=99ve seen everything from VPN=E2=80=99s = to nc. VPNs would be, imo, grossly unwarranted/massively overly complex/har= d to secure just to simply have a secure pipe for doing ZFS send|recv.=C2= =A0 Simply configuring an spiped PtP pipe between A and B seems the simplest, m= ost secure, performant option here. At least considering all the other opti= ons tossed out in this thread.=C2=A0 No one=E2=80=99s using spiped? O.o Thoughts?=C2=A0 Has anyone compared ssh to spiped regarding overhead and throughput in this= scenario? Chris On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists <sysadmin.lists@mailfence.co= m> wrote: On Feb 22, 2023 at 1:43 PM, Freddie Cash <fjwcash@gmail.com> wrote: [Sorry for top part, GMail sucks for replies.] If this is a LAN or private WAN where you trust the network, piping the sen= d stream through netcat will remove ssh from the equation. That's what we switched to using once it became almost impossible to get th= e "none" cipher working with ssh on FreeBSD. We use ssh to connect to the remote server and enable a netcat listener on = port X, then pipe the send through netcat to the remote system on port X. T= hat way it's logged and uses ssh for authentication. We easily saturate gigabit links between our ZFS systems using netcat. Cheers, Freddie Typos due to smartphone keyboard. On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz> wrot= e: On 22/02/2023 22:08, mike tancsa wrote: > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: >> Interresting numbers. I think I am the only one who get best speed=20 >> with chacha20-poly1305@openssh.com >> >> >> It seems the speed of SSH is limited by single core performance which=20 >> is very poor on this machine (Intel(R) Pentium(R) Dual=C2=A0 CPU E2160).= =20 >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. >=20 > The CPU I have has > aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard >=20 > which probably helps. That explains it aesni0: No AES or SHA support. >> I know there were some HPN patches to ssh, beside that is there any=20 >> option I can try to use less CPU? >> >> I will play with cpuset to pin ssh on one core and everything else on=20 >> the other core. >=20 > It looks like you are running into a CPU bottleneck TBH Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but=20 without some tweaks on ssh I will not gain more speed :( Thank you for your help! Miroslav Lachman You could pipe the stream through an encrypting program before piping to netcat, then decrypt on the recieving end. $ zfs send | crypt | netcat ipaddr 2222 $ netcat -vl 2222 | crypt | zfs recv I don't know if zfs can handle that, but worth a try. $ man crypt =C2=A0 =C2=A0 The enigma utility, also known as crypt is a very simple encr= yption =C2=A0 =C2=A0 =C2=A0program, working on a =E2=80=9Csecret-key=E2=80=9D basi= s.=C2=A0 It operates as a filter, i.e., =C2=A0 =C2=A0 =C2=A0it encrypts or decrypts a stream of data from standard = input, and writes =C2=A0 =C2=A0 =C2=A0the result to standard output.=C2=A0 Since its operatio= n is fully symmetrical, =C2=A0 =C2=A0 =C2=A0feeding the encrypted data stream again through the eng= ine (using the =C2=A0 =C2=A0 =C2=A0same secret key) will decrypt it. -- Sent with https://mailfence.com Secure and private email I've used it before, but forgot about it. But it's not part of base, and th= ere are tools in base which together perform a similar task, so that probably explains why many people haven't heard about it or forgot they had. Most everyone has at some point needed to transfer a couple files to a loca= l machine with a LAN connection but borked authentication services. In steps nc and optionally crypt or openssl to encrypt the data. Simple. -- Sent with https://mailfence.com Secure and private email ------=_Part_348126_1685684535.1677278715318 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline <div style=3D'font-family:Times New Roman; color:#000000; font-size:18px;'>= <div>On Feb 23, 2023 at 11:15 AM, Chris Watson <bsdunix44@gmail.com> = wrote:</div><div><blockquote type=3D"cite" cite=3D"<CAHnbxSScPFD5_JyeL0=3Dn= BA-jjRzBsWZYZ_T+Qf3gY7_nnvaWbA@mail.gmail.com>"><div><div dir=3D"auto" styl= e=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(4= 9,49,49)">[Sorry miroslav, I hit send without checking the To: this was mea= nt to be public] </div><div dir=3D"auto" style=3D"font-size:1rem;word-= spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"><br></div><div = dir=3D"auto" style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,4= 9,49);color:rgb(49,49,49)">I=E2=80=99m a bit late, but I mentioned this to = someone on this thread privately, I=E2=80=99m curious why =E2=80=98spiped= =E2=80=99 hasn=E2=80=99t been mentioned in this thread. I=E2=80=99ve seen e= verything from VPN=E2=80=99s to nc. VPNs would be, imo, grossly unwarranted= /massively overly complex/hard to secure just to simply have a secure pipe = for doing ZFS send|recv. </div><div dir=3D"auto" style=3D"word-spacing= :1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"><br></div><div dir=3D"= auto" style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);c= olor:rgb(49,49,49)">Simply configuring an spiped PtP pipe between A and B s= eems the simplest, most secure, performant option here. At least considerin= g all the other options tossed out in this thread. </div><div dir=3D"a= uto" style=3D"word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,4= 9)"><br></div><div dir=3D"auto" style=3D"font-size:1rem;word-spacing:1px;bo= rder-color:rgb(49,49,49);color:rgb(49,49,49)">No one=E2=80=99s using spiped= ? O.o</div><div dir=3D"auto" style=3D"word-spacing:1px;border-color:rgb(49,= 49,49);color:rgb(49,49,49)"><br></div><div dir=3D"auto" style=3D"font-size:= 1rem;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">Thoug= hts? </div><div dir=3D"auto" style=3D"word-spacing:1px;border-color:rg= b(49,49,49);color:rgb(49,49,49)"><br></div><div dir=3D"auto" style=3D"font-= size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)">= Has anyone compared ssh to spiped regarding overhead and throughput in this= scenario?</div><div dir=3D"auto" style=3D"font-size:1rem;word-spacing:1px;= border-color:rgb(49,49,49);color:rgb(49,49,49)"><br></div><div dir=3D"auto"= style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:= rgb(49,49,49)">Chris</div></div><div><br><div class=3D""><div dir=3D"ltr" c= lass=3D"">On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists <<a href=3D"mai= lto:sysadmin.lists@mailfence.com">sysadmin.lists@mailfence.com</a>> wrot= e:<br></div><blockquote class=3D"" style=3D"margin:0px 0px 0px 0.8ex;border= -left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:= rgb(204,204,204)"><div style=3D"font-family:"Times New Roman";fon= t-size:18px;color:rgb(0,0,0)"><br><div style=3D"font-family:"Times New= Roman""></div></div><div style=3D"font-family:"Times New Roman&q= uot;;font-size:18px;color:rgb(0,0,0)"><div style=3D"font-family:"Times= New Roman"">On Feb 22, 2023 at 1:43 PM, Freddie Cash <<a href=3D"m= ailto:fjwcash@gmail.com" target=3D"_blank" style=3D"font-family:"Times= New Roman"">fjwcash@gmail.com</a>> wrote:<blockquote type=3D"cite"= style=3D"font-family:"Times New Roman""><div dir=3D"auto" style= =3D"font-family:"Times New Roman"">[Sorry for top part, GMail suc= ks for replies.]<div dir=3D"auto" style=3D"font-family:"Times New Roma= n""><br></div><div dir=3D"auto" style=3D"font-family:"Times New R= oman"">If this is a LAN or private WAN where you trust the network, pi= ping the send stream through netcat will remove ssh from the equation.<div = dir=3D"auto" style=3D"font-family:"Times New Roman""><br></div><d= iv dir=3D"auto" style=3D"font-family:"Times New Roman"">That's wh= at we switched to using once it became almost impossible to get the "none" = cipher working with ssh on FreeBSD.</div><div dir=3D"auto" style=3D"font-fa= mily:"Times New Roman""><br></div><div dir=3D"auto" style=3D"font= -family:"Times New Roman"">We use ssh to connect to the remote se= rver and enable a netcat listener on port X, then pipe the send through net= cat to the remote system on port X. That way it's logged and uses ssh for a= uthentication.</div><div dir=3D"auto" style=3D"font-family:"Times New = Roman""><br></div><div dir=3D"auto" style=3D"font-family:"Times N= ew Roman"">We easily saturate gigabit links between our ZFS systems us= ing netcat.<br></div><div dir=3D"auto" style=3D"font-family:"Times New= Roman""><br></div><div dir=3D"auto" style=3D"font-family:"Times = New Roman""><br><br><div data-smartmail=3D"gmail_signature" dir=3D"aut= o" style=3D"font-family:"Times New Roman"">Cheers,<br>Freddie<br>= <br>Typos due to smartphone keyboard.</div></div></div></div><br><div style= =3D"font-family:"Times New Roman""><div dir=3D"ltr" style=3D"font= -family:"Times New Roman"">On Wed., Feb. 22, 2023, 1:31 p.m. Miro= slav Lachman, <<a href=3D"mailto:000.fbsd@quip.cz" target=3D"_blank" sty= le=3D"font-family:"Times New Roman"">000.fbsd@quip.cz</a>> wro= te:<br></div><blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left-widt= h:1px;border-left-style:solid;padding-left:1ex;font-family:"Times New = Roman";border-left-color:rgb(204,204,204)">On 22/02/2023 22:08, mike t= ancsa wrote:<br> > On 2/22/2023 4:03 PM, Miroslav Lachman wrote:<br> >> Interresting numbers. I think I am the only one who get best speed= <br> >> with <a href=3D"mailto:chacha20-poly1305@openssh.com" rel=3D"noref= errer" target=3D"_blank" style=3D"font-family:"Times New Roman"">= chacha20-poly1305@openssh.com</a><br> >><br> >><br> >> It seems the speed of SSH is limited by single core performance wh= ich <br> >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU E= 2160). <br> >> Even if CPU has 50% idle, ssh runs on 99.8% of single core.<br> > <br> > The CPU I have has<br> > aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard= <br> > <br> > which probably helps.<br> <br> That explains it<br> aesni0: No AES or SHA support.<br> <br> >> I know there were some HPN patches to ssh, beside that is there an= y <br> >> option I can try to use less CPU?<br> >><br> >> I will play with cpuset to pin ssh on one core and everything else= on <br> >> the other core.<br> > <br> > It looks like you are running into a CPU bottleneck TBH<br> <br> Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but <br> without some tweaks on ssh I will not gain more speed :(<br> <br> Thank you for your help!<br> <br> Miroslav Lachman<br> <br> <br></blockquote></div></blockquote><div style=3D"font-family:"Times N= ew Roman""><br></div></div></div><div style=3D"font-family:"Times= New Roman";font-size:18px;color:rgb(0,0,0)"><div style=3D"font-family= :"Times New Roman""><div style=3D"font-family:"Times New Rom= an""><div style=3D"font-family:"Times New Roman"">You could = pipe the stream through an encrypting program before piping to</div><div st= yle=3D"font-family:"Times New Roman"">netcat, then decrypt on the= recieving end.</div><div style=3D"font-family:"Times New Roman""= ><br></div><div style=3D"font-family:"Times New Roman"">$ zfs sen= d | crypt | netcat ipaddr 2222</div><div style=3D"font-family:"Times N= ew Roman"">$ netcat -vl 2222 | crypt | zfs recv</div><div style=3D"fon= t-family:"Times New Roman""><br></div><div style=3D"font-family:&= quot;Times New Roman"">I don't know if zfs can handle that, but worth = a try.</div></div><div style=3D"font-family:"Times New Roman""><b= r></div><div style=3D"font-family:"Times New Roman"">$ man crypt<= /div><div style=3D"font-family:"Times New Roman""><div style=3D"f= ont-family:"Times New Roman""> The enigma utility, a= lso known as crypt is a very simple encryption</div><div style=3D"font-fami= ly:"Times New Roman""> program, working on a = =E2=80=9Csecret-key=E2=80=9D basis. It operates as a filter, i.e.,</d= iv><div style=3D"font-family:"Times New Roman""> &nb= sp;it encrypts or decrypts a stream of data from standard input, and writes= </div><div style=3D"font-family:"Times New Roman""> = the result to standard output. Since its operation is fully sym= metrical,</div><div style=3D"font-family:"Times New Roman""> = ; feeding the encrypted data stream again through the engine (= using the</div><div style=3D"font-family:"Times New Roman""> = ; same secret key) will decrypt it.</div></div><div style=3D"f= ont-family:"Times New Roman""><br></div><div style=3D"font-family= :"Times New Roman""><br></div></div></div> --=20 Sent with <a href=3D"https://mailfence.com" target=3D"_blank">https://mailf= ence.com</a> =20 Secure and private email</blockquote></div></div></blockquote><div><br></di= v><div>I've used it before, but forgot about it. But it's not part of base,= and there</div><div>are tools in base which together perform a similar tas= k, so that probably</div><div>explains why many people haven't heard about = it or forgot they had.</div><div><br></div><div>Most everyone has at some p= oint needed to transfer a couple files to a local</div><div>machine with a = LAN connection but borked authentication services. In steps</div><div>nc an= d optionally crypt or openssl to encrypt the data. Simple.</div><div><br></= div><div><br></div></div></div> --=20 Sent with https://mailfence.com =20 Secure and private email ------=_Part_348126_1685684535.1677278715318--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1290947438.348129.1677278715319>