Date: Thu, 23 Feb 2023 15:28:08 -0500 From: Charles Sprickman <spork@bway.net> To: Chris Watson <bsdunix44@gmail.com> Cc: Sysadmin Lists <sysadmin.lists@mailfence.com>, Freddie Cash <fjwcash@gmail.com>, freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: speeding up zfs send | recv (update) Message-ID: <E9E879F2-5A55-4AED-8EE8-F1698CA721F5@bway.net> In-Reply-To: <CAHnbxSScPFD5_JyeL0=nBA-jjRzBsWZYZ_T%2BQf3gY7_nnvaWbA@mail.gmail.com> References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <CAOtMX2gifUmgqwSKpRGcfzCm_=BX_szNF1AF8WTMfAmbrJ5UWA@mail.gmail.com> <f6ea3387-faf8-4c63-d1e7-906fa397b00b@sentex.net> <a38578c6-b633-249d-90f0-0652377d76c0@quip.cz> <c229a502-fb76-ec6a-a56b-934d3b56e474@sentex.net> <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> <46455168-d7f1-6ca9-ad2f-9bcd3359e0f3@sentex.net> <78c78aec-a34b-f188-ef96-8ced9a1eda35@quip.cz> <CAOjFWZ7k7ANwcGyNCoYMg%2BLUBzAz2VyNfxQo5rKcrYj8XFgG3Q@mail.gmail.com> <741387429.91447.1677122934622@ichabod.co-bxl> <CAHnbxSScPFD5_JyeL0=nBA-jjRzBsWZYZ_T%2BQf3gY7_nnvaWbA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_DA2B7E2D-53BE-4B3C-B32E-B84FDDEE8615 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Feb 23, 2023, at 2:15 PM, Chris Watson <bsdunix44@gmail.com> wrote: >=20 > [Sorry miroslav, I hit send without checking the To: this was meant to = be public]=20 >=20 > I=E2=80=99m a bit late, but I mentioned this to someone on this thread = privately, I=E2=80=99m curious why =E2=80=98spiped=E2=80=99 hasn=E2=80=99t= been mentioned in this thread. I=E2=80=99ve seen everything from = VPN=E2=80=99s to nc. VPNs would be, imo, grossly unwarranted/massively = overly complex/hard to secure just to simply have a secure pipe for = doing ZFS send|recv.=20 >=20 > Simply configuring an spiped PtP pipe between A and B seems the = simplest, most secure, performant option here. At least considering all = the other options tossed out in this thread.=20 Does it scale across multiple cores? Charles >=20 > No one=E2=80=99s using spiped? O.o >=20 > Thoughts?=20 >=20 > Has anyone compared ssh to spiped regarding overhead and throughput in = this scenario? >=20 > Chris >=20 > On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists = <sysadmin.lists@mailfence.com <mailto:sysadmin.lists@mailfence.com>> = wrote: >=20 > On Feb 22, 2023 at 1:43 PM, Freddie Cash <fjwcash@gmail.com = <mailto:fjwcash@gmail.com>> wrote: >>=20 >> [Sorry for top part, GMail sucks for replies.] >>=20 >> If this is a LAN or private WAN where you trust the network, piping = the send stream through netcat will remove ssh from the equation. >>=20 >> That's what we switched to using once it became almost impossible to = get the "none" cipher working with ssh on FreeBSD. >>=20 >> We use ssh to connect to the remote server and enable a netcat = listener on port X, then pipe the send through netcat to the remote = system on port X. That way it's logged and uses ssh for authentication. >>=20 >> We easily saturate gigabit links between our ZFS systems using = netcat. >>=20 >>=20 >>=20 >> Cheers, >> Freddie >>=20 >> Typos due to smartphone keyboard. >>=20 >> On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav Lachman, <000.fbsd@quip.cz = <mailto:000.fbsd@quip.cz>> wrote: >> On 22/02/2023 22:08, mike tancsa wrote: >> > On 2/22/2023 4:03 PM, Miroslav Lachman wrote: >> >> Interresting numbers. I think I am the only one who get best speed=20= >> >> with chacha20-poly1305@openssh.com = <mailto:chacha20-poly1305@openssh.com> >> >> >> >> >> >> It seems the speed of SSH is limited by single core performance = which=20 >> >> is very poor on this machine (Intel(R) Pentium(R) Dual CPU = E2160).=20 >> >> Even if CPU has 50% idle, ssh runs on 99.8% of single core. >> >=20 >> > The CPU I have has >> > aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard >> >=20 >> > which probably helps. >>=20 >> That explains it >> aesni0: No AES or SHA support. >>=20 >> >> I know there were some HPN patches to ssh, beside that is there = any=20 >> >> option I can try to use less CPU? >> >> >> >> I will play with cpuset to pin ssh on one core and everything else = on=20 >> >> the other core. >> >=20 >> > It looks like you are running into a CPU bottleneck TBH >>=20 >> Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but=20 >> without some tweaks on ssh I will not gain more speed :( >>=20 >> Thank you for your help! >>=20 >> Miroslav Lachman >>=20 >>=20 >=20 > You could pipe the stream through an encrypting program before piping = to > netcat, then decrypt on the recieving end. >=20 > $ zfs send | crypt | netcat ipaddr 2222 > $ netcat -vl 2222 | crypt | zfs recv >=20 > I don't know if zfs can handle that, but worth a try. >=20 > $ man crypt > The enigma utility, also known as crypt is a very simple = encryption > program, working on a =E2=80=9Csecret-key=E2=80=9D basis. It = operates as a filter, i.e., > it encrypts or decrypts a stream of data from standard input, and = writes > the result to standard output. Since its operation is fully = symmetrical, > feeding the encrypted data stream again through the engine (using = the > same secret key) will decrypt it. >=20 >=20 > -- Sent with https://mailfence.com <https://mailfence.com/>; Secure and = private email --Apple-Mail=_DA2B7E2D-53BE-4B3C-B32E-B84FDDEE8615 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"Content-Type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"word-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;" class=3D""><br = class=3D""><div><blockquote type=3D"cite" class=3D""><div class=3D"">On = Feb 23, 2023, at 2:15 PM, Chris Watson <<a = href=3D"mailto:bsdunix44@gmail.com" class=3D"">bsdunix44@gmail.com</a>>= wrote:</div><br class=3D"Apple-interchange-newline"><div class=3D""><div = class=3D""><div dir=3D"auto" = style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:= rgb(49,49,49)" class=3D"">[Sorry miroslav, I hit send without checking = the To: this was meant to be public] </div><div dir=3D"auto" = style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:= rgb(49,49,49)" class=3D""><br class=3D""></div><div dir=3D"auto" = style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:= rgb(49,49,49)" class=3D"">I=E2=80=99m a bit late, but I mentioned this = to someone on this thread privately, I=E2=80=99m curious why = =E2=80=98spiped=E2=80=99 hasn=E2=80=99t been mentioned in this thread. = I=E2=80=99ve seen everything from VPN=E2=80=99s to nc. VPNs would be, = imo, grossly unwarranted/massively overly complex/hard to secure just to = simply have a secure pipe for doing ZFS send|recv. </div><div = dir=3D"auto" = style=3D"word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)" = class=3D""><br class=3D""></div><div dir=3D"auto" = style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:= rgb(49,49,49)" class=3D"">Simply configuring an spiped PtP pipe between = A and B seems the simplest, most secure, performant option here. At = least considering all the other options tossed out in this = thread. </div></div></div></blockquote><div><br = class=3D""></div><div>Does it scale across multiple cores?</div><div><br = class=3D""></div><div>Charles</div><br class=3D""><blockquote = type=3D"cite" class=3D""><div class=3D""><div class=3D""><div dir=3D"auto"= style=3D"word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)"= class=3D""><br class=3D""></div><div dir=3D"auto" = style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:= rgb(49,49,49)" class=3D"">No one=E2=80=99s using spiped? O.o</div><div = dir=3D"auto" = style=3D"word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)" = class=3D""><br class=3D""></div><div dir=3D"auto" = style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:= rgb(49,49,49)" class=3D"">Thoughts? </div><div dir=3D"auto" = style=3D"word-spacing:1px;border-color:rgb(49,49,49);color:rgb(49,49,49)" = class=3D""><br class=3D""></div><div dir=3D"auto" = style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:= rgb(49,49,49)" class=3D"">Has anyone compared ssh to spiped regarding = overhead and throughput in this scenario?</div><div dir=3D"auto" = style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:= rgb(49,49,49)" class=3D""><br class=3D""></div><div dir=3D"auto" = style=3D"font-size:1rem;word-spacing:1px;border-color:rgb(49,49,49);color:= rgb(49,49,49)" class=3D"">Chris</div></div><div class=3D""><br = class=3D""><div class=3D"gmail_quote"><div dir=3D"ltr" = class=3D"gmail_attr">On Wed, Feb 22, 2023 at 9:29 PM Sysadmin Lists = <<a href=3D"mailto:sysadmin.lists@mailfence.com" = class=3D"">sysadmin.lists@mailfence.com</a>> wrote:<br = class=3D""></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px = 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;borde= r-left-color:rgb(204,204,204)"><div style=3D"font-family: "Times = New Roman"; font-size: 18px;" class=3D""><br class=3D""><div = style=3D"font-family:"Times New Roman"" = class=3D""></div></div><div style=3D"font-family: "Times New = Roman"; font-size: 18px;" class=3D""><div = style=3D"font-family:"Times New Roman"" class=3D"">On Feb 22, = 2023 at 1:43 PM, Freddie Cash <<a href=3D"mailto:fjwcash@gmail.com" = target=3D"_blank" style=3D"font-family:"Times New Roman"" = class=3D"">fjwcash@gmail.com</a>> wrote:<blockquote type=3D"cite" = style=3D"font-family:"Times New Roman"" class=3D""><div = dir=3D"auto" style=3D"font-family:"Times New Roman"" = class=3D"">[Sorry for top part, GMail sucks for replies.]<div dir=3D"auto"= style=3D"font-family:"Times New Roman"" class=3D""><br = class=3D""></div><div dir=3D"auto" style=3D"font-family:"Times New = Roman"" class=3D"">If this is a LAN or private WAN where you trust = the network, piping the send stream through netcat will remove ssh from = the equation.<div dir=3D"auto" style=3D"font-family:"Times New = Roman"" class=3D""><br class=3D""></div><div dir=3D"auto" = style=3D"font-family:"Times New Roman"" class=3D"">That's what = we switched to using once it became almost impossible to get the "none" = cipher working with ssh on FreeBSD.</div><div dir=3D"auto" = style=3D"font-family:"Times New Roman"" class=3D""><br = class=3D""></div><div dir=3D"auto" style=3D"font-family:"Times New = Roman"" class=3D"">We use ssh to connect to the remote server and = enable a netcat listener on port X, then pipe the send through netcat to = the remote system on port X. That way it's logged and uses ssh for = authentication.</div><div dir=3D"auto" style=3D"font-family:"Times = New Roman"" class=3D""><br class=3D""></div><div dir=3D"auto" = style=3D"font-family:"Times New Roman"" class=3D"">We easily = saturate gigabit links between our ZFS systems using netcat.<br = class=3D""></div><div dir=3D"auto" style=3D"font-family:"Times New = Roman"" class=3D""><br class=3D""></div><div dir=3D"auto" = style=3D"font-family:"Times New Roman"" class=3D""><br = class=3D""><br class=3D""><div data-smartmail=3D"gmail_signature" = dir=3D"auto" style=3D"font-family:"Times New Roman"" = class=3D"">Cheers,<br class=3D"">Freddie<br class=3D""><br = class=3D"">Typos due to smartphone keyboard.</div></div></div></div><br = class=3D""><div style=3D"font-family:"Times New Roman"" = class=3D""><div dir=3D"ltr" style=3D"font-family:"Times New = Roman"" class=3D"">On Wed., Feb. 22, 2023, 1:31 p.m. Miroslav = Lachman, <<a href=3D"mailto:000.fbsd@quip.cz" target=3D"_blank" = style=3D"font-family:"Times New Roman"" = class=3D"">000.fbsd@quip.cz</a>> wrote:<br class=3D""></div><blockquote= style=3D"margin:0px 0px 0px = 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;font-= family:"Times New Roman";border-left-color:rgb(204,204,204)" = class=3D"">On 22/02/2023 22:08, mike tancsa wrote:<br class=3D""> > On 2/22/2023 4:03 PM, Miroslav Lachman wrote:<br class=3D""> >> Interresting numbers. I think I am the only one who get best = speed <br class=3D""> >> with <a href=3D"mailto:chacha20-poly1305@openssh.com" = rel=3D"noreferrer" target=3D"_blank" style=3D"font-family:"Times = New Roman"" class=3D"">chacha20-poly1305@openssh.com</a><br = class=3D""> >><br class=3D""> >><br class=3D""> >> It seems the speed of SSH is limited by single core performance = which <br class=3D""> >> is very poor on this machine (Intel(R) Pentium(R) Dual = CPU E2160). <br class=3D""> >> Even if CPU has 50% idle, ssh runs on 99.8% of single core.<br = class=3D""> > <br class=3D""> > The CPU I have has<br class=3D""> > aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on = motherboard<br class=3D""> > <br class=3D""> > which probably helps.<br class=3D""> <br class=3D""> That explains it<br class=3D""> aesni0: No AES or SHA support.<br class=3D""> <br class=3D""> >> I know there were some HPN patches to ssh, beside that is there = any <br class=3D""> >> option I can try to use less CPU?<br class=3D""> >><br class=3D""> >> I will play with cpuset to pin ssh on one core and everything = else on <br class=3D""> >> the other core.<br class=3D""> > <br class=3D""> > It looks like you are running into a CPU bottleneck TBH<br = class=3D""> <br class=3D""> Yes. Pinning on cores with cpuset helps a bit (about +3MiB/s) but <br = class=3D""> without some tweaks on ssh I will not gain more speed :(<br class=3D""> <br class=3D""> Thank you for your help!<br class=3D""> <br class=3D""> Miroslav Lachman<br class=3D""> <br class=3D""> <br class=3D""></blockquote></div></blockquote><div = style=3D"font-family:"Times New Roman"" class=3D""><br = class=3D""></div></div></div><div style=3D"font-family: "Times New = Roman"; font-size: 18px;" class=3D""><div = style=3D"font-family:"Times New Roman"" class=3D""><div = style=3D"font-family:"Times New Roman"" class=3D""><div = style=3D"font-family:"Times New Roman"" class=3D"">You could = pipe the stream through an encrypting program before piping to</div><div = style=3D"font-family:"Times New Roman"" class=3D"">netcat, = then decrypt on the recieving end.</div><div = style=3D"font-family:"Times New Roman"" class=3D""><br = class=3D""></div><div style=3D"font-family:"Times New Roman"" = class=3D"">$ zfs send | crypt | netcat ipaddr 2222</div><div = style=3D"font-family:"Times New Roman"" class=3D"">$ netcat = -vl 2222 | crypt | zfs recv</div><div style=3D"font-family:"Times = New Roman"" class=3D""><br class=3D""></div><div = style=3D"font-family:"Times New Roman"" class=3D"">I don't = know if zfs can handle that, but worth a try.</div></div><div = style=3D"font-family:"Times New Roman"" class=3D""><br = class=3D""></div><div style=3D"font-family:"Times New Roman"" = class=3D"">$ man crypt</div><div style=3D"font-family:"Times New = Roman"" class=3D""><div style=3D"font-family:"Times New = Roman"" class=3D""> The enigma utility, also known as = crypt is a very simple encryption</div><div = style=3D"font-family:"Times New Roman"" class=3D""> = program, working on a =E2=80=9Csecret-key=E2=80=9D = basis. It operates as a filter, i.e.,</div><div = style=3D"font-family:"Times New Roman"" class=3D""> = it encrypts or decrypts a stream of data from standard = input, and writes</div><div style=3D"font-family:"Times New = Roman"" class=3D""> the result to standard = output. Since its operation is fully symmetrical,</div><div = style=3D"font-family:"Times New Roman"" class=3D""> = feeding the encrypted data stream again through the engine = (using the</div><div style=3D"font-family:"Times New Roman"" = class=3D""> same secret key) will decrypt = it.</div></div><div style=3D"font-family:"Times New Roman"" = class=3D""><br class=3D""></div><div style=3D"font-family:"Times = New Roman"" class=3D""><br class=3D""></div></div></div> --=20 Sent with <a href=3D"https://mailfence.com/" target=3D"_blank" = class=3D"">https://mailfence.com</a>; =20 Secure and private email </blockquote></div></div> </div></blockquote></div><br class=3D""></body></html>= --Apple-Mail=_DA2B7E2D-53BE-4B3C-B32E-B84FDDEE8615--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E9E879F2-5A55-4AED-8EE8-F1698CA721F5>