Date: Wed, 22 Feb 2023 22:03:44 +0100 From: Miroslav Lachman <000.fbsd@quip.cz> To: mike tancsa <mike@sentex.net>, Alan Somers <asomers@freebsd.org> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: speeding up zfs send | recv (update) Message-ID: <1031e2b0-b245-1dc6-a499-8f4da3796543@quip.cz> In-Reply-To: <c229a502-fb76-ec6a-a56b-934d3b56e474@sentex.net> References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <CAOtMX2gifUmgqwSKpRGcfzCm_=BX_szNF1AF8WTMfAmbrJ5UWA@mail.gmail.com> <f6ea3387-faf8-4c63-d1e7-906fa397b00b@sentex.net> <a38578c6-b633-249d-90f0-0652377d76c0@quip.cz> <c229a502-fb76-ec6a-a56b-934d3b56e474@sentex.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On 22/02/2023 21:13, mike tancsa wrote: > aes128-gcm@openssh.com is what I settled on for the cipher. If you > just blast dd if=/dev/zero | ssh are you able to achieve close to > wirespeed ? As (I think) I mentioned in this thread, different zfs > datasets transmit at different speeds. Ones with tens of thousands of > small files are much slower than those with a few multi gigabit files. > The disk seems to be the limiting factor for me as graphing "time spent > in IO" via telegraf, shows the disks pretty well 100% busy. Yes, I read about it. Which is also interesting as I thought when zfs sends the whole filesystem it will not walk file by file as will rsync do. > dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c > aes128-gcm@openssh.com mdtancsa@coldstore "cat - > /dev/null" > 20650655744 bytes (21 GB, 19 GiB) transferred 18.001s, 1147 MB/s > dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c > chacha20-poly1305@openssh.com mdtancsa@coldstore "cat - > /dev/null" > 20481835008 bytes (20 GB, 19 GiB) transferred 43.000s, 476 MB/s > dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c aes128-ctr > mdtancsa@coldstore "cat - > /dev/null" > 20781727744 bytes (21 GB, 19 GiB) transferred 29.001s, 717 MB/s > Interresting numbers. I think I am the only one who get best speed with chacha20-poly1305@openssh.com # dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c aes128-gcm@openssh.com quip@aa.bb.cc.dd "cat - > /dev/null" 2074083328 bytes (2074 MB, 1978 MiB) transferred 50.003s, 41 MB/s # dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c chacha20-poly1305@openssh.com quip@aa.bb.cc.dd "cat - > /dev/null" 2069889024 bytes (2070 MB, 1974 MiB) transferred 33.006s, 63 MB/s # dd if=/dev/zero count=2000 bs=1m status=progress | ssh -c aes128-ctr quip@aa.bb.cc.dd "cat - > /dev/null" 2091909120 bytes (2092 MB, 1995 MiB) transferred 45.007s, 46 MB/s It seems the speed of SSH is limited by single core performance which is very poor on this machine (Intel(R) Pentium(R) Dual CPU E2160). Even if CPU has 50% idle, ssh runs on 99.8% of single core. I know there were some HPN patches to ssh, beside that is there any option I can try to use less CPU? I will play with cpuset to pin ssh on one core and everything else on the other core. Kind regards Miroslav Lachman
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1031e2b0-b245-1dc6-a499-8f4da3796543>