Date: Wed, 22 Feb 2023 15:13:55 -0500 From: mike tancsa <mike@sentex.net> To: Miroslav Lachman <000.fbsd@quip.cz>, Alan Somers <asomers@freebsd.org> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: speeding up zfs send | recv (update) Message-ID: <c229a502-fb76-ec6a-a56b-934d3b56e474@sentex.net> In-Reply-To: <a38578c6-b633-249d-90f0-0652377d76c0@quip.cz> References: <866d6937-a4e8-bec3-d61b-07df3065fca9@sentex.net> <CAOtMX2gifUmgqwSKpRGcfzCm_=BX_szNF1AF8WTMfAmbrJ5UWA@mail.gmail.com> <f6ea3387-faf8-4c63-d1e7-906fa397b00b@sentex.net> <a38578c6-b633-249d-90f0-0652377d76c0@quip.cz>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2/22/2023 1:06 PM, Miroslav Lachman wrote: > > I am facing similar problem with low performance of zfs send over > network. I have 2 machines in two different datacenters, both have > 1Gbps NIC and I would like to saturate the network but it seems > impossible even if "everything" seem to have enough unused resources. > The sending side is very old HP ML110 G5 with bge0 NIC, receiving side > is VM with enough CPU, RAM, 22TB storage and vtnet0 NIC. > Sender has about 25% CPU idle during sending, disks are not saturated > according to iostat -w -x but I still cannot see more than 52MiB/s. > Everything about zfs snapshot, send, receive etc. is handled by > syncoid from sanoid package (ssh + mbuffer + pv, no lzop) > > I thought it is slow because of ssh ciphers so I tried to change with > --sshciper but it was 10MiB/s slower when I changed from default > chacha20-poly1305@openssh.com to aes128-ctr aes128-gcm@openssh.com is what I settled on for the cipher. If you just blast dd if=/dev/zero | ssh are you able to achieve close to wirespeed ? As (I think) I mentioned in this thread, different zfs datasets transmit at different speeds. Ones with tens of thousands of small files are much slower than those with a few multi gigabit files. The disk seems to be the limiting factor for me as graphing "time spent in IO" via telegraf, shows the disks pretty well 100% busy. e.g. dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c aes128-gcm@openssh.com mdtancsa@coldstore "cat - > /dev/null" 20650655744 bytes (21 GB, 19 GiB) transferred 18.001s, 1147 MB/s 20000+0 records in 19982+36 records out 20971520000 bytes transferred in 18.259553 secs (1148523199 bytes/sec) vs dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c chacha20-poly1305@openssh.com mdtancsa@coldstore "cat - > /dev/null" 20481835008 bytes (20 GB, 19 GiB) transferred 43.000s, 476 MB/s 20000+0 records in 19961+78 records out 20971520000 bytes transferred in 43.947239 secs (477197671 bytes/sec) dd if=/dev/zero count=20000 bs=1m status=progress | ssh -c aes128-ctr mdtancsa@coldstore "cat - > /dev/null" 20781727744 bytes (21 GB, 19 GiB) transferred 29.001s, 717 MB/s 20000+0 records in 19973+54 records out 20971520000 bytes transferred in 29.263111 secs (716653818 bytes/sec) ---Mike
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c229a502-fb76-ec6a-a56b-934d3b56e474>