Date: Sat, 14 Nov 2015 10:18:29 -0800 From: John-Mark Gurney <jmg@funkthat.com> To: Allan Jude <allanjude@freebsd.org> Cc: freebsd-current@freebsd.org Subject: Re: OpenSSH HPN Message-ID: <20151114181829.GF65715@funkthat.com> In-Reply-To: <56476704.7030800@freebsd.org> References: <86io5a9ome.fsf@desk.des.no> <5643B3EB.1040002@FreeBSD.org> <20151112000651.GH48728@zxy.spb.ru> <5644C937.6030103@freebsd.org> <20151112175603.GZ65715@funkthat.com> <56451953.8070105@freebsd.org> <20151114074754.GE65715@funkthat.com> <56476704.7030800@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Allan Jude wrote this message on Sat, Nov 14, 2015 at 11:53 -0500: > On 2015-11-14 02:47, John-Mark Gurney wrote: > > Allan Jude wrote this message on Thu, Nov 12, 2015 at 17:57 -0500: > >> On 2015-11-12 12:56, John-Mark Gurney wrote: > >>> Allan Jude wrote this message on Thu, Nov 12, 2015 at 12:15 -0500: > >>>> On 2015-11-11 19:06, Slawa Olhovchenkov wrote: > >>>>> On Wed, Nov 11, 2015 at 01:32:27PM -0800, Bryan Drewery wrote: > >>>>> > >>>>>> On 11/10/2015 1:42 AM, Dag-Erling Smørgrav wrote: > >>>>>>> I would also like to remove the NONE cipher > >>>>>>> patch, which is also available in the port (off by default, just like in > >>>>>>> base). > >>>>>> > >>>>>> Fun fact, it's been broken in the port for several months with no > >>>>>> complaints. It was just reported and fixed upstream in the last day and > >>>>>> I wrote in a similar fix in the port. That speaks a lot about its usage > >>>>>> in the port currently. > >>>>> > >>>>> I am try using NPH/NONE with base ssh and confused: don't see > >>>>> performance rise, too complex to enable and too complex for use. > >>>> > >>>> I did a few quick (and dirty) benchmarks and it shows that the NONE > >>>> cipher definitely makes a difference. Version of OpenSSL also seems to > >>>> make a difference, as one might expect. > >>>> > >>>> Note: openssh from ports seems to link against both base and ports > >>>> libcrypto, I am still trying to make sure this isn't corrupting my > >>>> benchmark results. > >>> > >>> You don't need the aesni.ko module loaded for OpenSSL (which is how > >>> OpenSSH uses most crypto algos) to use AES-NI.. > >>> > >>> Also, do you set any sysctl's to play w/ the buffer sizes or anything? > >>> > >>>> I am still debugging my dummynet setup to be able to prove that HPN > >>>> makes a difference (but it does). > >>> > >>> Does my example on the page not work for you? > >>> > >>>> https://wiki.freebsd.org/SSHPerf > >>> > >> > >> I found that when I set even 5ms of delay with dummynet, bandwidth over > >> the LAN drops more than it should. Dummynet is limiting the rate rather > >> than just adding the delay. I am investigating. > >> > >> I found this document: > >> http://www.cs.unc.edu/~jeffay/dirt/FAQ/hstcp-howto.pdf > >> > >> Which is from the 6.x era, but suggests: > >> > >> "One subtle bug exists in the stock Dummynet implementation that should > >> be corrected for experiments. When a packet arrives in dummynet it is > >> shoved into a queue which limits the bandwidth a TCP flow may use. Upon > >> exit from the queue, the packet is transferred to a pipe where it sits > >> for any configured amount of delay time and might possibly be dropped > >> depending on the loss probability. Once the delay time has passed, the > >> packet is released to ip output." > >> > >> May be the cause of my problem > > > > Ahhh, probably need to adjust: > > net.inet.ip.dummynet.pipe_byte_limit: 1048576 > > net.inet.ip.dummynet.pipe_slot_limit: 100 > > > > But even w/ the above limits and 5ms, you should still be able to push > > 200MB/sec... > > I worked with Hiren and some of his dtrace magic and figured out that > dummynet was not my issue. I didn't end up needing to change the > dummynet pipe slot/byte limit in order to get the full 10gig/sec even > with 100ms delay from dummynet. > > You just need to adjust: > > net.inet.tcp.sendbuf_max=BDP > net.inet.tcp.recvbuf_max=BDP > > kern.ipc.maxsockbuf= ( BDP * (2048+256) ) / 2048 > > for a 50ms RTT: > > BDP = 10gbps * .05 = ~60mb I forgot to include _max adjustments on the page (the maxsockbuf was there), but in all of my tests, I can't get close to that.. In my case, I can demonstrate 20MB/sec+ over the link, and w/ a 100ms RTT, that'd be a 2MB buffer size, and even when I increase these to 8MB, and increase kern.ipc.maxsockbuf to 8MB too (otherwise _max is meaningless), I still get 1.5MB/sec, not even close... I do notice w/ nc that it'll slowly increase, and then suddenly back off, just to closely increase again... I wonder if there is some issue w/ TSO or tap that is causing issues... > It can also greatly help to increase: > net.inet.tcp.abc_l_var > > Which is how many additional segments the CWNDs is incremented by each > RTT during slow-start. > > > I am still working on my set of benchmarks to show what different HPN > makes with different RTT values, as well as what might be required to > achieve max throughput for SSH over both LAN and the internet. > > (For my company, we regularly transmit 500GB ZFS datasets over the > public internet on 1gbps or 2x1gbps connections) -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20151114181829.GF65715>