Date: Sat, 14 Nov 2015 11:53:24 -0500 From: Allan Jude <allanjude@freebsd.org> To: John-Mark Gurney <jmg@funkthat.com> Cc: freebsd-current@freebsd.org Subject: Re: OpenSSH HPN Message-ID: <56476704.7030800@freebsd.org> In-Reply-To: <20151114074754.GE65715@funkthat.com> References: <86io5a9ome.fsf@desk.des.no> <5643B3EB.1040002@FreeBSD.org> <20151112000651.GH48728@zxy.spb.ru> <5644C937.6030103@freebsd.org> <20151112175603.GZ65715@funkthat.com> <56451953.8070105@freebsd.org> <20151114074754.GE65715@funkthat.com>
index | next in thread | previous in thread | raw e-mail
[-- Attachment #1 --] On 2015-11-14 02:47, John-Mark Gurney wrote: > Allan Jude wrote this message on Thu, Nov 12, 2015 at 17:57 -0500: >> On 2015-11-12 12:56, John-Mark Gurney wrote: >>> Allan Jude wrote this message on Thu, Nov 12, 2015 at 12:15 -0500: >>>> On 2015-11-11 19:06, Slawa Olhovchenkov wrote: >>>>> On Wed, Nov 11, 2015 at 01:32:27PM -0800, Bryan Drewery wrote: >>>>> >>>>>> On 11/10/2015 1:42 AM, Dag-Erling Smørgrav wrote: >>>>>>> I would also like to remove the NONE cipher >>>>>>> patch, which is also available in the port (off by default, just like in >>>>>>> base). >>>>>> >>>>>> Fun fact, it's been broken in the port for several months with no >>>>>> complaints. It was just reported and fixed upstream in the last day and >>>>>> I wrote in a similar fix in the port. That speaks a lot about its usage >>>>>> in the port currently. >>>>> >>>>> I am try using NPH/NONE with base ssh and confused: don't see >>>>> performance rise, too complex to enable and too complex for use. >>>> >>>> I did a few quick (and dirty) benchmarks and it shows that the NONE >>>> cipher definitely makes a difference. Version of OpenSSL also seems to >>>> make a difference, as one might expect. >>>> >>>> Note: openssh from ports seems to link against both base and ports >>>> libcrypto, I am still trying to make sure this isn't corrupting my >>>> benchmark results. >>> >>> You don't need the aesni.ko module loaded for OpenSSL (which is how >>> OpenSSH uses most crypto algos) to use AES-NI.. >>> >>> Also, do you set any sysctl's to play w/ the buffer sizes or anything? >>> >>>> I am still debugging my dummynet setup to be able to prove that HPN >>>> makes a difference (but it does). >>> >>> Does my example on the page not work for you? >>> >>>> https://wiki.freebsd.org/SSHPerf >>> >> >> I found that when I set even 5ms of delay with dummynet, bandwidth over >> the LAN drops more than it should. Dummynet is limiting the rate rather >> than just adding the delay. I am investigating. >> >> I found this document: >> http://www.cs.unc.edu/~jeffay/dirt/FAQ/hstcp-howto.pdf >> >> Which is from the 6.x era, but suggests: >> >> "One subtle bug exists in the stock Dummynet implementation that should >> be corrected for experiments. When a packet arrives in dummynet it is >> shoved into a queue which limits the bandwidth a TCP flow may use. Upon >> exit from the queue, the packet is transferred to a pipe where it sits >> for any configured amount of delay time and might possibly be dropped >> depending on the loss probability. Once the delay time has passed, the >> packet is released to ip output." >> >> May be the cause of my problem > > Ahhh, probably need to adjust: > net.inet.ip.dummynet.pipe_byte_limit: 1048576 > net.inet.ip.dummynet.pipe_slot_limit: 100 > > But even w/ the above limits and 5ms, you should still be able to push > 200MB/sec... > I worked with Hiren and some of his dtrace magic and figured out that dummynet was not my issue. I didn't end up needing to change the dummynet pipe slot/byte limit in order to get the full 10gig/sec even with 100ms delay from dummynet. You just need to adjust: net.inet.tcp.sendbuf_max=BDP net.inet.tcp.recvbuf_max=BDP kern.ipc.maxsockbuf= ( BDP * (2048+256) ) / 2048 for a 50ms RTT: BDP = 10gbps * .05 = ~60mb It can also greatly help to increase: net.inet.tcp.abc_l_var Which is how many additional segments the CWNDs is incremented by each RTT during slow-start. I am still working on my set of benchmarks to show what different HPN makes with different RTT values, as well as what might be required to achieve max throughput for SSH over both LAN and the internet. (For my company, we regularly transmit 500GB ZFS datasets over the public internet on 1gbps or 2x1gbps connections) -- Allan Jude [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJWR2cHAAoJEBmVNT4SmAt+YJ4P/1Vh1G1VmELuoTtb13N4/1qH LyNZZNOL+8qsigItuh0QVPIb7Jhzdhbggh0R5ZiM1u0TStBIL2bICAingIiXouaU IY7YzpoyfIPJhPDLrRyaXkVDnnIRaDJdUFh6bNxyVkB0Xr5JPICS86B7WAL/VlVe 7QwGubTqzx8i16cdnqSHUkKp5fo7cpVSLu9LylkBrWO24BjHZJrS44h82kbikiVC YCLwjSdESBOsNMdrTwkX4xeSgVLn5AC2e0M/sXIIqR3VRuIVLc217qfnV54dZWm8 vFbI/5PX1iTtJAlYcLJVbG4cIKBgp2EQNNBwYbo4efBFc364SQ+9e4dfv2wV3ZNA 3yPfV/CqUPXpjN4sWTcrIIQ4cnT/2/OLBw67dP9zi3G0T8N0kFZMG7zNKHgv5nNL 7Bql8ywjGYYG3y5kitZS6MztNWtRNyd5ZZNsXEtUju6oztGqmP3QSh/16Tn1stqj av3DI96YjVf1dRCg7tLpv4X1J9L7XDpzvdtjiuN5EsX68Aw10GG8XuVeZV5pz9XN BZPlqcmRj2oOLxNHsNxNqAsyg3imBlE/0MgIagibiznDBYvDGcR7d9aL+mLk8wxU aJJnrxjNfiKkBFrat9jlkST0XtiRbLK2ZtLRH+AMpxWE5zagIZDo5hqhZhCFYTiD oOo1DM/zbiwgx/M2a2Wz =f3qp -----END PGP SIGNATURE-----help
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?56476704.7030800>