Date: Fri, 10 Apr 2009 14:23:23 -0700 From: Kip Macy <kmacy@freebsd.org> To: Freddie Cash <fjwcash@gmail.com> Cc: freebsd-stable@freebsd.org Subject: Re: Network sysctl tuning [was Re: ZFSKnownProblems - needs revision?] Message-ID: <3c1674c90904101423v68d1bb42q5b4ab510cc4cda1d@mail.gmail.com> In-Reply-To: <b269bc570904091542gce3c7a1v40afa0bb8e5ecb9e@mail.gmail.com> References: <gri5d9$15l$1@ger.gmane.org> <200904080959.49201.fjwcash@gmail.com> <49DD2B44.5020808@mawer.org> <b269bc570904091542gce3c7a1v40afa0bb8e5ecb9e@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I think most if not all of the gains are from increasing the maximum tcp socket buffer sizes. You might test it out with only those to confirm. -Kip On Thu, Apr 9, 2009 at 3:42 PM, Freddie Cash <fjwcash@gmail.com> wrote: > On Wed, Apr 8, 2009 at 3:55 PM, Antony Mawer <fbsd-stable@mawer.org> wrot= e: >> Freddie Cash wrote: >> ... >>> We've also heavily modified /etc/sysctl.conf and upped a bunch of the >>> network-related sysctls. =A0Doing so increased our SSH throughput from = ~30 >>> Mbits/sec across all connections to over 90 Mbits/sec per SSH connectio= n. >> >> Are you able to share any of these with the list? It would be useful to >> compare as a lot of these tunings people do individually and it would be >> good to allow others to test in their environments to see if they help, = as >> well as potentially adding them to the tuning man-page. > > They're all taken from the HPN-SSH website and various google searches > related to HPN-enabled OpenSSH. > > I don't know exactly what all the different, individual sysctls do, > nor whether this is the most optimal setup, but here's the sysctl.conf > that we use. =A0This is on 2 systems using a quad-port gigabit NIC where > the top two ports are connected via lagg(4) and the bottom two ports > are connected via lagg(4), with the two laggX interfaces on separate > networks. > > I did a bunch of scp/sftp transfers of 100 MB files filled with random > data pulled from /dev/random between these two boxes tweaking the > options one at a time, but didn't do too much in the way of > scientific/empirical measurements and comparisons beyond the > throughput data that scp/sftp shows. > > If there are any glaring errors, gotchas, or "why would you ever do > that"s, let me know. =A0:) > > # General network settings > net.isr.direct=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Wheth= er to enable Direct > Dispatch for netisr > > > # IP options > net.inet.ip.forwarding=3D0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Whether to en= able packet > forwarding for NAT/routing > net.inet.ip.process_options=3D0 =A0 =A0 =A0 =A0 =A0 # Disable processing = of IP > options (nothing uses this field) > net.inet.ip.random_id=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # Randomise the= IP header ID number > net.inet.ip.redirect=3D0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Whether to = allow redirect packets > #net.inet.ip.stealth=3D0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Whether to = appear in traceroute output > > > # ICMP options > net.inet.icmp.icmplim=3D200 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # Limit ICMP pack= ets to this > many per second > net.inet.icmp.drop_redirect=3D1 =A0 =A0 =A0 =A0 =A0 # Drop ICMP redirect = packets > net.inet.icmp.log_redirect=3D0 =A0 =A0 =A0 =A0 =A0 =A0# Don't log ICMP re= direct packets > > > # TCP options > net.inet.tcp.blackhole=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Drop packets = destined to unused ports > net.inet.tcp.inflight.enable=3D0 =A0 =A0 =A0 =A0 =A0# Use automatic TCP w= indow-scaling > net.inet.tcp.log_in_vain=3D0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Don't log the b= lackholed packets > net.inet.tcp.path_mtu_discovery=3D1 =A0 =A0 =A0 # Use ICMP type 3 to find= the MTU to use > net.inet.tcp.recvbuf_max=3D16777216 =A0 =A0 =A0 # The max size of the rec= eive > buffer (16 MB) > net.inet.tcp.recvspace=3D131072 =A0 =A0 =A0 =A0 =A0 # The initial size in= bytes of > the receive buffer > net.inet.tcp.sack.enable=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Enable Selectiv= e ACKs > net.inet.tcp.sendbuf_max=3D16777216 =A0 =A0 =A0 # The max size of the sen= d buffer > net.inet.tcp.sendspace=3D131072 =A0 =A0 =A0 =A0 =A0 # The initial size in= bytes of > the send buffer > net.inet.tcp.syncookies=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # Enable SYN cook= ie protection > net.inet.tcp.rfc1323=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Enable RFC1= 323 extensions > (TCP window scaling) > > > # UDP options > net.inet.udp.blackhole=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Drop packets = destined to unused ports > net.inet.udp.checksum=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # Enable UDP ch= ecksums > net.inet.udp.log_in_vain=3D0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Don't log the b= lackholed packets > net.inet.udp.recvspace=3D65536 =A0 =A0 =A0 =A0 =A0 =A0# Size in bytes of = the receive buffer > > > # Debug options > debug.minidump=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Disab= le the small kernel > core dump (only mem in use) > debug.mpsafevfs=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # Enable = threaded VFS subsystem > > > # Kernel options > kern.coredump=3D0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # Disab= le kernel core dumps > kern.ipc.maxsockbuf=3D4194304 =A0 =A0 =A0 =A0 =A0 =A0 # Set the max size = of the > socket buffers (4 MB) > kern.ipc.somaxconn=3D1024 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # Expand the IP= listen queue > kern.maxvnodes=3D250000 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # Bump up the= max number of vnodes > > > # PCI bus options > hw.pci.enable_msix=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Enable Me= ssage Signalled > Interrupts - Extended > hw.pci.enable_msi=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 # Enable Me= ssage Signalled Interrupts > hw.pci.enable_io_modes=3D1 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0# Enable altern= ate I/O access modes > > -- > Freddie Cash > fjwcash@gmail.com > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > --=20 All that is necessary for the triumph of evil is that good men do nothing. Edmund Burke
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?3c1674c90904101423v68d1bb42q5b4ab510cc4cda1d>