From owner-freebsd-stable@FreeBSD.ORG Thu Apr 9 23:10:14 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2B2DF106566C for ; Thu, 9 Apr 2009 23:10:14 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.229]) by mx1.freebsd.org (Postfix) with ESMTP id F1E5B8FC16 for ; Thu, 9 Apr 2009 23:10:13 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by rv-out-0506.google.com with SMTP id l9so786230rvb.43 for ; Thu, 09 Apr 2009 16:10:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=dNJHbFORKAuqGvUqdcWfXrpORALmHM9FDZEmt/lc2bI=; b=KCfobQuNGWwZjbvyt07+aeBf/hPd8QpyzaUnUO4m/fK3t+n6n5Btkgzs8zFbd2BxbP fSl8btJGwn0LHEidwdgeJUeEjAtu0CECg193fQE30L/DUOJA1JKyyG67/utHNFWpf23E GHqCRda7lye1SAq3O8iBwWQufgODdxktQ1l4A= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=ZPjOz5ARfqqF+rj+/ZLpngSAgzzcxdjQY80a+DttfhhOeAgjRy/RarJU1NtgjBvADj GhlfGf08Oida4QFLvK94rcC0gyn09Oq8qbMOtC8xvclHFQX1lyA3Ps3YErxzfIApEYaO h+RC813oB3gYNY2LRfui03WiHUsJwm42nUB7Q= MIME-Version: 1.0 Received: by 10.140.193.15 with SMTP id q15mr1220932rvf.274.1239316954522; Thu, 09 Apr 2009 15:42:34 -0700 (PDT) In-Reply-To: <49DD2B44.5020808@mawer.org> References: <200904080959.49201.fjwcash@gmail.com> <49DD2B44.5020808@mawer.org> Date: Thu, 9 Apr 2009 15:42:34 -0700 Message-ID: From: Freddie Cash To: Antony Mawer Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org Subject: Re: Network sysctl tuning [was Re: ZFSKnownProblems - needs revision?] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Apr 2009 23:10:14 -0000 On Wed, Apr 8, 2009 at 3:55 PM, Antony Mawer wrote: > Freddie Cash wrote: > ... >> We've also heavily modified /etc/sysctl.conf and upped a bunch of the >> network-related sysctls. =C2=A0Doing so increased our SSH throughput fro= m ~30 >> Mbits/sec across all connections to over 90 Mbits/sec per SSH connection= . > > Are you able to share any of these with the list? It would be useful to > compare as a lot of these tunings people do individually and it would be > good to allow others to test in their environments to see if they help, a= s > well as potentially adding them to the tuning man-page. They're all taken from the HPN-SSH website and various google searches related to HPN-enabled OpenSSH. I don't know exactly what all the different, individual sysctls do, nor whether this is the most optimal setup, but here's the sysctl.conf that we use. This is on 2 systems using a quad-port gigabit NIC where the top two ports are connected via lagg(4) and the bottom two ports are connected via lagg(4), with the two laggX interfaces on separate networks. I did a bunch of scp/sftp transfers of 100 MB files filled with random data pulled from /dev/random between these two boxes tweaking the options one at a time, but didn't do too much in the way of scientific/empirical measurements and comparisons beyond the throughput data that scp/sftp shows. If there are any glaring errors, gotchas, or "why would you ever do that"s, let me know. :) # General network settings net.isr.direct=3D1 # Whether to enable Direct Dispatch for netisr # IP options net.inet.ip.forwarding=3D0 # Whether to enable packet forwarding for NAT/routing net.inet.ip.process_options=3D0 # Disable processing of IP options (nothing uses this field) net.inet.ip.random_id=3D1 # Randomise the IP header ID numb= er net.inet.ip.redirect=3D0 # Whether to allow redirect packe= ts #net.inet.ip.stealth=3D0 # Whether to appear in traceroute= output # ICMP options net.inet.icmp.icmplim=3D200 # Limit ICMP packets to this many per second net.inet.icmp.drop_redirect=3D1 # Drop ICMP redirect packets net.inet.icmp.log_redirect=3D0 # Don't log ICMP redirect packets # TCP options net.inet.tcp.blackhole=3D1 # Drop packets destined to unused= ports net.inet.tcp.inflight.enable=3D0 # Use automatic TCP window-scalin= g net.inet.tcp.log_in_vain=3D0 # Don't log the blackholed packet= s net.inet.tcp.path_mtu_discovery=3D1 # Use ICMP type 3 to find the MTU= to use net.inet.tcp.recvbuf_max=3D16777216 # The max size of the receive buffer (16 MB) net.inet.tcp.recvspace=3D131072 # The initial size in bytes of the receive buffer net.inet.tcp.sack.enable=3D1 # Enable Selective ACKs net.inet.tcp.sendbuf_max=3D16777216 # The max size of the send buffer net.inet.tcp.sendspace=3D131072 # The initial size in bytes of the send buffer net.inet.tcp.syncookies=3D1 # Enable SYN cookie protection net.inet.tcp.rfc1323=3D1 # Enable RFC1323 extensions (TCP window scaling) # UDP options net.inet.udp.blackhole=3D1 # Drop packets destined to unused= ports net.inet.udp.checksum=3D1 # Enable UDP checksums net.inet.udp.log_in_vain=3D0 # Don't log the blackholed packet= s net.inet.udp.recvspace=3D65536 # Size in bytes of the receive bu= ffer # Debug options debug.minidump=3D1 # Disable the small kernel core dump (only mem in use) debug.mpsafevfs=3D1 # Enable threaded VFS subsystem # Kernel options kern.coredump=3D0 # Disable kernel core dumps kern.ipc.maxsockbuf=3D4194304 # Set the max size of the socket buffers (4 MB) kern.ipc.somaxconn=3D1024 # Expand the IP listen queue kern.maxvnodes=3D250000 # Bump up the max number of vnode= s # PCI bus options hw.pci.enable_msix=3D1 # Enable Message Signalled Interrupts - Extended hw.pci.enable_msi=3D1 # Enable Message Signalled Interr= upts hw.pci.enable_io_modes=3D1 # Enable alternate I/O access mod= es --=20 Freddie Cash fjwcash@gmail.com