From owner-freebsd-stable@FreeBSD.ORG Sun Apr 17 23:15:48 2005 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3BFED16A4CE for ; Sun, 17 Apr 2005 23:15:48 +0000 (GMT) Received: from mailhost.xciv.org (vantage.xciv.org [213.228.237.55]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6AEAB43D31 for ; Sun, 17 Apr 2005 23:15:47 +0000 (GMT) (envelope-from paul@xciv.org) Received: from localhost ([127.0.0.1] helo=xciv.org) by mailhost.xciv.org with esmtp id 1DNIzN-000Hkn-00; Mon, 18 Apr 2005 00:15:41 +0100 X-Mailer: exmh version 2.6.3 04/04/2003 with nmh-1.0.4 To: freebsd-stable@freebsd.org Organization: XCIV, London UK Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Mon, 18 Apr 2005 00:15:41 +0100 Message-ID: <68248.1113779741@xciv.org> From: Paul Civati X-XCIV-MailScanner: Found to be clean Subject: Tuning for router performance X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: freebsd-stable@freebsd.org List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Apr 2005 23:15:48 -0000 I'm trying to tune 5.x for maximum router performance, in terms of packets per second. I'm starting to hit errors at 150Kpps. input (em0) output packets errs bytes packets errs bytes colls 1 0 66 1 0 178 0 75751 3391 5124070 1 0 298 0 150150 0 9609602 1 0 178 0 150150 19837 10809670 1 0 186 0 150150 0 9609602 1 0 178 0 150150 19840 10809466 1 0 186 0 150150 0 9609602 1 0 178 0 150150 19842 10809466 1 0 186 0 146029 0 9345858 1 0 178 0 141518 32596 10206142 1 0 186 0 Set-up: RELENG_5 as of Apr 03 21:33 * P4 3.0GHz / 2GB DDR400 RAM * Supermicro P4SCi with 2 on-board Intel gige * Additional 2-port Intel gige on 64-bit PCI-X With this many interfaces I have disabled unnecessary devices in the BIOS such as USB to reduce IRQ sharing. APIC also seems to do quite some renumbering of IRQs, no idea if this is a problem or not. Kernel config additions over GENERIC: options DEVICE_POLLING #network device mixed interrupt-polling handling options HZ=1000 #for polling options TCP_SIGNATURE #TCP MD5 signatures options FAST_IPSEC #IPSec needed for TCP MD5 options IPFILTER #ipfilter support options IPFILTER_LOG #ipfilter logging device crypto # crypto needed for TCP MD5 device vlan # 802.1q ethernet VLAN tagging device carp # common address resolution protocol /etc/sysctl.conf: kern.polling.enable=1 net.inet.ip.forwarding=1 net.inet.ip.fastforwarding=1 Empty ipfilter ruleset. [ Full dmesg and kernel config etc at http://www.rackred.com/temp/ ] My test set-up is to use a Linux based click-router with udpgen on one side, and a Linux based click-router with udpcount on the other side. [ SRC host ] ---- [em0 router under test em3] ---- [ DST host ] So traffic test is passing data through the router. Information gleaned from the archives suggests tweaking sysctl's such as net.isr.enable and disabling harvest options but these haven't really made any noticeable difference for me. Does anyone have any suggestions? I'm sure I have read of other people getting better performance than this. -Paul- -- Paul Civati Rack Sense Ltd - Managed/Business hosting - www.racksense.com RackRed - Value SSL certificates and servers - www.rackred.com