From owner-freebsd-performance@FreeBSD.ORG Wed Dec 5 02:34:01 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28C5516A417 for ; Wed, 5 Dec 2007 02:34:00 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id B907D13C465 for ; Wed, 5 Dec 2007 02:34:00 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 61E4C46E0B; Tue, 4 Dec 2007 21:20:08 -0500 (EST) Date: Wed, 5 Dec 2007 02:15:06 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Philipp Wuensche In-Reply-To: <4755ED57.6030603@h3q.com> Message-ID: <20071205021327.N41018@fledge.watson.org> References: <4755ED57.6030603@h3q.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "freebsd-performance@freebsd.org" Subject: Re: tuning for high connection rates X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Dec 2007 02:34:01 -0000 On Wed, 5 Dec 2007, Philipp Wuensche wrote: > we are running a FreeBSD 7-BETA4 with SCHED_4BSD on a Intel Core2Dual E6600 > 2.4GHz system for our bittorrent Opentracker. > > The system handles about 20Kpps (18Mbit/s) incoming and 15kpps (22 Mbit/s) > outgoing traffic serving 4000 connections/sec using TCP. The connections are > very short-living, all answered within one packet. > > You can find the system stats at > http://outpost.h3q.com/stalker/munin/opentracker/opentracker.html > > We are now running into some limits at peak time, system is up to 100% and > em0 takes about 80% on one CPU while the Opentracker software only takes > 10-15% CPU. The system is still responsible and answers all the requests, > but we are worried what will happen if the tracker grows at the current > rate. > > Currently we are out of ideas for tuning, so we kindly ask for ideas on > tuning the system to bring down the CPU usage from the em and the system CPU > usage. We tried tuning the em int_delay and abs_int_delay but without > success. Could you show us the output from "top -S" left running for a few minutes in the steady state. Could you try setting the sysctl net.isr.direct to 0, and see how that affects performance, CPU time reports, and "top -S" output? Robert N M Watson Computer Laboratory University of Cambridge > > We have updated to the latest em driver: > > em0: port > 0x4000-0x401f mem 0xe8000000-0xe801ffff irq 16 at device 0.0 on pci13 > em0: Using MSI interrupt > em0: Ethernet address: 00:30:48:92:06:5f > em0: [FILTER] > > The debug output of em0 looks like this: > > em0: CTRL = 0x40140248 RCTL = 0x8002 > em0: Packet buffer = Tx=20k Rx=12k > em0: Flow control watermarks high = 10240 low = 8740 > em0: tx_int_delay = 66, tx_abs_int_delay = 66 > em0: rx_int_delay = 32, rx_abs_int_delay = 66 > em0: fifo workaround = 0, fifo_reset_count = 0 > em0: hw tdh = 183, hw tdt = 183 > em0: hw rdh = 139, hw rdt = 139 > em0: Num Tx descriptors avail = 223 > em0: Tx Descriptors not avail1 = 6225 > em0: Tx Descriptors not avail2 = 3 > em0: Std mbuf failed = 0 > em0: Std mbuf cluster failed = 0 > em0: Driver dropped packets = 0 > em0: Driver tx dma failure in encap = 0 > > We did some tuning already and our current sysctl.conf looks like this: > > kern.ipc.somaxconn=32768 > net.inet.icmp.icmplim=3000 > kern.ipc.maxsockets=300000 > net.inet.tcp.delayed_ack=1 > net.inet.tcp.finwait2_timeout=15000 > net.inet.tcp.fast_finwait2_recycle=1 > net.inet.tcp.maxtcptw=196607 > dev.em.0.rx_processing_limit=-1 > > greetings, > cryx > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >