Date: Wed, 5 Dec 2007 02:15:06 +0000 (GMT) From: Robert Watson <rwatson@FreeBSD.org> To: Philipp Wuensche <cryx-freebsd@h3q.com> Cc: "freebsd-performance@freebsd.org" <freebsd-performance@freebsd.org> Subject: Re: tuning for high connection rates Message-ID: <20071205021327.N41018@fledge.watson.org> In-Reply-To: <4755ED57.6030603@h3q.com> References: <4755ED57.6030603@h3q.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, 5 Dec 2007, Philipp Wuensche wrote: > we are running a FreeBSD 7-BETA4 with SCHED_4BSD on a Intel Core2Dual E6600 > 2.4GHz system for our bittorrent Opentracker. > > The system handles about 20Kpps (18Mbit/s) incoming and 15kpps (22 Mbit/s) > outgoing traffic serving 4000 connections/sec using TCP. The connections are > very short-living, all answered within one packet. > > You can find the system stats at > http://outpost.h3q.com/stalker/munin/opentracker/opentracker.html > > We are now running into some limits at peak time, system is up to 100% and > em0 takes about 80% on one CPU while the Opentracker software only takes > 10-15% CPU. The system is still responsible and answers all the requests, > but we are worried what will happen if the tracker grows at the current > rate. > > Currently we are out of ideas for tuning, so we kindly ask for ideas on > tuning the system to bring down the CPU usage from the em and the system CPU > usage. We tried tuning the em int_delay and abs_int_delay but without > success. Could you show us the output from "top -S" left running for a few minutes in the steady state. Could you try setting the sysctl net.isr.direct to 0, and see how that affects performance, CPU time reports, and "top -S" output? Robert N M Watson Computer Laboratory University of Cambridge > > We have updated to the latest em driver: > > em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port > 0x4000-0x401f mem 0xe8000000-0xe801ffff irq 16 at device 0.0 on pci13 > em0: Using MSI interrupt > em0: Ethernet address: 00:30:48:92:06:5f > em0: [FILTER] > > The debug output of em0 looks like this: > > em0: CTRL = 0x40140248 RCTL = 0x8002 > em0: Packet buffer = Tx=20k Rx=12k > em0: Flow control watermarks high = 10240 low = 8740 > em0: tx_int_delay = 66, tx_abs_int_delay = 66 > em0: rx_int_delay = 32, rx_abs_int_delay = 66 > em0: fifo workaround = 0, fifo_reset_count = 0 > em0: hw tdh = 183, hw tdt = 183 > em0: hw rdh = 139, hw rdt = 139 > em0: Num Tx descriptors avail = 223 > em0: Tx Descriptors not avail1 = 6225 > em0: Tx Descriptors not avail2 = 3 > em0: Std mbuf failed = 0 > em0: Std mbuf cluster failed = 0 > em0: Driver dropped packets = 0 > em0: Driver tx dma failure in encap = 0 > > We did some tuning already and our current sysctl.conf looks like this: > > kern.ipc.somaxconn=32768 > net.inet.icmp.icmplim=3000 > kern.ipc.maxsockets=300000 > net.inet.tcp.delayed_ack=1 > net.inet.tcp.finwait2_timeout=15000 > net.inet.tcp.fast_finwait2_recycle=1 > net.inet.tcp.maxtcptw=196607 > dev.em.0.rx_processing_limit=-1 > > greetings, > cryx > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20071205021327.N41018>