Date: Wed, 05 Dec 2007 01:14:15 +0100 From: Philipp Wuensche <cryx-freebsd@h3q.com> To: "freebsd-performance@freebsd.org" <freebsd-performance@freebsd.org> Subject: tuning for high connection rates Message-ID: <4755ED57.6030603@h3q.com>
next in thread | raw e-mail | index | archive | help
Hi, we are running a FreeBSD 7-BETA4 with SCHED_4BSD on a Intel Core2Dual E6600 2.4GHz system for our bittorrent Opentracker. The system handles about 20Kpps (18Mbit/s) incoming and 15kpps (22 Mbit/s) outgoing traffic serving 4000 connections/sec using TCP. The connections are very short-living, all answered within one packet. You can find the system stats at http://outpost.h3q.com/stalker/munin/opentracker/opentracker.html We are now running into some limits at peak time, system is up to 100% and em0 takes about 80% on one CPU while the Opentracker software only takes 10-15% CPU. The system is still responsible and answers all the requests, but we are worried what will happen if the tracker grows at the current rate. Currently we are out of ideas for tuning, so we kindly ask for ideas on tuning the system to bring down the CPU usage from the em and the system CPU usage. We tried tuning the em int_delay and abs_int_delay but without success. We have updated to the latest em driver: em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port 0x4000-0x401f mem 0xe8000000-0xe801ffff irq 16 at device 0.0 on pci13 em0: Using MSI interrupt em0: Ethernet address: 00:30:48:92:06:5f em0: [FILTER] The debug output of em0 looks like this: em0: CTRL = 0x40140248 RCTL = 0x8002 em0: Packet buffer = Tx=20k Rx=12k em0: Flow control watermarks high = 10240 low = 8740 em0: tx_int_delay = 66, tx_abs_int_delay = 66 em0: rx_int_delay = 32, rx_abs_int_delay = 66 em0: fifo workaround = 0, fifo_reset_count = 0 em0: hw tdh = 183, hw tdt = 183 em0: hw rdh = 139, hw rdt = 139 em0: Num Tx descriptors avail = 223 em0: Tx Descriptors not avail1 = 6225 em0: Tx Descriptors not avail2 = 3 em0: Std mbuf failed = 0 em0: Std mbuf cluster failed = 0 em0: Driver dropped packets = 0 em0: Driver tx dma failure in encap = 0 We did some tuning already and our current sysctl.conf looks like this: kern.ipc.somaxconn=32768 net.inet.icmp.icmplim=3000 kern.ipc.maxsockets=300000 net.inet.tcp.delayed_ack=1 net.inet.tcp.finwait2_timeout=15000 net.inet.tcp.fast_finwait2_recycle=1 net.inet.tcp.maxtcptw=196607 dev.em.0.rx_processing_limit=-1 greetings, cryx
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4755ED57.6030603>