From owner-freebsd-performance@FreeBSD.ORG Wed Dec 5 22:19:57 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9A21B16A417 for ; Wed, 5 Dec 2007 22:19:57 +0000 (UTC) (envelope-from cryx-freebsd@h3q.com) Received: from mail.h3q.com (mail.h3q.com [213.73.89.199]) by mx1.freebsd.org (Postfix) with ESMTP id 13DE113C447 for ; Wed, 5 Dec 2007 22:19:56 +0000 (UTC) (envelope-from cryx-freebsd@h3q.com) Received: (qmail 53582 invoked from network); 5 Dec 2007 22:19:55 -0000 Received: from unknown (HELO goa.local) (smtpsend@85.179.28.51) by mail.h3q.com with AES256-SHA encrypted SMTP; 5 Dec 2007 22:19:55 -0000 Message-ID: <475723FB.4020304@h3q.com> Date: Wed, 05 Dec 2007 23:19:39 +0100 From: Philipp Wuensche User-Agent: Thunderbird 2.0.0.9 (Macintosh/20071031) MIME-Version: 1.0 To: Mike Tancsa References: <4755ED57.6030603@h3q.com> <200712051616.lB5GGGvb029587@lava.sentex.ca> <4756DE7E.1000104@h3q.com> <200712051729.lB5HTAkn029896@lava.sentex.ca> In-Reply-To: <200712051729.lB5HTAkn029896@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: tuning for high connection rates X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Dec 2007 22:19:57 -0000 Mike Tancsa wrote: > At 12:23 PM 12/5/2007, Philipp Wuensche wrote: >> Mike Tancsa wrote: >> > At 07:14 PM 12/4/2007, Philipp Wuensche wrote: >> > >> >> The debug output of em0 looks like this: >> >> >> >> em0: CTRL = 0x40140248 RCTL = 0x8002 >> >> em0: Packet buffer = Tx=20k Rx=12k >> >> em0: Flow control watermarks high = 10240 low = 8740 >> >> em0: tx_int_delay = 66, tx_abs_int_delay = 66 >> >> em0: rx_int_delay = 32, rx_abs_int_delay = 66 >> >> em0: fifo workaround = 0, fifo_reset_count = 0 >> >> em0: hw tdh = 183, hw tdt = 183 >> >> em0: hw rdh = 139, hw rdt = 139 >> >> em0: Num Tx descriptors avail = 223 >> >> em0: Tx Descriptors not avail1 = 6225 >> >> em0: Tx Descriptors not avail2 = 3 >> >> em0: Std mbuf failed = 0 >> >> em0: Std mbuf cluster failed = 0 >> >> em0: Driver dropped packets = 0 >> >> em0: Driver tx dma failure in encap = 0 >> > >> > If you do a >> > sysctl -w dev.em.0.stats=1 >> > >> > It will spit the nic stats to syslog. What are the results ? >> >> em0: Excessive collisions = 0 >> em0: Sequence errors = 0 >> em0: Defer count = 0 >> em0: Missed Packets = 12876719 >> em0: Receive No Buffers = 5950326 >> em0: Receive Length Errors = 0 >> em0: Receive errors = 0 >> em0: Crc errors = 0 >> em0: Alignment errors = 0 >> em0: Collision/Carrier extension errors = 0 >> em0: RX overruns = 56256 >> em0: watchdog timeouts = 0 >> em0: XON Rcvd = 0 >> em0: XON Xmtd = 0 >> em0: XOFF Rcvd = 0 >> em0: XOFF Xmtd = 0 >> em0: Good Packets Rcvd = 3384408375 >> em0: Good Packets Xmtd = 2657550034 >> em0: TSO Contexts Xmtd = 6925441 >> em0: TSO Contexts Failed = 0 >> >> >> > Also, >> > what does ifconfig em0 look like (i.e. what options do you have set, >> > speed etc) >> >> Its on a 100Mbit/s switch, we haven't changed the options: >> >> em0: flags=8843 metric 0 mtu 1500 >> >> options=19b >> media: Ethernet autoselect (100baseTX ) > > Some people have reported that TSO is a "bad thing" on 100Mb. Can you > try disabling that ? Also, you seem to have a lot of RX overruns and > missed packets such that the nic cannot process things fast enough. I > havent done any benchmarks yet, but the Yandex people claim their > modified EM driver can handle higher PPS rates than the stock em > driver. Not sure if they have a RELENG_7 port or not but they might > have some insight. After switching to net.isr.direct=0 and 346609775 good packets later, RX overruns haven't increased by one! Thats nice. Still interrupt is using up the CPU. I'm not quite sure if polling would help now!? current stats: em0: Excessive collisions = 0 em0: Sequence errors = 0 em0: Defer count = 0 em0: Missed Packets = 12885592 em0: Receive No Buffers = 5953357 em0: Receive Length Errors = 0 em0: Receive errors = 0 em0: Crc errors = 0 em0: Alignment errors = 0 em0: Collision/Carrier extension errors = 0 em0: RX overruns = 56296 em0: watchdog timeouts = 0 em0: XON Rcvd = 0 em0: XON Xmtd = 0 em0: XOFF Rcvd = 0 em0: XOFF Xmtd = 0 em0: Good Packets Rcvd = 3731018150 em0: Good Packets Xmtd = 2936659631 em0: TSO Contexts Xmtd = 7734211 em0: TSO Contexts Failed = 0 We will try disabling TSO to see if anything changes. greetings, cryx