From owner-freebsd-performance@FreeBSD.ORG Wed Dec 5 17:29:12 2007 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B0BB316A417 for ; Wed, 5 Dec 2007 17:29:12 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 5BA9713C459 for ; Wed, 5 Dec 2007 17:29:12 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.13.8/8.13.8) with ESMTP id lB5HTBr3082929; Wed, 5 Dec 2007 12:29:11 -0500 (EST) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id lB5HTAkn029896 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 5 Dec 2007 12:29:10 -0500 (EST) (envelope-from mike@sentex.net) Message-Id: <200712051729.lB5HTAkn029896@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Wed, 05 Dec 2007 12:31:12 -0500 To: Philipp Wuensche From: Mike Tancsa In-Reply-To: <4756DE7E.1000104@h3q.com> References: <4755ED57.6030603@h3q.com> <200712051616.lB5GGGvb029587@lava.sentex.ca> <4756DE7E.1000104@h3q.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Cc: freebsd-performance@freebsd.org Subject: Re: tuning for high connection rates X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Dec 2007 17:29:12 -0000 At 12:23 PM 12/5/2007, Philipp Wuensche wrote: >Mike Tancsa wrote: > > At 07:14 PM 12/4/2007, Philipp Wuensche wrote: > > > >> The debug output of em0 looks like this: > >> > >> em0: CTRL = 0x40140248 RCTL = 0x8002 > >> em0: Packet buffer = Tx=20k Rx=12k > >> em0: Flow control watermarks high = 10240 low = 8740 > >> em0: tx_int_delay = 66, tx_abs_int_delay = 66 > >> em0: rx_int_delay = 32, rx_abs_int_delay = 66 > >> em0: fifo workaround = 0, fifo_reset_count = 0 > >> em0: hw tdh = 183, hw tdt = 183 > >> em0: hw rdh = 139, hw rdt = 139 > >> em0: Num Tx descriptors avail = 223 > >> em0: Tx Descriptors not avail1 = 6225 > >> em0: Tx Descriptors not avail2 = 3 > >> em0: Std mbuf failed = 0 > >> em0: Std mbuf cluster failed = 0 > >> em0: Driver dropped packets = 0 > >> em0: Driver tx dma failure in encap = 0 > > > > If you do a > > sysctl -w dev.em.0.stats=1 > > > > It will spit the nic stats to syslog. What are the results ? > >em0: Excessive collisions = 0 >em0: Sequence errors = 0 >em0: Defer count = 0 >em0: Missed Packets = 12876719 >em0: Receive No Buffers = 5950326 >em0: Receive Length Errors = 0 >em0: Receive errors = 0 >em0: Crc errors = 0 >em0: Alignment errors = 0 >em0: Collision/Carrier extension errors = 0 >em0: RX overruns = 56256 >em0: watchdog timeouts = 0 >em0: XON Rcvd = 0 >em0: XON Xmtd = 0 >em0: XOFF Rcvd = 0 >em0: XOFF Xmtd = 0 >em0: Good Packets Rcvd = 3384408375 >em0: Good Packets Xmtd = 2657550034 >em0: TSO Contexts Xmtd = 6925441 >em0: TSO Contexts Failed = 0 > > > > Also, > > what does ifconfig em0 look like (i.e. what options do you have set, > > speed etc) > >Its on a 100Mbit/s switch, we haven't changed the options: > >em0: flags=8843 metric 0 mtu 1500 > options=19b > media: Ethernet autoselect (100baseTX ) Some people have reported that TSO is a "bad thing" on 100Mb. Can you try disabling that ? Also, you seem to have a lot of RX overruns and missed packets such that the nic cannot process things fast enough. I havent done any benchmarks yet, but the Yandex people claim their modified EM driver can handle higher PPS rates than the stock em driver. Not sure if they have a RELENG_7 port or not but they might have some insight. ---Mike