From owner-freebsd-hackers@FreeBSD.ORG Fri May 24 14:47:38 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6E3BE5E0 for ; Fri, 24 May 2013 14:47:38 +0000 (UTC) (envelope-from afischer@marvell.com) Received: from na3sys009aog117.obsmtp.com (na3sys009aog117.obsmtp.com [74.125.149.242]) by mx1.freebsd.org (Postfix) with ESMTP id 3CE776E5 for ; Fri, 24 May 2013 14:47:38 +0000 (UTC) Received: from SC-OWA01.marvell.com ([199.233.58.136]) (using TLSv1) by na3sys009aob117.postini.com ([74.125.148.12]) with SMTP ID DSNKUZ99guz85HL7JFM4HQkgBAEo4ZIXZTNO@postini.com; Fri, 24 May 2013 07:47:38 PDT Received: from maili.marvell.com (10.93.76.43) by sc-owa01.marvell.com (10.93.76.21) with Microsoft SMTP Server id 8.3.213.0; Fri, 24 May 2013 07:46:45 -0700 Received: from [10.9.2.76] (unknown [10.9.2.76]) by maili.marvell.com (Postfix) with ESMTP id 0B2091CCDAE; Fri, 24 May 2013 07:46:43 -0700 (PDT) Subject: Re: Low Tx-Rx performance with 10Gb NICs From: Axel Fischer To: Igor Mozolevsky In-Reply-To: References: <175CCF5F49938B4D99B2E3EF7F558EBE381FA6E5AA@SC-VEXCH4.marvell.com> Content-Type: text/plain; charset="UTF-8" Date: Fri, 24 May 2013 16:46:38 +0200 Message-ID: <1369406798.20748.30.camel@EL-DT095.site> MIME-Version: 1.0 X-Mailer: Evolution 2.32.1 Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Fri, 24 May 2013 16:27:59 +0000 Cc: Hackers freeBSD , Ralf Assmann , Lino Sanfilippo , Markus Althoff X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 May 2013 14:47:38 -0000 Hi Igor, my name is Axel Fischer. working at Marvell SC. In addition to your reply to my colleague Lino Sanfilippo I did some performance measurements on FreeBSD 9 with a commercial 10 GBit network card. Unlike on other OS the FreeBSD performance for duplex rx/tx operation never exceeded the limit about 9.5 GBit/s. Normally a performance of at least 16 GBit (up to line speed 20 GBit in duplex mode) is expected. As Lino already mentioned the CPU/bus system (in general the HW) does not set a limit. Furthermore I noticed that the CPU(s) load is not very high, about 30 %. Here is an overview of the measurements: netperf rx-tx 4 streams / 60s 1768.16 Mb/s Port=2001 RX 999.33 Mb/s Port=2002 RX 72.16 Mb/s Port=1001 TX 61.49 Mb/s Port=1002 TX 2302.76 Mb/s Port=2003 RX 73.48 Mb/s Port=1003 TX 2416.23 Mb/s Port=2004 RX 76.02 Mb/s Port=1004 TX ==== > RX+TX Total Result: Mb/s 7769.63 CPU load: last pid: 1739; load averages: 0.97, 0.49, 0.21 up 0+00:02:26 11:02:52 46 processes: 2 running, 44 sleeping CPU 0: 2.0% user, 0.0% nice, 23.2% system, 0.4% interrupt, 74.4% idle CPU 1: 1.2% user, 0.0% nice, 19.7% system, 0.4% interrupt, 78.7% idle CPU 2: 0.0% user, 0.0% nice, 0.0% system, 80.7% interrupt, 19.3% idle CPU 3: 0.0% user, 0.0% nice, 0.8% system, 1.6% interrupt, 97.6% idle CPU 4: 2.4% user, 0.0% nice, 25.6% system, 0.0% interrupt, 72.0% idle CPU 5: 3.1% user, 0.0% nice, 25.6% system, 0.0% interrupt, 71.3% idle CPU 6: 0.0% user, 0.0% nice, 0.4% system, 32.7% interrupt, 66.9% idle CPU 7: 0.4% user, 0.0% nice, 1.6% system, 0.0% interrupt, 98.0% idle Mem: 14M Active, 7548K Inact, 66M Wired, 24K Cache, 16M Buf, 3326M Free Swap: 4096M Total, 4096M Free Additionally I noticed the following TCP errors with netstat -s ...: 1186 data packets (1717328 bytes) retransmitted 6847875 window update packets 2319 duplicate acks 25831 out-of-order packets (37403288 bytes) 3733 discarded due to memory problems (drops) 1186 segment rexmits in SACK recovery episodes 1717328 byte rexmits in SACK recovery episodes My questions: - What is the max. performance (duplex) on FreeBSD 9 that you have measured with a 10 GBit NIC ? (Expected > 16 GBit/s on appropriate HW) Thank you in advance, Axel -------- Weitergeleitete Nachricht -------- Von: Igor Mozolevsky An: Lino Sanfilippo Kopie: Hackers freeBSD , Axel Fischer , Ralf Assmann , Markus Althoff Betreff: Re: Low Tx-Rx performance with 10Gb NICs Datum: Thu, 23 May 2013 11:21:08 -0700 On 23 May 2013 19:00, Lino Sanfilippo wrote: > Is there a known issue concerning high traffic on Tx and Rx paths? Are there any system > settings I could adjust to get the expected performance? Any hints are very appreciated. check your ierrs and oerrs: netstat -s 1, I've noticed I'm getting ierrs on em chips, but none on fxp chips (connected to the same wire/switch); might be unrelated to yours, but worth a check... -- Axel Fischer | R&D Software / SW Engineer | Marvell Semiconductor Germany GmbH Office +49 (7243) 502 370 | Fax +49 (7243) 502 982 afischer@marvell.com M A R V E L L |www.marvell.com This communication, together with any attachments hereto or links contained herein, is for the sole use of the intended recipient(s) and may contain information that is confidential or legally protected. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, dissemination, distribution or use of this communication is STRICTLY PROHIBITED. If you have received this communication in error, please notify the sender immediately by return e-mail message and delete the original and all copies of the communication, along with any attachments hereto or links herein, from your system. Marvell Semiconductor Germany GmbH, Siemensstr. 23, 76275 Ettlingen, Amtsgericht Mannheim HRB 361620 Geschäftsführer: Dipl.-Volksw. Mathias Horak