From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 14:14:23 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B32BC813 for ; Tue, 28 May 2013 14:14:23 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 34BF9E0E for ; Tue, 28 May 2013 14:14:21 +0000 (UTC) Received: (qmail 92833 invoked from network); 28 May 2013 15:12:55 -0000 Received: from unknown (HELO [62.48.0.94]) ([62.48.0.94]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 28 May 2013 15:12:55 -0000 Message-ID: <51A4BBBC.8020405@freebsd.org> Date: Tue, 28 May 2013 16:14:20 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: afischer@marvell.com Subject: Re: Low Tx-Rx performance with 10Gb NICs References: <175CCF5F49938B4D99B2E3EF7F558EBE381FA6E5AA@SC-VEXCH4.marvell.com> <1369406798.20748.30.camel@EL-DT095.site> In-Reply-To: <1369406798.20748.30.camel@EL-DT095.site> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 14:14:23 -0000 On 24.05.2013 16:46, Axel Fischer wrote: > Hi Igor, > > my name is Axel Fischer. working at Marvell SC. Hi Axel, > In addition to your reply to my colleague Lino > Sanfilippo I did some performance measurements > on FreeBSD 9 with a commercial 10 GBit network > card. Which driver? > Unlike on other OS the FreeBSD performance > for duplex rx/tx operation never exceeded the > limit about 9.5 GBit/s. Normally a performance > of at least 16 GBit (up to line speed 20 GBit > in duplex mode) is expected. > As Lino already mentioned the CPU/bus system > (in general the HW) does not set a limit. > Furthermore I noticed that the CPU(s) load > is not very high, about 30 %. Your RX/TX is probably serialized in the driver and you can only make use of one core. > Here is an overview of the measurements: > > netperf rx-tx 4 streams / 60s > > 1768.16 Mb/s Port=2001 RX > 999.33 Mb/s Port=2002 RX > 72.16 Mb/s Port=1001 TX > 61.49 Mb/s Port=1002 TX > 2302.76 Mb/s Port=2003 RX > 73.48 Mb/s Port=1003 TX > 2416.23 Mb/s Port=2004 RX > 76.02 Mb/s Port=1004 TX > ==== > RX+TX Total Result: Mb/s 7769.63 > > > CPU load: > > last pid: 1739; load averages: 0.97, 0.49, 0.21 up 0+00:02:26 > 11:02:52 > 46 processes: 2 running, 44 sleeping > CPU 0: 2.0% user, 0.0% nice, 23.2% system, 0.4% interrupt, 74.4% idle > CPU 1: 1.2% user, 0.0% nice, 19.7% system, 0.4% interrupt, 78.7% idle > CPU 2: 0.0% user, 0.0% nice, 0.0% system, 80.7% interrupt, 19.3% idle > CPU 3: 0.0% user, 0.0% nice, 0.8% system, 1.6% interrupt, 97.6% idle > CPU 4: 2.4% user, 0.0% nice, 25.6% system, 0.0% interrupt, 72.0% idle > CPU 5: 3.1% user, 0.0% nice, 25.6% system, 0.0% interrupt, 71.3% idle > CPU 6: 0.0% user, 0.0% nice, 0.4% system, 32.7% interrupt, 66.9% idle > CPU 7: 0.4% user, 0.0% nice, 1.6% system, 0.0% interrupt, 98.0% idle > Mem: 14M Active, 7548K Inact, 66M Wired, 24K Cache, 16M Buf, 3326M Free > Swap: 4096M Total, 4096M Free > > Additionally I noticed the following TCP errors > with netstat -s ...: > > 1186 data packets (1717328 bytes) retransmitted This may happen and is typically not cause for concern on a loaded system. > 6847875 window update packets Normal. > 2319 duplicate acks Related to the retransmits. > 25831 out-of-order packets (37403288 bytes) This is unusual. What kind of test setup do you have, back-to-back cards or a switch in between? Out of order normally shouldn't happen unless over the internet. > 3733 discarded due to memory problems (drops) > 1186 segment rexmits in SACK recovery episodes > 1717328 byte rexmits in SACK recovery episodes Again related to retransmits. > My questions: > > - What is the max. performance (duplex) on > FreeBSD 9 that you have measured with a 10 GBit > NIC ? > (Expected > 16 GBit/s on appropriate HW) Certain large CDN are known to push more than 20Gbit/s production traffic per machine. Please also my other message from today to hackers@ "Re: preemptive kernel" with Message-ID: <51A4B991.3070805@freebsd.org>. -- Andre > Thank you in advance, > Axel > > > -------- Weitergeleitete Nachricht -------- > Von: Igor Mozolevsky > An: Lino Sanfilippo > Kopie: Hackers freeBSD , Axel Fischer > , Ralf Assmann , Markus > Althoff > Betreff: Re: Low Tx-Rx performance with 10Gb NICs > Datum: Thu, 23 May 2013 11:21:08 -0700 > > On 23 May 2013 19:00, Lino Sanfilippo wrote: > >> Is there a known issue concerning high traffic on Tx and Rx paths? Are there any system >> settings I could adjust to get the expected performance? Any hints are very appreciated. > > check your ierrs and oerrs: netstat -s 1, I've noticed I'm getting > ierrs on em chips, but none on fxp chips (connected to the same > wire/switch); might be unrelated to yours, but worth a check... > > >