From owner-freebsd-hackers@FreeBSD.ORG Thu May 23 18:04:22 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E833B607 for ; Thu, 23 May 2013 18:04:22 +0000 (UTC) (envelope-from lsanfil@marvell.com) Received: from na3sys009aog131.obsmtp.com (na3sys009aog131.obsmtp.com [74.125.149.247]) by mx1.freebsd.org (Postfix) with ESMTP id B0EB06B6 for ; Thu, 23 May 2013 18:04:22 +0000 (UTC) Received: from SC-OWA01.marvell.com ([199.233.58.136]) (using TLSv1) by na3sys009aob131.postini.com ([74.125.148.12]) with SMTP ID DSNKUZ5aH60FdDaApMLkZfmuJnxRaOan+bzG@postini.com; Thu, 23 May 2013 11:04:22 PDT Received: from SC-VEXCH4.marvell.com ([0000:0000:0000:0000:0000:0000:0.0.0.1]) by SC-OWA01.marvell.com ([10.93.76.21]) with mapi; Thu, 23 May 2013 11:00:21 -0700 From: Lino Sanfilippo To: Hackers freeBSD Date: Thu, 23 May 2013 11:00:16 -0700 Subject: Low Tx-Rx performance with 10Gb NICs Thread-Topic: Low Tx-Rx performance with 10Gb NICs Thread-Index: Ac5X32GSZCFCGAkmTY2F8befgEbjIw== Message-ID: <175CCF5F49938B4D99B2E3EF7F558EBE381FA6E5AA@SC-VEXCH4.marvell.com> Accept-Language: de-DE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-cr-hashedpuzzle: At8Q BVoP Bxxx COw2 C133 C2/D C85l FVks GDzM Gvv3 IQnL IRq5 KBPG KWis K145 LLM/; 1; ZgByAGUAZQBiAHMAZAAtAGgAYQBjAGsAZQByAHMAQABmAHIAZQBlAGIAcwBkAC4AbwByAGcA; Sosha1_v1; 7; {2BF2B9B9-7DE6-4C06-9E37-1A4D0A4BEA20}; bABzAGEAbgBmAGkAbABAAG0AYQByAHYAZQBsAGwALgBjAG8AbQA=; Thu, 23 May 2013 18:00:16 GMT; TABvAHcAIABUAHgALQBSAHgAIABwAGUAcgBmAG8AcgBtAGEAbgBjAGUAIAB3AGkAdABoACAAMQAwAEcAYgAgAE4ASQBDAHMA x-cr-puzzleid: {2BF2B9B9-7DE6-4C06-9E37-1A4D0A4BEA20} acceptlanguage: de-DE, en-US MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Axel Fischer , Ralf Assmann , Markus Althoff X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 May 2013 18:04:23 -0000 Hi all, I am currently doing some performance tests with 10Gb NICs and encounter a = strange behavior in case when I do Rx and Tx at the same time: while the Rx rate stays more = or less stable (almost the same that I see with only Rx traffic) the Tx rate breaks down = drastically. The tests are done with netperf (4 TCP streams for Rx and Tx respectively),= the test machine is an Intel i7 (with HT 8 cores at 3,4 GHz) with 16GB RAM running 32 bit FreeB= SD 9.0 with default system settings. The results are like the following: TX Only: 2290.32 Mb/s Port=3D1001 TX 2357.73 Mb/s Port=3D1002 TX 2340.08 Mb/s Port=3D1003 TX 2382.87 Mb/s Port=3D1004 TX TX Total Result: Mb/s 9371 RX Only: 1257.43 Mb/s Port=3D1001 RX 1901.75 Mb/s Port=3D1002 RX 2605.19 Mb/s Port=3D1003 RX 1986.69 Mb/s Port=3D1004 RX RX Total Result: Mb/s 7751.06 Rx+TX: 251.11 Mb/s Port=3D1001 TX 3069.74 Mb/s Port=3D2001 RX 178.35 Mb/s Port=3D1002 TX 1118.30 Mb/s Port=3D2002 RX 138.05 Mb/s Port=3D1003 TX 1661.22 Mb/s Port=3D2003 RX 129.23 Mb/s Port=3D1004 TX 1851.75 Mb/s Port=3D2004 RX RX+TX Total Result: Mb/s 8397.75 While the results for Tx and Rx are similar to what I get on Linux, the results for Tx+Rx are not. On Linux I see total rates of about 15-18 Gb= /s. I expected to see the same on FreeBSD and I wonder why I do not. The result= s are quite the same for Intel, Broadcom and Marvell NICs, so I suspect the c= ause to be in the protocol stack rather than in the drivers. Is there a known issue concerning high traffic on Tx and Rx paths? Are the= re any system settings I could adjust to get the expected performance? Any hints are very= appreciated. Regards, Lino