From owner-freebsd-net@FreeBSD.ORG Thu May 28 17:49:40 2015 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 81878BB1; Thu, 28 May 2015 17:49:40 +0000 (UTC) (envelope-from flewis@panasas.com) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1on0066.outbound.protection.outlook.com [157.56.110.66]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E520CA6; Thu, 28 May 2015 17:49:38 +0000 (UTC) (envelope-from flewis@panasas.com) Received: from BY2PR08MB1797.namprd08.prod.outlook.com (25.163.46.23) by BY2PR08MB157.namprd08.prod.outlook.com (10.242.39.16) with Microsoft SMTP Server (TLS) id 15.1.172.22; Thu, 28 May 2015 17:49:29 +0000 Received: from CY1PR0801MB1563.namprd08.prod.outlook.com (25.163.143.139) by BY2PR08MB1797.namprd08.prod.outlook.com (25.163.46.23) with Microsoft SMTP Server (TLS) id 15.1.172.22; Thu, 28 May 2015 17:49:25 +0000 Received: from CY1PR0801MB1563.namprd08.prod.outlook.com ([25.163.143.139]) by CY1PR0801MB1563.namprd08.prod.outlook.com ([25.163.143.139]) with mapi id 15.01.0172.012; Thu, 28 May 2015 17:49:25 +0000 From: "Lewis, Fred" To: Adrian Chadd , "Sundararajan, Lakshmi" CC: "Pokala, Ravi" , "freebsd-net@freebsd.org" , "jfv@freebsd.org" , "erj@freebsd.org" , "freebsd-hackers@freebsd.org" Subject: Re: Performance issues with Intel Fortville (XL710/ixl(4)) Thread-Topic: Performance issues with Intel Fortville (XL710/ixl(4)) Thread-Index: AQHQkn+7WBa4imHGwkeMI3kSTJaMlZ2HA5GAgAfVrACAApv3AA== Date: Thu, 28 May 2015 17:49:24 +0000 Message-ID: References: <5564b545.0287440a.5871.7050@mx.google.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=flewis@panasas.com; x-ms-exchange-messagesentrepresentingtype: 1 x-originating-ip: [66.31.107.140] x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(42134001)(42139001); SRVR:BY2PR08MB1797; UriScan:; BCL:0; PCL:0; RULEID:(42134001)(42139001); SRVR:BY2PR08MB157; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(520003)(5005006)(3002001); SRVR:BY2PR08MB1797; BCL:0; PCL:0; RULEID:; SRVR:BY2PR08MB1797; x-forefront-prvs: 0590BBCCBC x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(501624003)(24454002)(51704005)(189002)(164054003)(479174004)(199003)(377454003)(122556002)(81156007)(5001770100001)(15975445007)(50986999)(68736005)(102836002)(76176999)(106116001)(54356999)(2900100001)(101416001)(1720100001)(106356001)(2950100001)(105586002)(87936001)(4001540100001)(189998001)(40100003)(97736004)(2656002)(99286002)(62966003)(36756003)(77156002)(19580395003)(46102003)(86362001)(66066001)(64706001)(5001860100001)(19580405001)(92566002)(5001960100002)(5001830100001)(5002640100001)(94096001); DIR:OUT; SFP:1101; SCL:1; SRVR:BY2PR08MB1797; H:CY1PR0801MB1563.namprd08.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; received-spf: None (protection.outlook.com: panasas.com does not designate permitted sender hosts) Content-Type: text/plain; charset="Windows-1252" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 28 May 2015 17:49:24.8514 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: acf01c9d-c699-42af-bdbb-44bf582e60b0 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY2PR08MB1797 X-OriginatorOrg: panasas.com X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 May 2015 17:49:40 -0000 Hi Adrian, It is my understanding that you are maintaining the RSS code. The Panasas folks have this question for you. We are doing our testing w/ 11-CURRENT, but we will initially ship Intel XL710 40G NIC (Fortville) running on 10.1-RELEASE or 10.2-RELEASE. The presence of RSS - even though it is disabled by default - makes the driver back-port non-trivial. Is there an estimate on when the 11-CURRENT version of the ixl driver (1.4.1) and (especially) the supporting RSS code infrastructure will get MFCed to 10-STABLE? Thanks, -Fred On 5/26/15 5:58 PM, "Adrian Chadd" wrote: >hi! > >Try enabling RSS and PCBGROUPS on -HEAD. The ixl driver should work. > >(I haven't tested it though; I've had other things going on here.) > > > >-adrian > > >On 21 May 2015 at 15:20, Lakshmi Narasimhan Sundararajan > wrote: >> Hi FreeBSD Team! >> >> We seem to have found a problem to Tx performance. >> >> We found that the tx handling is spread on all CPUs causing probably >>cache trashing resulting in poor performance. >> >> But once we used cpuset to bind interrupt thread and iperf process to >>the same CPU, performance was close to line rate. I used userland cpuset >>command to perform this manually. I want this constrained in the kernel >>config/code through some tunables, and I am seeking your help/pointers >>in that regard. >> >> >> My followup questions are as follows. >> >> a) How are Tx interrupts steered from the NIC to the CPU on the >>transmit path? Would tx_complete# interrupt for packets transmitted from >>CPU#x, be serviced on the same CPU? If not, how to get this binding done? >> >> >> b) I would like to use a pool of CPUs dedicated to service NIC >>interrupts. Especially on the transmit path, I would want the >>tx_interrupts to be handled on the same CPU on which request was >>submitted. How to get this done? >> >> >> I played with the current ISR setting but I did not see any difference >>in how Interrupts are scheduled across CPU. The max interrupt threads >>even though set to one, the interrupt threads are scheduled on any CPU. >>Even if I set bindthreads to =8C1=B9. There is no difference in interrupt >>thread scheduling. >> >> >> root@mau-da-27-4-1:~ # sysctl net.isr >> net.isr.dispatch: direct >> net.isr.maxthreads: 1 >> net.isr.bindthreads: 0 >> net.isr.maxqlimit: 10240 >> net.isr.defaultqlimit: 256 >> net.isr.maxprot: 16 >> net.isr.numthreads: 1 >> >> >> I would sincerely appreciate if you can provide some pointers on these >>items above. >> >> >> >> >> Thanks >> >> LN >> >> >> >> >> >> >> >> From: Pokala, Ravi >> Sent: Wednesday, May 20, 2015 3:34 AM >> To: freebsd-net@freebsd.org, jfv@freebsd.org, erj@freebsd.org >> Cc: freebsd-hackers@freebsd.org, Lewis, Fred, Sundararajan, Lakshmi >> >> >> >> >> >> Hi folks, >> >> At Panasas, we are working with the Intel XL710 40G NIC (aka Fortville), >> and we're seeing some performance issues w/ 11-CURRENT (r282653). >> >> Motherboard: Intel S2600KP (aka Kennedy Pass) >> CPU: E5-2660 v3 @ 2.6GHz (aka Haswell Xeon) >> (1 socket x 10 physical cores x 2 SMT threads) =3D 20 logical >>cores >> NIC: Intel XL710, 2x40Gbps QSFP, configured in 4x10Gbps mode >> RAM: 4x 16GB DDR4 DIMMs >> >> What we've seen so far: >> >> - TX performance is pretty consistently lower than RX performance. All >> numbers below are for unidrectional tests using `iperf': >> 10Gbps links threads/link TX Gbps RX Gbps TX/RX >> 1 1 9.02 9.85 91.57% >> 1 8 8.49 9.91 85.67% >> 1 16 7.00 9.91 70.63% >> 1 32 6.68 9.92 67.40% >> >> - With multiple active links, both TX and RX performance suffer >>greatly; >> the aggregate bandwidth tops out at about a third of the theoretical >> 40Gbps implied by 4x 10Gbps. >> 10Gbps links threads/link TX Gbps RX Gbps % of >>40Gbps >> 4 1 13.39 13.38 33.4% >> >> - Multi-link bidirectional throughput is absolutely terrible; the >> aggregate is less than a tenth of the theoretical 40Gbps. >> 10Gbps links threads/link TX Gbps RX Gbps % of >>40Gbps >> 4 1 3.83 2.96 9.6% / >>7.4% >> >> - Occasional interrupt storm messages are seen from the IRQs >>associated >> with the NICs. Since that can impact performance, those runs were not >> included in the data listed above. >> >> Our questions: >> >> - How stable is ixl(4) in -CURRENT? By that, we mean both how quickly >>is >> the driver changing, and does the driver cause any system instability? >> >> - What type of performance have others been getting w/ Fortville? In >> 40Gbps mode? In 4x10Gbps mode? >> >> - Does anyone have any tuning parameters they can recommend for this >> card? >> >> - We did our testing w/ 11-CURRENT, but we will initially ship >>Fortville >> running on 10.1-RELEASE or 10.2-RELEASE. The presence of RSS - even >>though >> it is disabled by default - makes the driver back-port non-trivial. Is >> there an estimate on when the 11-CURRENT version of the driver (1.4.1) >> will get MFCed to 10-STABLE? >> >> My colleagues Lakshmi and Fred (CCed) are working on this; please make >> sure to include them if you have any comments. >> >> Thanks, >> >> Ravi >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"