From owner-freebsd-net@freebsd.org Thu Dec 10 19:42:35 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4EC729D7889 for ; Thu, 10 Dec 2015 19:42:35 +0000 (UTC) (envelope-from dennix.pearson@gmail.com) Received: from mail-io0-x241.google.com (mail-io0-x241.google.com [IPv6:2607:f8b0:4001:c06::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 185CF17A0 for ; Thu, 10 Dec 2015 19:42:35 +0000 (UTC) (envelope-from dennix.pearson@gmail.com) Received: by ioo197 with SMTP id 197so9097718ioo.1 for ; Thu, 10 Dec 2015 11:42:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=02aY70kvHI8MFAhyPSy/oWseFFaqcnoxc+NNTtzTD6E=; b=N4HvCIF6IoCsTCY5renxSYF6oE+zUP2Z0nSPVrzGw00ReM1nBMcbowTMN+F78lUcEb Nc3MN5nELtZ+GaYATBs83YGBrWc3NdV5KS4JaK/h3NNf/rD9nhE+2SNJjxx1Uuyb443r vkHXr3Yw3kPDedXweo9DI52SjtyoCfjKF9iiakCrz/BOhGkciPqXFLmGE1caU1iPkoro kHAyhvvz+HhtI0nskeLQEUZMKrNd6M8wcqTbcHrT4iGdFI+WiUMaW9oZoGabTpBt01I2 A8x+9reTMzo6FQexhxj8X1kdF4bmMtp0Lefr7aQdgLOufF09Gf4VsftUDnlnnV4JGhOA bE/Q== MIME-Version: 1.0 X-Received: by 10.107.156.195 with SMTP id f186mr24188ioe.54.1449776554354; Thu, 10 Dec 2015 11:42:34 -0800 (PST) Received: by 10.79.27.149 with HTTP; Thu, 10 Dec 2015 11:42:34 -0800 (PST) In-Reply-To: References: <5aae0ee63c44627223d5d179f1901d00@pyret.net> <0E4C2D93-FBAF-48CB-A704-499ABFC892B9@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2A8A@ORSMSX111.amr.corp.intel.com> <99E53825-99F8-4E82-A710-6BC07B123F77@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2D52@ORSMSX111.amr.corp.intel.com> Date: Thu, 10 Dec 2015 17:42:34 -0200 Message-ID: Subject: Re: ixl 40G bad performance? From: Denis Pearson To: Adrian Chadd Cc: "Eggert, Lars" , "Pieper, Jeffrey E" , Kevin Oberman , Daniel Engberg , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2015 19:42:35 -0000 On Thu, Dec 10, 2015 at 4:40 PM, Adrian Chadd wrote: > On 10 December 2015 at 10:29, Denis Pearson > wrote: > > On Thu, Dec 10, 2015 at 2:18 PM, Eggert, Lars wrote: > > > >> On 2015-10-26, at 18:40, Eggert, Lars wrote: > >> > On 2015-10-26, at 17:08, Pieper, Jeffrey E < > jeffrey.e.pieper@intel.com> > >> wrote: > >> >> As a caveat, this was using default netperf message sizes. > >> > > >> > I get the same ~3 Gb/s with the default netperf sizes and driver > 1.4.5. > >> > >> Now there is version 1.4.8 on the Intel website, but it doesn't change > >> things for me. > >> > > > > I had the opportunity to see similar numbers and behavior while using > XL710 > > 1.4.3 as of FreeBSD r291085 while in DPDK poll mode, but driver 1.2.8 as > of > > r292035 was providing expected numbers. While removing rxcsum/txcsum did > > not provide differences, fully removing RSS + disabling rx/cxsum support > > provided better numbers. > > Can someone debug this a bit more? (My kit with ixl NICs in it is > still not up and available. :( ) > > Device RSS, even without kernel RSS enabled, shouldn't cause a massive > performance drop. If it is then something else odd is going on. > Do you have a diff where you removed things? > I can probably find out a snapshot with the code at the time and extract a diff, yes. I just don't know how it worths wasting the time when the problem is not reproducible on the current 1.4.8 driver which will hopefully get into -CURRENT (if it's not already there?). And it's much more specific, the performance drop happened on dpdk poll mode, not the usual kernel operation so a simple diff only pointing out the changes for the driver to actually build and run without rss will still require a testlab and different ways to generate traffic. This is why I suggested a transceiver change or replug first. Anyway RSS performance dropping problem is far from a FreeBSD specific problem, while researching I could find the exact same complaints on Windows users starting from windows 8 while having RSS@4 or RSS@16 or RSS completely disabled, some times with acceptable results only when it was disabled (despiste the fact that MiniportInterruptDPC was using a whole CPU when RSS was off results were still better). So I guess this is just a side effect of when it's just good to have NIC features turned off. The reason, I'm not an engineer to answer, but I would guess it's related to other NIC features also doing something with the packet or any sort of errors netstat or driver status may not tell. I was able to see the problem even with low pps rates and big packet sizes, as well as avg pkt size of 768bytes so I don't think it's any sort of card resource starvation. I can manage to have the whole lab up and running by the weekend if you want to investigate and compare, just ping me off list. > > -adrian > > > However now with driver 1.4.8 and the same set of hardware setup, except > > for a different transceiver, I can get 36Gbps/24Mpps with no further > > tweaks, so if you can replace your transceiver, shall be a different test > > as a starting point. >