From owner-freebsd-net@freebsd.org Thu Dec 10 18:48:28 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 701DF9D6B6F for ; Thu, 10 Dec 2015 18:48:28 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lf0-x235.google.com (mail-lf0-x235.google.com [IPv6:2a00:1450:4010:c07::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EB9FB1B06 for ; Thu, 10 Dec 2015 18:48:27 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by lffu14 with SMTP id u14so63355995lff.1 for ; Thu, 10 Dec 2015 10:48:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=FvEMUncJsyyPzzZPg4apo6aH3DV69Hdf3yHPidDXZV0=; b=B8zD9dquaA0PI49eGtoJSxTlu5agohLTLf3XavHsMxG9pknLShtLYqfGIlohfOhns2 IZafpCFpZkqqND/4BLWOmNlq0gbQCAddMSSewiJy+eYbNjQwoEGNFHKBJZXHLAbgiPr1 K0CWeqCwcnbvDR67y/9SHvMHNutES8lpAZ33LX5RN2zPATXyf8eLene4Yda8VBTYc5TC rPb90jQxS+hlx/3UKBmBfpO9bS2Dy+bZeLYd8UGE/XMSYY6M63vaenhfZNgPyt5lnP2K Rxc/6fnZcEqJuOKv8QwkIixKTYUMNpJKfIrbaMmzGuWnESfCUr6mSlSshQ56TAOUsX5T yQCg== MIME-Version: 1.0 X-Received: by 10.25.4.208 with SMTP id 199mr5903702lfe.96.1449773306075; Thu, 10 Dec 2015 10:48:26 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.97.100 with HTTP; Thu, 10 Dec 2015 10:48:26 -0800 (PST) In-Reply-To: References: <5aae0ee63c44627223d5d179f1901d00@pyret.net> <0E4C2D93-FBAF-48CB-A704-499ABFC892B9@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2A8A@ORSMSX111.amr.corp.intel.com> <99E53825-99F8-4E82-A710-6BC07B123F77@netapp.com> <2A35EA60C3C77D438915767F458D6568807F2D52@ORSMSX111.amr.corp.intel.com> Date: Thu, 10 Dec 2015 10:48:26 -0800 X-Google-Sender-Auth: QH2RcnCI9LcjPgoFF9zKh1cFKQs Message-ID: Subject: Re: ixl 40G bad performance? From: Luigi Rizzo To: Adrian Chadd Cc: Denis Pearson , "Pieper, Jeffrey E" , Kevin Oberman , Daniel Engberg , "Eggert, Lars" , "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Dec 2015 18:48:28 -0000 On Thu, Dec 10, 2015 at 10:40 AM, Adrian Chadd wrote: > On 10 December 2015 at 10:29, Denis Pearson wrote: >> On Thu, Dec 10, 2015 at 2:18 PM, Eggert, Lars wrote: >> >>> On 2015-10-26, at 18:40, Eggert, Lars wrote: >>> > On 2015-10-26, at 17:08, Pieper, Jeffrey E >>> wrote: >>> >> As a caveat, this was using default netperf message sizes. >>> > >>> > I get the same ~3 Gb/s with the default netperf sizes and driver 1.4.5. >>> >>> Now there is version 1.4.8 on the Intel website, but it doesn't change >>> things for me. >>> >> >> I had the opportunity to see similar numbers and behavior while using XL710 >> 1.4.3 as of FreeBSD r291085 while in DPDK poll mode, but driver 1.2.8 as of >> r292035 was providing expected numbers. While removing rxcsum/txcsum did >> not provide differences, fully removing RSS + disabling rx/cxsum support >> provided better numbers. > > Can someone debug this a bit more? (My kit with ixl NICs in it is > still not up and available. :( ) > > Device RSS, even without kernel RSS enabled, shouldn't cause a massive > performance drop. If it is then something else odd is going on. I am not sure whether we are digressing (Lars' complaint was about poor bulk throughput, now i see DPDK and high packet rates mentioned so i feel obliged to pitch in!) but a related piece of info: last spring, with netmap and i40e on linux (don't remember which driver/firmware), we saw that enabling FlowDirector killed the pps throughput (from 32 down to 18 Mpps). FlowDirector is a device feature which was probably affecting ordinary processing on the NIC, either because of bugs or because of consuming controller resources. The same may be possibly happening with other device features. cheers luigi > > Do you have a diff where you removed things? > > > -adrian > >> However now with driver 1.4.8 and the same set of hardware setup, except >> for a different transceiver, I can get 36Gbps/24Mpps with no further >> tweaks, so if you can replace your transceiver, shall be a different test >> as a starting point. > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+-------------------------------