From owner-freebsd-net@freebsd.org Mon Oct 19 15:55:56 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD027A19C2B for ; Mon, 19 Oct 2015 15:55:56 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8B702B4 for ; Mon, 19 Oct 2015 15:55:56 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 8A3AEA19C2A; Mon, 19 Oct 2015 15:55:56 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 89D2EA19C29 for ; Mon, 19 Oct 2015 15:55:56 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lb0-x22d.google.com (mail-lb0-x22d.google.com [IPv6:2a00:1450:4010:c04::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1051FB3; Mon, 19 Oct 2015 15:55:56 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by lbbwb3 with SMTP id wb3so86199714lbb.1; Mon, 19 Oct 2015 08:55:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=F9AMt1eUK1vcQHHjhQaO7coZmUGhztKS/cTtirL5EVA=; b=CKo3RgXTywk0dVOUwwIDphzGVAZ6ZBp5+sJcxmWC/wqiYorwHWiqEzKv1yXqpmcdVg BUOX/UIeJJflyVQ/ef7YJ8PSLwhdO1v62/t8lBuBxyKcvImEBMwlOae66v6hyJwRoueA 5etGro0H+EZ1bk3XKpAcJhlbJpHoaJbSnna6VIsXAmH+aJyIvCqiLEjGuJ0t32VKOShE hdCNsdIT0vndFn5TP+n7isDDqlGbdGr5BvRm9/SQCZG/kjF/lHC3gMa9OzSrxMc0Mco9 wBpUYEba6tUOxJPHJNIgbnyR+OgaBxVSkxxbIeieP3LgH3WBjBn/1rXfY6Kjh8fbsrbC uAWw== MIME-Version: 1.0 X-Received: by 10.112.134.73 with SMTP id pi9mr14993048lbb.83.1445270154005; Mon, 19 Oct 2015 08:55:54 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.177.4 with HTTP; Mon, 19 Oct 2015 08:55:53 -0700 (PDT) In-Reply-To: References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> Date: Mon, 19 Oct 2015 08:55:53 -0700 X-Google-Sender-Auth: Gjof0kR7rV8irksJdTek3unGLww Message-ID: Subject: Re: ixl 40G bad performance? From: Luigi Rizzo To: "Eggert, Lars" Cc: "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 15:55:56 -0000 On Mon, Oct 19, 2015 at 8:34 AM, Eggert, Lars wrote: > Hi, > > in order to eliminate network or hardware weirdness, I've rerun the test = with Linux 4.3rc6, where I get 13.1 Gbits/sec throughput and 52 usec flood = ping latency. Not great either, but in line with earlier experiments with M= ellanox NICs and an untuned Linux system. > ... >> There must be some other control for the actual (fixed, not dynamic) >> moderation. > > The only other sysctls in ixl(4) that look relevant are: > > hw.ixl.rx_itr > The RX interrupt rate value, set to 8K by default. > > hw.ixl.tx_itr > The TX interrupt rate value, set to 4K by default. > yes those. raise to 20-50k and see what you get in terms of ping latency. Note that 4k on tx means you get to reclaim buffers in the tx queue (unless it is done opportunistically) every 250us which is dangerously close to the 300us capacity of the queue itself. cheers luigi > I'll play with those. > >>> Also no change with "hw.ixl.ringsz=3D256" in loader.conf. >> >> Any better success with 2048 slots? >> 3.5 gbit is what I used to see on the ixgbe with tso disabled, probably >> hitting a CPU bound. > > Will try. > > Thanks! > > Lars --=20 -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+-------------------------------