From owner-freebsd-net@freebsd.org Mon Oct 19 15:11:22 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6CBE3A190D6 for ; Mon, 19 Oct 2015 15:11:22 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4AAE4A01 for ; Mon, 19 Oct 2015 15:11:22 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 4923CA190D5; Mon, 19 Oct 2015 15:11:22 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48B9BA190D4 for ; Mon, 19 Oct 2015 15:11:22 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lf0-x22b.google.com (mail-lf0-x22b.google.com [IPv6:2a00:1450:4010:c07::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B14C59FF; Mon, 19 Oct 2015 15:11:21 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by lffy185 with SMTP id y185so113268797lff.2; Mon, 19 Oct 2015 08:11:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=cmggJVsJ0IpkPuioz/Oqe0ScHX7iFNs/4Fu0L9mcCPM=; b=ifvs0CU5ETUC3rrnRxOfvSgPDvGJJAKyxnPlNJ8kKmGWYRNezzuE/BzKyoe2iCrrAo JpGh2pSGkw2n52dtniril/auqyY8ix+wluN5HTcQBGebtuuEIn8aF0D438mfh1qb6/pL uR6Y5fpT8SURnm2cF2VAkl2cOh5B+gHnXMjLfJ9Z3MEppljD48VfQEqY9UtG0UuYl5vh sNqcs50xY0rD70tFr7VN8TsoIUeiK+IxDuaQapAW3KjBMdsywRBUjfJMK+dWj0UtIHxr fNB4FsepUcJzm/v11H4TJoj4WCE7dHOm2waqAg8wlapMBQN7ui2Ar9/QXekB1cw4QQLQ S1iA== MIME-Version: 1.0 X-Received: by 10.25.24.195 with SMTP id 64mr9864471lfy.71.1445267479620; Mon, 19 Oct 2015 08:11:19 -0700 (PDT) Sender: rizzo.unipi@gmail.com Received: by 10.114.177.4 with HTTP; Mon, 19 Oct 2015 08:11:19 -0700 (PDT) In-Reply-To: References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> Date: Mon, 19 Oct 2015 08:11:19 -0700 X-Google-Sender-Auth: hkj5Fxk2sqeMzOek1vpVKf1Eaig Message-ID: Subject: Re: ixl 40G bad performance? From: Luigi Rizzo To: "Eggert, Lars" Cc: "jfv@FreeBSD.org" , "ricera10@gmail.com" , Giuseppe Lettieri , "net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Oct 2015 15:11:22 -0000 On Monday, October 19, 2015, Eggert, Lars wrote: > Hi, > > On 2015-10-19, at 16:20, Luigi Rizzo > > wrote: > > > > i would look at the following: > > - c states and clock speed - make sure you never go below C1, > > and fix the clock speed to max. > > Sure these parameters also affect the 10G card, but there > > may be strange interaction that trigger the power saving > > modes in different ways > > I already have powerd_flags="-a max -b max -n max" in rc.conf, which I > hope should be enough. I suspect it might not touch the c states, but better check. The safest is disable them in the bios. > > > - interrupt moderation (may affect ping latency, > > do not remember how it is set in ixl but probably a sysctl > > ixl(4) describes two sysctls that sound like they control AIM, and they > default to off: > > hw.ixl.dynamic_tx_itr: 0 > hw.ixl.dynamic_rx_itr: 0 > > There must be some other control for the actual (fixed, not dynamic) moderation. > > - number of queues (32 is a lot i wouldn't use more than 4-8), > > may affect cpu-socket affinity > > With hw.ixl.max_queues=4 in loader.conf, performance is still unchanged. > > > - tso and flow director - i have seen bad effects of > > accelerations so i would run the iperf test with > > of these features disabled on both sides, and then enable > > them one at a time > > No change with "ifconfig -tso4 -tso6 -rxcsum -txcsum -lro". > > How do I turn off flow director? I am not sure if it is enabled I'm FreeBSD. It is in linux and almost halves the pkt rate with netmap (from 35 down to 19mpps). Maybe it is not too bad for bulk TCP. > > > - queue sizes - the driver seems to use 1024 slots which is > > about 1.5 MB queued, which in turn means you have 300us > > (and possibly half of that) to drain the queue at 40Gbit/s. > > 150-300us may seem an eternity, but if a couple of cores fall > > into c7 your budget is gone and the loss will trigger a > > retransmission and window halving etc. > > Also no change with "hw.ixl.ringsz=256" in loader.conf. Any better success with 2048 slots? 3.5 gbit is what I used to see on the ixgbe with tso disabled, probably hitting a CPU bound. Cheers Luigi > This is really weird. > > Lars > -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+-------------------------------