From owner-freebsd-net@freebsd.org Fri Oct 23 21:43:10 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64A47A1C6EC for ; Fri, 23 Oct 2015 21:43:10 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 38C891E2D for ; Fri, 23 Oct 2015 21:43:10 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 38D29A1C6EB; Fri, 23 Oct 2015 21:43:10 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 385F2A1C6EA for ; Fri, 23 Oct 2015 21:43:10 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: from mail-qg0-f53.google.com (mail-qg0-f53.google.com [209.85.192.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E649E1E2C; Fri, 23 Oct 2015 21:43:09 +0000 (UTC) (envelope-from ricera10@gmail.com) Received: by qgeo38 with SMTP id o38so78006919qge.0; Fri, 23 Oct 2015 14:43:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-type; bh=k6BJ7qKf2QAS+nkyxFPthAZjqSCrFDDfmyytETa1Bb8=; b=Jg7xb8MkMv3w6wWs4new7/OeNdZcpqBvM8ivRb+c9vw81h/eXlb0LgBELEmMIWLbcS hUwKDNOAAQH8UoNc/E1wfMz5ZZI86odo0CI92TyOgpTGrO733OtHFfe/Cc//dXvsqiAB pam954FX4Akpri6qjFB0+iCXfkrJwHW6i+cDMlzc6Gyxi9IWkUxcnHMMNoALEXUo1bzS CF9vo7qZ87wID/5HMSJsn0tCncbfU6wb6vyyPoZNc28WW2FGdArt3gwQhI/1qQ5sV2PE EQuPqgBxAUwx0hclz9pLKexI0U3mqRkH4YEhfzQa7B0ZErqYaIBjnXAekO9l+EzWfqoZ o2Iw== X-Received: by 10.140.146.13 with SMTP id 13mr29955233qhs.1.1445636181279; Fri, 23 Oct 2015 14:36:21 -0700 (PDT) Received: from mail-qg0-f41.google.com (mail-qg0-f41.google.com. [209.85.192.41]) by smtp.gmail.com with ESMTPSA id 42sm8233992qky.39.2015.10.23.14.36.20 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 23 Oct 2015 14:36:20 -0700 (PDT) Received: by qgad10 with SMTP id d10so77384685qga.3; Fri, 23 Oct 2015 14:36:20 -0700 (PDT) X-Received: by 10.140.94.55 with SMTP id f52mr27419721qge.0.1445636180376; Fri, 23 Oct 2015 14:36:20 -0700 (PDT) MIME-Version: 1.0 References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> <20151023162337.L1149@besplex.bde.org> In-Reply-To: <20151023162337.L1149@besplex.bde.org> From: Eric Joyner Date: Fri, 23 Oct 2015 21:36:10 +0000 Message-ID: Subject: Re: ixl 40G bad performance? To: Bruce Evans Cc: "Eggert, Lars" , Ian Smith , Kevin Oberman , "net@freebsd.org" , "jfv@FreeBSD.org" , Luigi Rizzo , Giuseppe Lettieri Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Oct 2015 21:43:10 -0000 Bruce mostly has it right -- ITR is the minimum latency between interrupts. But, it does actually guarantee a minimum period between interrupts. Though, Fortville actually is unique a little bit in that there is another ITR setting that can ensure a certain average number of interrupts per second (called Interrupt Rate Limiting), though, but I don't think this is used in the current version of the driver. I see that the sysctl does clobber the global value, but have you tried lowering the interval / raising the rate? You could try something like 10usecs, and see if that helps. We'll do some more investigation here -- 3Gb/s on a 40Gb/s using default settings is terrible, and we shouldn't let that be happening. - Eric On Thu, Oct 22, 2015 at 10:36 PM Bruce Evans wrote: > On Wed, 21 Oct 2015, Bruce Evans wrote: > > > Fix for em: > > > > X diff -u2 if_em.c~ if_em.c > > X --- if_em.c~ 2015-09-28 06:29:35.000000000 +0000 > > X +++ if_em.c 2015-10-18 18:49:36.876699000 +0000 > > X @@ -609,8 +609,8 @@ > > X em_tx_abs_int_delay_dflt); > > X em_add_int_delay_sysctl(adapter, "itr", > > X - "interrupt delay limit in usecs/4", > > X + "interrupt delay limit in usecs", > > X &adapter->tx_itr, > > X E1000_REGISTER(hw, E1000_ITR), > > X - DEFAULT_ITR); > > X + 1000000 / MAX_INTS_PER_SEC); > > X X /* Sysctl for limiting the amount of work done in the taskqueue */ > > > > "delay limit" is fairly good wording. Other parameters tend to give long > > delays, but itr limits the longest delay due to interrupt moderation to > > whatever the itr respresents. > > Everything in the last paragraph is backwards (inverted). Other > parameters tend to give short delays. They should be set to small > values to minimise latency. Then under load, itr limits the interrupt > _rate_ from above. The interrupt delay is the inverse of the interrupt > rate, so it is limited from below. So "delay limit" is fairly bad > wording. Normally, limits are from above, but the inversion makes > the itr limit from below. > > This is most easily understood by converting itr to a rate: itr = 125 > means a rate limit of 8000 Hz. It doesn't quite mean that the latency > is at least 125 usec. No one wants to ensure large latencies, and the > itr setting only ensures a minimal average latency them under load. > > Bruce >