From owner-freebsd-net@freebsd.org Fri Oct 23 05:36:23 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 21B47A164D2 for ; Fri, 23 Oct 2015 05:36:23 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 03CC77A5 for ; Fri, 23 Oct 2015 05:36:23 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id 0142DA164D1; Fri, 23 Oct 2015 05:36:23 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F40FBA164D0 for ; Fri, 23 Oct 2015 05:36:22 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail109.syd.optusnet.com.au (mail109.syd.optusnet.com.au [211.29.132.80]) by mx1.freebsd.org (Postfix) with ESMTP id A7E0C7A3; Fri, 23 Oct 2015 05:36:22 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c211-30-166-197.carlnfd1.nsw.optusnet.com.au (c211-30-166-197.carlnfd1.nsw.optusnet.com.au [211.30.166.197]) by mail109.syd.optusnet.com.au (Postfix) with ESMTPS id 31840D67561; Fri, 23 Oct 2015 16:36:09 +1100 (AEDT) Date: Fri, 23 Oct 2015 16:36:07 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Bruce Evans cc: "Eggert, Lars" , Ian Smith , Kevin Oberman , "net@freebsd.org" , "jfv@FreeBSD.org" , "ricera10@gmail.com" , Luigi Rizzo , Giuseppe Lettieri Subject: Re: ixl 40G bad performance? In-Reply-To: <20151020232218.G1833@besplex.bde.org> Message-ID: <20151023162337.L1149@besplex.bde.org> References: <79830D9D-94E6-47A9-92B9-D63DF5432272@netapp.com> <20151020190541.B15983@sola.nimnet.asn.au> <6CD6754D-FC0E-4B24-AAEC-7C9D68284141@netapp.com> <20151020232218.G1833@besplex.bde.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.1 cv=cK4dyQqN c=1 sm=1 tr=0 a=KA6XNC2GZCFrdESI5ZmdjQ==:117 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=kj9zAlcOel0A:10 a=7HLWPhpPLtQj6zYKw5gA:9 a=CjuIK1q_8ugA:10 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Oct 2015 05:36:23 -0000 On Wed, 21 Oct 2015, Bruce Evans wrote: > Fix for em: > > X diff -u2 if_em.c~ if_em.c > X --- if_em.c~ 2015-09-28 06:29:35.000000000 +0000 > X +++ if_em.c 2015-10-18 18:49:36.876699000 +0000 > X @@ -609,8 +609,8 @@ > X em_tx_abs_int_delay_dflt); > X em_add_int_delay_sysctl(adapter, "itr", > X - "interrupt delay limit in usecs/4", > X + "interrupt delay limit in usecs", > X &adapter->tx_itr, > X E1000_REGISTER(hw, E1000_ITR), > X - DEFAULT_ITR); > X + 1000000 / MAX_INTS_PER_SEC); > X X /* Sysctl for limiting the amount of work done in the taskqueue */ > > "delay limit" is fairly good wording. Other parameters tend to give long > delays, but itr limits the longest delay due to interrupt moderation to > whatever the itr respresents. Everything in the last paragraph is backwards (inverted). Other parameters tend to give short delays. They should be set to small values to minimise latency. Then under load, itr limits the interrupt _rate_ from above. The interrupt delay is the inverse of the interrupt rate, so it is limited from below. So "delay limit" is fairly bad wording. Normally, limits are from above, but the inversion makes the itr limit from below. This is most easily understood by converting itr to a rate: itr = 125 means a rate limit of 8000 Hz. It doesn't quite mean that the latency is at least 125 usec. No one wants to ensure large latencies, and the itr setting only ensures a minimal average latency them under load. Bruce