From owner-freebsd-net@freebsd.org Sat Jan 19 20:06:30 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 091AE14917CE for ; Sat, 19 Jan 2019 20:06:30 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3DDB86C106 for ; Sat, 19 Jan 2019 20:06:29 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: by mailman.ysv.freebsd.org (Postfix) id 00E2F14917CD; Sat, 19 Jan 2019 20:06:29 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D27C314917CC for ; Sat, 19 Jan 2019 20:06:28 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail104.syd.optusnet.com.au (mail104.syd.optusnet.com.au [211.29.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id 130816C105 for ; Sat, 19 Jan 2019 20:06:27 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from [192.168.0.102] (c110-21-101-228.carlnfd1.nsw.optusnet.com.au [110.21.101.228]) by mail104.syd.optusnet.com.au (Postfix) with ESMTPS id 72D62430EB4; Sun, 20 Jan 2019 07:06:17 +1100 (AEDT) Date: Sun, 20 Jan 2019 07:06:15 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Eugene Grosbein cc: Martin Birgmeier , net@freebsd.org Subject: Re: [Bug 235031] [em] em0: poor NFS performance, strange behavior In-Reply-To: <3e407ee7-54e3-a6ac-5535-d11aceca9558@grosbein.net> Message-ID: <20190120061258.X3312@besplex.bde.org> References: <20190119204156.D929@besplex.bde.org> <3e407ee7-54e3-a6ac-5535-d11aceca9558@grosbein.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.2 cv=FNpr/6gs c=1 sm=1 tr=0 a=PalzARQSbocsUSjMRkwAPg==:117 a=PalzARQSbocsUSjMRkwAPg==:17 a=kj9zAlcOel0A:10 a=H0GPC0OhAAAA:8 a=DRwI9Vhvhbz9f9Gp7HoA:9 a=CjuIK1q_8ugA:10 a=KczGKrPSgCPlefTG41c3:22 X-Rspamd-Queue-Id: 130816C105 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.97 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.965,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jan 2019 20:06:30 -0000 On Sun, 20 Jan 2019, Eugene Grosbein wrote: > 19.01.2019 17:21, Bruce Evans wrote: > >> Your problem looks more like lost interrupts. All em NICs should interrupt >> at the default interrupt moderation rate of 8 kHz under load. Once there >> are are that many interrupts, there is not much else that can go wrong (nfs >> would have to be working to generate that many interrupts). > > I have a patch (in production since 8.x) that makes em(4) support hw.em.max_interrupt_rate > just like igb(4) supports hw.igb.max_interrupt_rate: > > http://www.grosbein.net/freebsd/patches/em_sysctl-11.0.diff.gz > > It also brings in sysctls dev.em.X.max_interrupt_rate and hw.em.max_interrupt_rate sets defaults for them. This is inverted and spelled dev.em.X.itr for em. Hmm, em already has this, but it is only a read-only tunable. igb seems to have gone away. In FreeBSD-11, its dev.em.X.max_interrupt_rate is also only a tunable. I use the variants of the following fix for itr in FreeBSD-[7-13] XX Index: if_em.c XX =================================================================== XX --- if_em.c (revision 332488) XX +++ if_em.c (working copy) XX @@ -908,10 +910,10 @@ XX E1000_REGISTER(hw, E1000_TADV), XX em_tx_abs_int_delay_dflt); XX em_add_int_delay_sysctl(adapter, "itr", XX - "interrupt delay limit in usecs/4", XX + "interrupt delay limit in usecs", XX &adapter->tx_itr, XX E1000_REGISTER(hw, E1000_ITR), XX - DEFAULT_ITR); XX + 1000000 / MAX_INTS_PER_SEC); XX XX hw->mac.autoneg = DO_AUTO_NEG; XX hw->phy.autoneg_wait_to_complete = FALSE; This fixes the description and the initial value for the sysctl to match the code. The description almost matches the buggy initial value. The hardware has power of 2 units, but the code scales to microseconds. Except the initial value has was in hardware units scaled by another power of 2 which made them nearly microseconds/4. The code sets the initial value to a representation of 125 usec (8 kHz), but the sysctl says that the initial value is 488 and the description says that this is a representation of 488/4 = 122 usec. However, writing back this value using sysctl gives 488 usec (~2 kHz). The magic number 122 is 125 mis-scaled by 1000/1024. FreeBSD[7-10] have lem in a separate file with the bug duplicated, so need the patch duplicated. FreeBSD[7-8] don't have a sysctl for this. They default to 125 usec and there is no way to see or change the value. I usually want the smaller value of 0, and hard-code this when there is no sysctl. DEFAULT_ITR is used mainly to obfuscate this. IGB_DEFAULT_ITR and IGB_LINK_ITR are also defined, but are not used even in versions of FreeBSD that have igb. > I use hw.em.max_interrupt_rate=32000 for 1GB link passing average sized packets > (about 600 bytes per packet at average) but driver's default 8000 should be nearly fine > for full size packets (1500 or above) and this 8000 limit cannot be reason for such low throughput. 0 for itr maxes out at about 100 kHz here. This is good for low latency with small packets. My version of bge dynamically modifies the rate to match the rx load (no moderation for light loads). tx is handled specially and only needs 1 interrupt every few seconds for freeing resources. Bruce