From owner-freebsd-net@freebsd.org Sat Jan 19 19:02:20 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 525B1148F654 for ; Sat, 19 Jan 2019 19:02:20 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 9A7CA69C11 for ; Sat, 19 Jan 2019 19:02:19 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: by mailman.ysv.freebsd.org (Postfix) id 5DA4E148F651; Sat, 19 Jan 2019 19:02:19 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4AA87148F650 for ; Sat, 19 Jan 2019 19:02:19 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C364C69C0C for ; Sat, 19 Jan 2019 19:02:18 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id x0JJ0j4x002695 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 19 Jan 2019 20:00:49 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: brde@optusnet.com.au Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x0JJ0imU082481 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sun, 20 Jan 2019 02:00:44 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: [Bug 235031] [em] em0: poor NFS performance, strange behavior To: Bruce Evans , Martin Birgmeier References: <20190119204156.D929@besplex.bde.org> Cc: net@freebsd.org From: Eugene Grosbein Message-ID: <3e407ee7-54e3-a6ac-5535-d11aceca9558@grosbein.net> Date: Sun, 20 Jan 2019 02:00:39 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20190119204156.D929@besplex.bde.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: C364C69C0C X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.988,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 19 Jan 2019 19:02:20 -0000 19.01.2019 17:21, Bruce Evans wrote: > Your problem looks more like lost interrupts. All em NICs should interrupt > at the default interrupt moderation rate of 8 kHz under load. Once there > are are that many interrupts, there is not much else that can go wrong (nfs > would have to be working to generate that many interrupts). I have a patch (in production since 8.x) that makes em(4) support hw.em.max_interrupt_rate just like igb(4) supports hw.igb.max_interrupt_rate: http://www.grosbein.net/freebsd/patches/em_sysctl-11.0.diff.gz It also brings in sysctls dev.em.X.max_interrupt_rate and hw.em.max_interrupt_rate sets defaults for them. I use hw.em.max_interrupt_rate=32000 for 1GB link passing average sized packets (about 600 bytes per packet at average) but driver's default 8000 should be nearly fine for full size packets (1500 or above) and this 8000 limit cannot be reason for such low throughput. > Bugs in iflib are easy to avoid by running FreeBSD-11. PRO-1000 is supported > by most versions of FreeBSD and doesn't have the bug fixed by the above in > FreeBSD[7-11]. Agreed.