From owner-freebsd-net@freebsd.org Sun Jan 20 08:18:41 2019 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 63EA214AA723 for ; Sun, 20 Jan 2019 08:18:41 +0000 (UTC) (envelope-from d8zNeCFG@aon.at) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D34358BCEB for ; Sun, 20 Jan 2019 08:18:40 +0000 (UTC) (envelope-from d8zNeCFG@aon.at) Received: by mailman.ysv.freebsd.org (Postfix) id 961CD14AA722; Sun, 20 Jan 2019 08:18:40 +0000 (UTC) Delivered-To: net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 70ECD14AA721 for ; Sun, 20 Jan 2019 08:18:40 +0000 (UTC) (envelope-from d8zNeCFG@aon.at) Received: from smtpout-fallback.aon.at (smtpout-fallback.aon.at [195.3.96.120]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C3AD88BCE1 for ; Sun, 20 Jan 2019 08:18:39 +0000 (UTC) (envelope-from d8zNeCFG@aon.at) Received: (qmail 26673 invoked from network); 20 Jan 2019 08:18:38 -0000 Received: from unknown (HELO smtpout.aon.at) ([172.18.1.204]) (envelope-sender ) by fallback44.highway.telekom.at (qmail-ldap-1.03) with SMTP for ; 20 Jan 2019 08:18:38 -0000 X-A1Mail-Track-Id: 1547972318:26672:fallback44:172.18.1.204:1 Received: (qmail 12213 invoked from network); 20 Jan 2019 08:18:30 -0000 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on WARSBL608.highway.telekom.at X-Spam-Level: Received: from 188-23-243-129.adsl.highway.telekom.at (HELO gandalf.xyzzy) ([188.23.243.129]) (envelope-sender ) by smarthub85.res.a1.net (qmail-ldap-1.03) with DHE-RSA-AES256-GCM-SHA384 encrypted SMTP; 20 Jan 2019 08:18:30 -0000 X-A1Mail-Track-Id: 1547972309:12166:smarthub85:188.23.243.129:1 Received: from mizar.xyzzy (mizar.xyzzy [192.168.1.19]) by gandalf.xyzzy (8.15.2/8.15.2) with ESMTP id x0K8ITCn004663; Sun, 20 Jan 2019 09:18:29 +0100 (CET) (envelope-from d8zNeCFG@aon.at) Subject: Re: [Bug 235031] [em] em0: poor NFS performance, strange behavior To: Bruce Evans References: <20190119204156.D929@besplex.bde.org> <3e407ee7-54e3-a6ac-5535-d11aceca9558@grosbein.net> <20190120061258.X3312@besplex.bde.org> <16ce1832-13da-d7bb-cce2-6682e058b5a6@aon.at> <20190120145627.X1077@besplex.bde.org> Cc: Eugene Grosbein , net@freebsd.org From: Martin Birgmeier Message-ID: Date: Sun, 20 Jan 2019 09:18:29 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <20190120145627.X1077@besplex.bde.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: C3AD88BCE1 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 20 Jan 2019 08:18:41 -0000 Regarding duplex, ifconfig shows the following: [0]# ifconfig em0 em0: flags=8843 metric 0 mtu 1500         options=81249b         ether f0:de:f1:98:86:a9         inet 192.168.1.19 netmask 0xffffff00 broadcast 192.168.1.255         inet6 fe80::f2de:f1ff:fe98:86a9%em0 prefixlen 64 scopeid 0x1         inet6 fec0:0:0:4d42::13 prefixlen 64         inet6 fec0::4d42:f2de:f1ff:fe98:86a9 prefixlen 64 autoconf         inet6 2002:bc17:f381:4d42:f2de:f1ff:fe98:86a9 prefixlen 64 autoconf         media: Ethernet autoselect (1000baseT )         status: active         nd6 options=23 [0]# This seems to be o.k. -- Martin On 20.01.19 06:28, Bruce Evans wrote: > On Sat, 19 Jan 2019, Martin Birgmeier wrote: > >> I just tried the patch by Bruce (from the mail sent 10 hours ago), but >> it makes no difference. >> >> Also, it does not seem like bad frames or too high an interrupt rate are >> the problem (the machine should easily handle what is coming from its >> NFS client which only has a 100 Mbps interface). >> >> I believe that the simplifications introduced to sys/dev/e1000 between >> 11.2 and 12.0 have broken something. > > They aren't exactly simplifications :-). > > Did you check for the common problex of a duplex mismatch?  ISR that some > versions if iflib'ed em didn't negotiate right for your speed of 100 > Mbps. > > Here I can break nfs using "ifconfig em0 media 100baseTX mediaopt > full-duplex" and forgetting the mediaopt part.  This gives half-duplex. > ipv4 ping still works, but its latency increases from ~125 usec to ~76 > msec.  The latter latency destroys nfs performance.  After the media > change, there are a lot of DUP packets with an initial latency of ~43 > second and the latency decreasing by the ping interval of 1 second for > the next 42 or 43 DUPs until the backlog is cleared; the latency is > then between 71 and 80 msec.  Changing the media and mediaopt back > to 1000baseT[X] full-duplex restores low latency but causes 1 DUP with > delay ~19 seconds > > Suspend/resume used to give much the same misbehaviour, by not stopping > the NIC when reinitializing it in resume.  This was fixed in r342855. > This might be the bug!  iflib_media_change() calls iflib_init_locked() > liked resume used to, so seems to be missing stopping.  Changing this > should fix at least the DUPs. > > The function names or layering are confusing.  iflib_init_locked() > doesn't initialize the if.  iflib_if_init_locked() does that.  All > iflib_init_locked() does is call iflib_stop(), then iflib_init_locked(). > and iflib.  Grep shows the following related iflib*init*() calls: > - iflib_netmap_register manually inlines iflib_if_init_locked().  This >   is a style bug > - iflib_media_change() only calls iflib_init_locked().  This seems to be >   a bug > - _task_fn_admin() calls iflib_if_init_locked() for resetting.  This > seems >   to be correctly obfuscated > - iflib_if_init_locked() calls iflib_init_locked().  This is part of >   implementing the obfuscation - iflib_if_init() calls > iflib_if_init_locked().  This is correct > - iflib_if_ioctl(): SIOCSIFMTU calls iflib_stop(), then does some > locking, >   then sets the mtu in software, then calls iflib_init_locked().  This >   seems to be correct, and shows that the iflib_if_init_locked() is not >   even generally useful.  This gives down/up for non-null changes.  This >   works correctly (some ping packets are lost, but there are no DUPs. > - iflib_if_ioctl(): SIOCSIFCAP is like SIOCSIFMTU, except I didn't test >   it and its splitting of stopping and init'ing is a bit messier because >   both operations are under a more complicated conditional. > - iflib_if_ioctl(): calls iflib_if_init().  This >   is correct. > - iflib_vlan_[un]register() call iflib_if_init_locked().  This seems > to be >   correctly obfuscated > - iflib_device_resume() calls iflib_if_init_locked().  This is correctly >   obfuscated > - if_setinitfn() is called to set iflib_if_init as the init function.  > This >   is correct. > > Summary: only media change seems to be broken, but there are some > style bugs. > > The bug apparently btoke resume by reinitializing an active state > (even locking doesn't help much, but I now remember than resume > succeeded every 10-100 tries in the buggy versions -- there were always > a lot of DUPs, but sometimes to low latency came back).  My tests > usually used zzz and my zzz and other utilities are on nfs, so nfs was > fairly active just before suspend. > > I don't know if iflib_media_change() is called at boot time, especially > if the media is autoselect.  At boot time, the state might be less > active or closer to the reset state, so that even a manual media change > that surely calls iflib_media_change() has more chance of working than > at resume time with zzz and other utilities on nfs. > > I don't know what the media was after the broken resume.  Its reported > result can't be trusted anyway.  To recover from the broken resume, it > usually worked to repeat down/up a few times.  This is consistent with > bug -- eventually, previous down/up's change the state to close enough > to stopped.  But using the interface in any way (including pinging it > to see if it is still broken) makes it not so close to being stopped. > > Bruce