From owner-freebsd-questions@freebsd.org Sun Oct 6 20:50:17 2019 Return-Path: <owner-freebsd-questions@freebsd.org> Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D0032133586 for <freebsd-questions@mailman.nyi.freebsd.org>; Sun, 6 Oct 2019 20:50:17 +0000 (UTC) (envelope-from SRS0=MWKz=X7=mail.sermon-archive.info=doug@sermon-archive.info) Received: from mail.sermon-archive.info (sermon-archive.info [71.177.216.148]) by mx1.freebsd.org (Postfix) with ESMTP id 46mbN85pW2z3Jv2 for <freebsd-questions@freebsd.org>; Sun, 6 Oct 2019 20:50:16 +0000 (UTC) (envelope-from SRS0=MWKz=X7=mail.sermon-archive.info=doug@sermon-archive.info) Received: from [10.0.1.251] (mini [10.0.1.251]) by mail.sermon-archive.info (Postfix) with ESMTPSA id 46mbN70cvzz2fjvD; Sun, 6 Oct 2019 13:50:15 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.11\)) Subject: Re: Intermittent connectivity loss with em(4) From: Doug Hardie <bc979@lafn.org> In-Reply-To: <ba24db9b-6a34-76d4-7c95-4c2e64d0a5a0@pp.dyndns.biz> Date: Sun, 6 Oct 2019 13:50:14 -0700 Cc: FreeBSD Questions <freebsd-questions@freebsd.org> Content-Transfer-Encoding: quoted-printable Message-Id: <94B563F6-55C4-46BC-BD79-5CC2AD86E6C1@mail.sermon-archive.info> References: <ba24db9b-6a34-76d4-7c95-4c2e64d0a5a0@pp.dyndns.biz> To: =?utf-8?Q?Morgan_Wesstr=C3=B6m?= <freebsd-database@pp.dyndns.biz> X-Mailer: Apple Mail (2.3445.104.11) X-Virus-Scanned: clamav-milter 0.101.2 at mail X-Virus-Status: Clean X-Rspamd-Queue-Id: 46mbN85pW2z3Jv2 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of SRS0=MWKz=X7=mail.sermon-archive.info=doug@sermon-archive.info designates 71.177.216.148 as permitted sender) smtp.mailfrom=SRS0=MWKz=X7=mail.sermon-archive.info=doug@sermon-archive.info X-Spamd-Result: default: False [-1.51 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.95)[-0.949,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:71.177.216.148]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-0.17)[asn: 5650(-0.79), country: US(-0.05)]; NEURAL_HAM_LONG(-0.99)[-0.987,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[148.216.177.71.list.dnswl.org : 127.0.10.0]; FORGED_SENDER(0.30)[bc979@lafn.org,SRS0=MWKz=X7=mail.sermon-archive.info=doug@sermon-archive.info]; RCVD_NO_TLS_LAST(0.10)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5650, ipnet:71.177.216.0/23, country:US]; FROM_NEQ_ENVFROM(0.00)[bc979@lafn.org,SRS0=MWKz=X7=mail.sermon-archive.info=doug@sermon-archive.info]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions <freebsd-questions.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-questions/> List-Post: <mailto:freebsd-questions@freebsd.org> List-Help: <mailto:freebsd-questions-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-questions>, <mailto:freebsd-questions-request@freebsd.org?subject=subscribe> X-List-Received-Date: Sun, 06 Oct 2019 20:50:17 -0000 > On 6 October 2019, at 12:34, Morgan Wesstr=C3=B6m = <freebsd-database@pp.dyndns.biz> wrote: >=20 > I'd appreciate some help with the following problem because I'm = probably blind to the obvious from all my trouble shooting. >=20 > I have a small Supermicro Atom-based ITX-board with 3x Intel 82574L = NICs (2 on mainboard, 1 in PCIe slot) and I experience intermittent = network connectivity loss every few minutes. The machine is currently = running FreeBSD 12.0-RELEASE-p10. I loop a single ping once per minute = to dns.google (8.8.8.8) and output looks like this: >=20 > THINGS I'VE TESTED WITHOUT RESOLVING THE PROBLEM >=20 > - Tried all three NICs in the computer but they all show the same = behaviour. > - Tried a different ping target. > - Tried the LiveCD environment from the older FreeBSD 11.3-RELEASE = memstick. > - Disabled MSI-X interrupts. > - Additionally disabled MSI interrupts. > - Recompiled em(4) and enabled DEBUG_INIT, DEBUG_IOCTL and DEBUG_HW = but this only generates a few more message in dmesg during boot. There = is nothing shown in dmesg or otherwise when connectivity is lost. >=20 > THING'S I'VE TRIED THAT SEEMINGLY RESOLVES OR ALLEVIATES THE PROBLEM >=20 > - Booting the system on Linux (4.19 kernel) with just a simple command = prompt and running the same ping test does _NOT_ show any connectivity = loss. At least not during the hour or so I tested. To me this rules out = any hardware related problems as well as the network connection itself. > - Generating small amounts of network traffic on the interface (like = an ssh session) seems to reduce the problem. I can then run the ping = test for maybe 30 minutes without loss of connectivity in FreeBSD but = eventually it fails too. >=20 > I really need help to understand what's going on here. I have a gut = feeling some power saving is playing a trick on me but = hw.em.smart_pwr_down is set to 0 default and I have no indication of any = power saving function kicking in. How can I debug what's going on in = em(4)? Am I just stupid and missing something obvious here? I just tried the same thing using 12.0-RELEASE-p9. No drops. mail# ping -i 60 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=3D0 ttl=3D54 time=3D10.528 ms 64 bytes from 8.8.8.8: icmp_seq=3D1 ttl=3D54 time=3D11.403 ms 64 bytes from 8.8.8.8: icmp_seq=3D2 ttl=3D54 time=3D10.351 ms 64 bytes from 8.8.8.8: icmp_seq=3D3 ttl=3D54 time=3D10.787 ms 64 bytes from 8.8.8.8: icmp_seq=3D4 ttl=3D54 time=3D12.280 ms 64 bytes from 8.8.8.8: icmp_seq=3D5 ttl=3D54 time=3D10.038 ms 64 bytes from 8.8.8.8: icmp_seq=3D6 ttl=3D54 time=3D10.640 ms 64 bytes from 8.8.8.8: icmp_seq=3D7 ttl=3D54 time=3D11.090 ms 64 bytes from 8.8.8.8: icmp_seq=3D8 ttl=3D54 time=3D10.615 ms 64 bytes from 8.8.8.8: icmp_seq=3D9 ttl=3D54 time=3D8.972 ms 64 bytes from 8.8.8.8: icmp_seq=3D10 ttl=3D54 time=3D10.016 ms 64 bytes from 8.8.8.8: icmp_seq=3D11 ttl=3D54 time=3D11.097 ms 64 bytes from 8.8.8.8: icmp_seq=3D12 ttl=3D54 time=3D13.035 ms 64 bytes from 8.8.8.8: icmp_seq=3D13 ttl=3D54 time=3D11.251 ms 64 bytes from 8.8.8.8: icmp_seq=3D14 ttl=3D54 time=3D9.588 ms 64 bytes from 8.8.8.8: icmp_seq=3D15 ttl=3D54 time=3D10.649 ms 64 bytes from 8.8.8.8: icmp_seq=3D16 ttl=3D54 time=3D9.965 ms 64 bytes from 8.8.8.8: icmp_seq=3D17 ttl=3D54 time=3D9.900 ms 64 bytes from 8.8.8.8: icmp_seq=3D18 ttl=3D54 time=3D11.253 ms 64 bytes from 8.8.8.8: icmp_seq=3D19 ttl=3D54 time=3D9.440 ms 64 bytes from 8.8.8.8: icmp_seq=3D20 ttl=3D54 time=3D11.544 ms 64 bytes from 8.8.8.8: icmp_seq=3D21 ttl=3D54 time=3D11.068 ms 64 bytes from 8.8.8.8: icmp_seq=3D22 ttl=3D54 time=3D10.196 ms 64 bytes from 8.8.8.8: icmp_seq=3D23 ttl=3D54 time=3D12.446 ms 64 bytes from 8.8.8.8: icmp_seq=3D24 ttl=3D54 time=3D10.467 ms 64 bytes from 8.8.8.8: icmp_seq=3D25 ttl=3D54 time=3D11.100 ms 64 bytes from 8.8.8.8: icmp_seq=3D26 ttl=3D54 time=3D9.746 ms 64 bytes from 8.8.8.8: icmp_seq=3D27 ttl=3D54 time=3D10.627 ms 64 bytes from 8.8.8.8: icmp_seq=3D28 ttl=3D54 time=3D10.339 ms 64 bytes from 8.8.8.8: icmp_seq=3D29 ttl=3D54 time=3D10.166 ms 64 bytes from 8.8.8.8: icmp_seq=3D30 ttl=3D54 time=3D10.055 ms 64 bytes from 8.8.8.8: icmp_seq=3D31 ttl=3D54 time=3D9.303 ms 64 bytes from 8.8.8.8: icmp_seq=3D32 ttl=3D54 time=3D11.240 ms 64 bytes from 8.8.8.8: icmp_seq=3D33 ttl=3D54 time=3D11.407 ms 64 bytes from 8.8.8.8: icmp_seq=3D34 ttl=3D54 time=3D10.930 ms 64 bytes from 8.8.8.8: icmp_seq=3D35 ttl=3D54 time=3D10.166 ms 64 bytes from 8.8.8.8: icmp_seq=3D36 ttl=3D54 time=3D11.400 ms 64 bytes from 8.8.8.8: icmp_seq=3D37 ttl=3D54 time=3D10.946 ms ^C --- 8.8.8.8 ping statistics --- 38 packets transmitted, 38 packets received, 0.0% packet loss round-trip min/avg/max/stddev =3D 8.972/10.685/13.035/0.846 ms I have had issues with em nics in the past and don't use them anymore. = However, none of the issues were as blatant as you are seeing. The dmsg = log is only written to during boot. Messages during operation are = written to /var/log/messages. Check there to see if there is anything = that correlates with the outages. -- Doug