From nobody Sun May 1 19:15:27 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 61B261AA945E for ; Sun, 1 May 2022 19:15:42 +0000 (UTC) (envelope-from marklmi@yahoo.com) Received: from sonic313-20.consmr.mail.gq1.yahoo.com (sonic313-20.consmr.mail.gq1.yahoo.com [98.137.65.83]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Krwt44Zqcz3PBp for ; Sun, 1 May 2022 19:15:40 +0000 (UTC) (envelope-from marklmi@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1651432533; bh=MRwZ1w2lcjUlpjjxpg3nk0cZQT79VImXffvge+s5uVE=; h=Subject:From:In-Reply-To:Date:Cc:References:To:From:Subject:Reply-To; b=bSnO9FIroSVmfLYs4bXe2srOSDci9VbEpli10Qz6YizsqvY7f2PqWPHKIv2KaAVM7LcN0qQlA3C3ycYbocBZSokZCEGH4QXcZ2155pjFK0PidGH6ao6OqWZLMkojIlZn7vYYGKZyx9doJWQSYXDqQfMQSo3Z7GjqOCKt9+QGDh7cmojxeEJMazAvif/ZqsZ4IM0QRVknTyi0CAPBwO8OZB6BcuUaiSZseTyYq4GTNFLgXhZZaNhO+E3pkZ68dgmXnnE+l2hOUpw+aHOQGJqPeMJAX6qDjbetGMxJO1gHOXyol0twzrhpi7ZRx9F6UFL4MxPjqfl4A9buFi7jA2crWA== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1651432533; bh=PBmbN1QUw9wqGGV9So3TuK0E0DnyT+hQmfojsZemsKn=; h=X-Sonic-MF:Subject:From:Date:To:From:Subject; b=tAE5+sdj3JiYkwvuLmoWse1yHgtBcjPTXTdZndvCSdBCCaAxJsUhPWJiPBOYDjcUZ9Wfirhtff3UPP7a4b16+cta0K1yS4JBfTpwZOsP2sHPJYLGdmBnIrUoj5ibYZU5lcn3VQLcxPeNQBXrleQHGMaEEu8gLcCTuse6QWZCDxHJkohvopG/8X/QcOlCKj3lGnlTUmcdKLfUSb6QlkeYspXenCEZ2Xm73SU6mNaYF76PcIcmmX9w1sz0H9XEiR8ObuVol2i/mVNVSWx7euZIUEVz24avZYVV1xd8kOMC3c4xNY4ClJviFA9T7N2nka6a79I3a/p86A8VRFHEaPs5eg== X-YMail-OSG: E.23QzEVM1mauu_jyNW16JTFipGjxG5USm4Cbv_GASMGUc81wtSjg1L1qkCmkL5 eeYbU.Td2gUXbOr84r9rG8U1FmgRqn.YhpqJZCJojGNKWHfBtjMvyyhUlBWkVgYlIlojpVIKXtf1 qqHMTfHa4HaWT6CYtXMDGhZU_vns_bTyrBPY5FwaV15ZOXo5x1KJq4yHvYVs4UzF7hYf3IP46zWd yjKNSYjOef70bY6RST.C680T6QSPAhdR9_Z5m_UbtcKN8MA48R_ZVe7kvEsH5OOT7KIJjK9rXEpO oQ3DhQcyWMTDPX4cF8CJv02x381.oD8AIEhtQQ_EmYij2.oc.JrpWXl7be_lZfB.TLT5WJrh1ABc are_DxgbeMm16zt2dxj5SVdU7UOgW2pTd0uDzcKP12xuLW1m9Y2b8sIlZVNnC1xi7jqTqmQ6QtT9 Hsh9AminY0TNnZG.DU426FPCvmDX7t67eUWp66LvLeZM2QNuKdyaVObkCwyx1O_pvHeeX.eNjJhS j426GTqDlpVoaS3TB3coFLDLnDoy906ccjRhBV6r6sJlY1b5S6a.Xxb1jQvEwbq_iPXQvY4H74mU TG197E0Iz9K43t6n_kjcpYWVfkWYafkXY7g08m.UoXAvuwQjYS8n9gXoI7IJ4PonxhHRi.S25aTI vmlvydt1vvfiF0xy.9jHiBQQxjKYwQbslU_w0.ldM2HHoHZ_Rufpg8lmQvckanxTeV1d8cdeZmjI Ibn6HVqsw0JAAh59tyY_5pNJoCDBz5zPpEy13qslUpXlM6uX3P64ANv6bNCrliCkAanH3gjvcIhu 86sQi4toxaeoWirHFjoQ7ko.5SBQsLU_MVRAmsK0j2j9.5rexzYQtqRX7Nkk6UoWs7DaRS9UoOK. 41NoUxYUfLAN_65kU5FMlim0Zt7Al25J4T600T8rNPHGdhM_q1yDGG7QkrfqPP3gJz508pnHnlFH GK4Rt94dgkugGIr9I2XCap4UQACuByZbkKg2CH0M5QiGt46DqEkGl5nhLjh_5UXUvkJ2QpSH47Fg upG2qVsG5wPg50W_Wm2ePX759ur.uacb4uS2.eM23KMKQPLTOujRY0qoXNmGBR3pl5whAWURgWu8 LCRRGLRr0iJw8HfaE3xWP7dY4Trnfkpdiact4SqOGsIgcW5W2PFQhhi5CR9ZFnR3463PAM1_5Opq YBILqSRYeZhyiCFZygGMLnisZL7KxE7XNm26ZbYrpqRB1vRHH8DwTeahqHRpKtOQ.AN2FVAJ0pUq GyjSryR3udaxOHEk4opqFCTnOX8NboB_.fCeXE_S2NPWczm7BAyVUMBVSgL_Douv8K9.H_mGfYtB Ehp1xyjwOvsQTv4qwpP31aWF1JNZ4x5kzF8JZi_9XG0TEq8IruoaZo1b1KM6uVsbSDif.gy0Kc.L G04WazPeD.PYhQRKEsbl9pUmJzeRz5_xWcBP8thoYcqVye7hyg0QCGZd4oLGgjveGCu6Zuxms0zy JIBRfYwWZ1Nj2u500W14DhmqpvDTQVIe4Zp0LzH.jVtWM1eHod4FPZznnfxBs9Frz5ITaniACquy eQubd1Q5E0zkPnjohDXEkCqDmsOIhQaLNpZC2Nw1wl7u1v9LB2J0IgRtWRw5YN6TEK_rVmx2OHvJ Fam1QsjglCqMf6k5B8jzbYHMDQ0C6jJe0MSX_aLzs_urHCkIbneith3cClL7UmZ5.dewBByyQlKp 7xRvKVIBMWvcAFTXxGx2Vpl1hLOG0eA92gzOEmA4oNBIdibeBvEYJAzR8ilE_WtHWWVm.etQp20P bOPhhNNNQVhc10TcMgZcwlbyAjVU_3i2IunFVKO7oLxVVvyvZ2bIkZJ5.3Ki_BDwovNNzEeJ7_D1 XCSL2G0iiELYx63Kzx4E2M01tFDFxhGNcYETgVeHjlit4_lEgU9qEF_IOxq4qOibYoYy24pd6Eyg rXpL8DwYpVQoa5w4wC.ArPq76JZsjyobA1MPhxeJn9a5N.YUynnH8T5z98nX8LNDu3LEhiJApgL7 PyMvKXABay6.kwQZ_S6Vq0F6RmwxhSGs7ui_YXhZ6EIkBsTLfygoXg0pt8LgYiW1YbxuVXb4FfeF 3f26q8kiKnhdySUgHRaD_6eWBeX2nd6u2j5rVMm1CNBHSG3LDtVe2X.sfOpad7BFwuCkttR.l5c1 AoElYR31BwSzae9MlMRxaN0dXD.SAkAZcCgFSl2y2ckybezXoxb3Gu6ktNky1r2p_EYy0mXPrdne 7XiOjSDkV.HgqSxi1fX5l4sNehGtnIHOX1_L0UwkEBpLTQQti0ib90aRJ4Qe_BbFtYdpbprq.vHK r6v1aaF1IRP2Cc5nC88IIwRdPTjfjxKX59e7_ X-Sonic-MF: Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.gq1.yahoo.com with HTTP; Sun, 1 May 2022 19:15:33 +0000 Received: by hermes--canary-production-bf1-5f4c6455f8-b7pn2 (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 424593ac6b261432aef943d9ee0e71a2; Sun, 01 May 2022 19:15:30 +0000 (UTC) Content-Type: text/plain; charset=us-ascii List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 14.0 \(3654.120.0.1.13\)) Subject: Re: 60+% ping packet loss on Pi3 under -current and stable-13 From: Mark Millard In-Reply-To: <20220501181254.GA14961@www.zefox.net> Date: Sun, 1 May 2022 12:15:27 -0700 Cc: Bakul Shah , freebsd-net@freebsd.org, freebsd-arm@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20220314010330.GA70447@www.zefox.net> <9FA3F874-2987-44A2-A987-F905E78CCA65@yahoo.com> <20220428023226.GA5666@www.zefox.net> <8A57B411-AA06-47F4-935D-EDF45D8DF0EC@yahoo.com> <70C2DF4B-D08B-491D-B7B5-1EAD0D1BF0E3@yahoo.com> <20220429005206.GA1171@www.zefox.net> <20220430021207.GA7600@www.zefox.net> <20220501181254.GA14961@www.zefox.net> To: bob prohaska X-Mailer: Apple Mail (2.3654.120.0.1.13) X-Rspamd-Queue-Id: 4Krwt44Zqcz3PBp X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yahoo.com header.s=s2048 header.b=bSnO9FIr; dmarc=pass (policy=reject) header.from=yahoo.com; spf=pass (mx1.freebsd.org: domain of marklmi@yahoo.com designates 98.137.65.83 as permitted sender) smtp.mailfrom=marklmi@yahoo.com X-Spamd-Result: default: False [-3.37 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; FREEMAIL_FROM(0.00)[yahoo.com]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; NEURAL_HAM_LONG(-0.88)[-0.876]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; RCVD_IN_DNSWL_NONE(0.00)[98.137.65.83:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[freebsd-net]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36647, ipnet:98.137.64.0/20, country:US]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yahoo.com:dkim] X-ThisMailContainsUnwantedMimeParts: N On 2022-May-1, at 11:12, bob prohaska wrote: > On Sat, Apr 30, 2022 at 06:39:57PM -0700, Bakul Shah wrote: >> On Apr 29, 2022, at 7:12 PM, bob prohaska wrote: >>>=20 >>> Since about December of 2021 I've been noticing problems with >>> wired network connectivity on a pair of raspberry pi 3 machines >>> using wired network connections. One runs stable-13.1, the other >>> runs -current, both are up to date as of a few days ago. >>>=20 >>> Essentially both machines fail to respond to inbound network >>> connections via ssh or ping after reboot. If I get on the=20 >>> serial console and start an outbound ping to anywhere, both >>> machines respond to incoming pings with about a 65% packet >>> loss.=20 >=20 >> Suggest running tcpdump on the rpi3 to see what is going on >> when connected to the public vs private net.=20 >>=20 >=20 > Public net first, since that's where the machine is now. = Gateway.zefox.net > is the name of my router's public interface, dcn.org belongs to my isp = and > fusionbroadband is their service provider.. >=20 > While on the -current Pi3 serial console (with no outbound ping = running)=20 > and no inbound traffic from my hosts I see after a couple minutes: >=20 > root@www:/mnt # tcpdump > tcpdump: verbose output suppressed, use -v or -vv for full protocol = decode > listening on ue0, link-type EN10MB (Ethernet), capture size 262144 = bytes > 10:39:40.887853 ARP, Request who-has www.zefox.org tell = gateway.zefox.net, length 46 > 10:39:40.887929 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui = Unknown), length 28 > 10:39:40.893220 ARP, Request who-has = 50-1-20-1.dsl.static.fusionbroadband.com tell www.zefox.org, length 28 > 10:39:40.915469 ARP, Reply 50-1-20-1.dsl.static.fusionbroadband.com = is-at 00:1b:90:d2:4a:c4 (oui Unknown), length 50 > 10:39:40.915529 IP www.zefox.org.50714 > spoke.dcn.davis.ca.us.domain: = 51409+ PTR? 28.20.1.50.in-addr.arpa. (41) > 10:39:40.943602 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.50714: = 51409 1/3/6 PTR www.zefox.org. (265) > 10:39:40.945416 IP www.zefox.org.15986 > spoke.dcn.davis.ca.us.domain: = 44966+ PTR? 31.20.1.50.in-addr.arpa. (41) > 10:39:40.973487 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.15986: = 44966 1/3/6 PTR gateway.zefox.net. (266) > 10:39:40.975037 IP www.zefox.org.57611 > spoke.dcn.davis.ca.us.domain: = 31749+ PTR? 1.20.1.50.in-addr.arpa. (40) > 10:39:46.288219 IP www.zefox.org.49710 > wheel.dcn.davis.ca.us.domain: = 31749+ PTR? 1.20.1.50.in-addr.arpa. (40) > 10:39:46.316239 IP wheel.dcn.davis.ca.us.domain > www.zefox.org.49710: = 31749 1/3/6 PTR 50-1-20-1.dsl.static.fusionbroadband.com. (291) > 10:39:46.318267 IP www.zefox.org.17061 > spoke.dcn.davis.ca.us.domain: = 37579+ PTR? 2.253.150.168.in-addr.arpa. (44) > 10:39:46.346851 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.17061: = 37579* 1/2/2 PTR spoke.dcn.davis.ca.us. (145) > 10:39:46.348674 IP www.zefox.org.40440 > spoke.dcn.davis.ca.us.domain: = 20572+ PTR? 1.253.150.168.in-addr.arpa. (44) > 10:39:51.420705 IP www.zefox.org.64019 > wheel.dcn.davis.ca.us.domain: = 20572+ PTR? 1.253.150.168.in-addr.arpa. (44) > 10:39:51.448850 IP wheel.dcn.davis.ca.us.domain > www.zefox.org.64019: = 20572* 1/2/2 PTR wheel.dcn.davis.ca.us. (145) > 10:40:40.147603 ARP, Request who-has = 50-1-20-1.dsl.static.fusionbroadband.com tell ns1.zefox.net, length 46 > 10:40:40.148844 IP www.zefox.org.46127 > spoke.dcn.davis.ca.us.domain: = 12186+ PTR? 29.20.1.50.in-addr.arpa. (41) > 10:40:40.176486 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.46127: = 12186 1/3/6 PTR ns1.zefox.net. (262) > 10:40:57.688225 ARP, Request who-has www.zefox.org tell = gateway.zefox.net, length 46 > 10:40:57.688305 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui = Unknown), length 28 > 10:42:14.488727 ARP, Request who-has www.zefox.org tell = gateway.zefox.net, length 46 > 10:42:14.488804 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui = Unknown), length 28 > 10:42:43.761226 ARP, Request who-has = 50-1-20-1.dsl.static.fusionbroadband.com tell www.zefox.com, length 46 > 10:42:43.762522 IP www.zefox.org.56181 > spoke.dcn.davis.ca.us.domain: = 28779+ PTR? 26.20.1.50.in-addr.arpa. (41) > 10:42:43.790361 IP spoke.dcn.davis.ca.us.domain > www.zefox.org.56181: = 28779 1/3/6 PTR www.zefox.com. (265) > 10:43:31.289103 ARP, Request who-has www.zefox.org tell = gateway.zefox.net, length 46 > 10:43:31.289181 ARP, Reply www.zefox.org is-at b8:27:eb:71:46:4e (oui = Unknown), length 28 >=20 > If I now start an inbound ping from one of my hosts it gets no reply = and=20 > tcpdump reports no additional traffic. With an outbound ping running = there's > at least a sparse reply. >=20 > ^C > 28 packets captured > 28 packets received by filter > 0 packets dropped by kernel > root@www:/mnt #=20 >=20 > The "oui unknown" looks like some sort of failure..... > Can you ping www.zefox.org? I have no outside vantage point. > There is still no outbound ping running and I would expect > you'll get no or very sparse reply.=20 >=20 >=20 > Thus far only the two Pi3s suffer from connectivity problems; Pi2s and = a Pi4 have > no difficulty on the same address block. Is there a switch for tcpdump = that will > limit records to relevant traffic? Otherwise it's a flood. >=20 > These results were obtained after standing idle overnight and > are rather different (in ways I don't understand) from behavior > immediately after reboot, I'll have to repeat as I learn more. I wonder if there is a notable difference between monitoring traffic from 2 places: A) from the machine seeing the problem vs. B) from a machine not having problems but connected were all the traffic would be on the wire it is connected to. It may be that monitoring from both and comparing/contrasting the reported traffic from the two provides additional evidence. There may be modes of monitoring that are relevant for this. But I'm not familiar with any detail here. For reference: # ping www.zefox.org PING www.zefox.org (50.1.20.28): 56 data bytes ^C --- www.zefox.org ping statistics --- 32 packets transmitted, 0 packets received, 100.0% packet loss I found the command traceroute and it reports: # traceroute www.zefox.org traceroute to www.zefox.org (50.1.20.28), 64 hops max, 40 byte packets 1 192.168.1.1 (192.168.1.1) 0.697 ms 0.486 ms 1.277 ms 2 172.30.26.66 (172.30.26.66) 30.019 ms 172.30.26.67 (172.30.26.67) 41.720 ms 172.30.26.66 (172.30.26.66) 28.645 ms 3 68.85.243.125 (68.85.243.125) 8.967 ms 68.85.243.77 (68.85.243.77) 11.462 ms 68.85.243.125 (68.85.243.125) 10.254 ms 4 24.124.129.106 (24.124.129.106) 7.510 ms 96.216.60.165 (96.216.60.165) 10.176 ms 24.124.129.106 (24.124.129.106) 8.945 ms 5 68.85.243.197 (68.85.243.197) 10.837 ms 96.216.60.165 (96.216.60.165) 10.252 ms 68.85.243.197 (68.85.243.197) 16.036 ms 6 68.85.243.197 (68.85.243.197) 14.660 ms be-36211-cs01.seattle.wa.ibone.comcast.net (68.86.93.49) 14.629 ms 68.85.243.197 (68.85.243.197) 8.849 ms 7 be-2412-pe12.seattle.wa.ibone.comcast.net (96.110.34.142) 14.607 ms be-36221-cs02.seattle.wa.ibone.comcast.net (68.86.93.53) 14.122 ms be-2212-pe12.seattle.wa.ibone.comcast.net (96.110.34.134) 13.877 ms 8 be-2412-pe12.seattle.wa.ibone.comcast.net (96.110.34.142) 14.133 ms = * 13.663 ms 9 be2075.ccr21.sfo01.atlas.cogentco.com (154.54.0.233) 30.176 ms * be3717.ccr22.sfo01.atlas.cogentco.com (154.54.86.209) 29.002 ms 10 be3717.ccr22.sfo01.atlas.cogentco.com (154.54.86.209) 28.477 ms be2430.ccr31.sjc04.atlas.cogentco.com (154.54.88.186) 27.203 ms be2075.ccr21.sfo01.atlas.cogentco.com (154.54.0.233) 28.515 ms 11 38.104.141.82 (38.104.141.82) 29.820 ms be2430.ccr31.sjc04.atlas.cogentco.com (154.54.88.186) 28.605 ms 38.104.141.82 (38.104.141.82) 33.735 ms 12 38.104.141.82 (38.104.141.82) 27.160 ms 0.xe-0-3-0.scrm-gw1.scrmca01.sonic.net (135.180.179.146) 32.336 ms 38.104.141.82 (38.104.141.82) 31.867 ms 13 0.xe-0-0-0.cr1.scrmca13.sonic.net (135.180.179.166) 31.761 ms 0.xe-0-3-0.scrm-gw1.scrmca01.sonic.net (135.180.179.146) 29.864 ms 0.xe-0-0-0.cr1.scrmca13.sonic.net (135.180.179.166) 31.711 ms 14 0.xe-0-0-0.cr1.scrmca13.sonic.net (135.180.179.166) 30.373 ms gig1-1-1.gw.wscrca11.sonic.net (50.1.36.106) 35.567 ms 0.xe-0-0-0.cr1.scrmca13.sonic.net (135.180.179.166) 31.146 ms 15 gig1-1-1.gw.davsca11.sonic.net (50.1.36.110) 31.513 ms gig1-1-1.gw.wscrca11.sonic.net (50.1.36.106) 31.203 ms gig1-1-1.gw.davsca11.sonic.net (50.1.36.110) 31.354 ms 16 gig1-1-1.gw.davsca11.sonic.net (50.1.36.110) 30.125 ms * 31.996 ms 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * ^C (There did not seem to be much point in having it continue.) =3D=3D=3D Mark Millard marklmi at yahoo.com