From nobody Fri Jul 19 16:08:54 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WQZP75JB5z5RtR6 for ; Fri, 19 Jul 2024 16:09:15 +0000 (UTC) (envelope-from ararslan@comcast.net) Received: from resqmta-h2p-567062.sys.comcast.net (resqmta-h2p-567062.sys.comcast.net [IPv6:2001:558:fd02:2446::a]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WQZP63sj8z52Xc for ; Fri, 19 Jul 2024 16:09:14 +0000 (UTC) (envelope-from ararslan@comcast.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=comcast.net header.s=20190202a header.b=0W5y4cOu; dmarc=pass (policy=quarantine) header.from=comcast.net; spf=pass (mx1.freebsd.org: domain of ararslan@comcast.net designates 2001:558:fd02:2446::a as permitted sender) smtp.mailfrom=ararslan@comcast.net Received: from resomta-h2p-540626.sys.comcast.net ([96.102.179.210]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 256/256 bits) (Client did not present a certificate) by resqmta-h2p-567062.sys.comcast.net with ESMTPS id UmqJsm6kjYAjuUqAMsELkZ; Fri, 19 Jul 2024 16:09:06 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=20190202a; t=1721405346; bh=WI6IJnWQbylSIKVaftEkQG5zaXIqfJoseu/Wobk0bJ8=; h=Received:Received:Content-Type:Mime-Version:Subject:From:Date: Message-Id:To:Xfinity-Spam-Result; b=0W5y4cOuT00Rz8i5441B+sqJpb8A4Z2v964Xw4A4D5tnMnMCREXgQftqYEloOCGP6 X3gGutZZrfSKpHyggITQJIV0DnJfAm9IMyZkbeoQe1x7tHJW56rA17ISh2Tx5dZXc1 qS2COvifjfuzmDI+lIhMX6gq2xveVQjnweOspw0dDxMtIpPUcCX1Odse8gTMbCuAvt D9QnhWFi3qyjO+B7amDvu8NAgoJFCArzmUOsGdTqbfbTDOXkq7LjQh19pRKdO/ivY8 Fo4ErBsKzRQiEQHj+fXtB/xBWj13Q5Pdyawpu8iIcbGx4ub4rHypDlrDLUWDin+GhN uDD9VMoGwikew== Received: from smtpclient.apple ([67.160.29.205]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 256/256 bits) (Client did not present a certificate) by resomta-h2p-540626.sys.comcast.net with ESMTPSA id UqAKs2H5ZGEJIUqALsOXnu; Fri, 19 Jul 2024 16:09:06 +0000 Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\)) Subject: Re: Diagnosing virtual machine network issues From: Alex Arslan In-Reply-To: <4a5a177a-5356-453c-8a09-f1d63d5d2e16@sentex.net> Date: Fri, 19 Jul 2024 09:08:54 -0700 Cc: freebsd-hackers@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <4AB1C33B-DD93-4484-B63A-9FF8FE612B15@comcast.net> References: <4a5a177a-5356-453c-8a09-f1d63d5d2e16@sentex.net> To: mike tancsa X-Mailer: Apple Mail (2.3774.600.62) X-CMAE-Envelope: MS4xfOVb3c66GXFXJoKbc5vj3GoDRxoMFVffEV2w8LvC0xx/eeC++nhn9d5OCer7gXoUlfn1hJFtW41pZPBbA6xD13Nm7JM+llS+9zICRQywIGFaZhbePuVS CCBuruz3Xzy4HQQsI7Q18Nb+i+nPG0udM8Vh0KT+TFC9jFFgVD+V9ms+5l8T4Kcs190k7Hwps7mijzARiECS87Oj9kD/tbJG3AuX3eXSfcujUQKRYW8qBn7B X-Spamd-Bar: / X-Spamd-Result: default: False [-1.00 / 15.00]; HFILTER_HELO_5(3.00)[resqmta-h2p-567062.sys.comcast.net]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; DMARC_POLICY_ALLOW(-0.50)[comcast.net,quarantine]; R_SPF_ALLOW(-0.20)[+ip6:2001:558:fd02:2446::/64]; R_DKIM_ALLOW(-0.20)[comcast.net:s=20190202a]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[comcast.net]; TO_DN_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; ASN(0.00)[asn:7922, ipnet:2001:558::/29, country:US]; FREEMAIL_ENVFROM(0.00)[comcast.net]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[comcast.net:+]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; APPLE_MAILER_COMMON(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; DWL_DNSWL_NONE(0.00)[comcast.net:dkim] X-Rspamd-Queue-Id: 4WQZP63sj8z52Xc > I would start a pcap inside and outside of the VM for all udp port 53 = traffic as a start to see if its a network issue going out of the box. = If it happens frequently and you think it might be the network, perhaps = try with the Intel em driver instead of the virtio network driver ? Thanks so much for your help! The way I implemented your pcap suggestion was to use tcpdump, hopefully that's correct. I ran tcpdump simultaneously on the host and VM then ran the code where libcurl gives a timeout rather than the expected domain resolution failure. The output is below. I'm pretty well outside of my depth here; what is it I'm looking for that would be indicative of a network issue going out of the VM? Linux host: $ sudo /usr/sbin/tcpdump -v -i any 'host 192.168.122.35 and port 53' tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), = capture size 262144 bytes 21:06:03.320754 IP (tos 0x0, ttl 64, id 29048, offset 0, flags [none], = proto UDP (17), length 60) 192.168.122.35.24119 > amdci6.domain: 23532+ A? domain.invalid. (32) 21:06:03.320754 IP (tos 0x0, ttl 64, id 29048, offset 0, flags [none], = proto UDP (17), length 60) 192.168.122.35.24119 > amdci6.domain: 23532+ A? domain.invalid. (32) 21:06:03.321633 IP (tos 0x0, ttl 64, id 27798, offset 0, flags [none], = proto UDP (17), length 73) 192.168.122.35.18137 > amdci6.domain: 61699+ PTR? = 35.122.168.192.in-addr.arpa. (45) 21:06:03.321633 IP (tos 0x0, ttl 64, id 27798, offset 0, flags [none], = proto UDP (17), length 73) 192.168.122.35.18137 > amdci6.domain: 61699+ PTR? = 35.122.168.192.in-addr.arpa. (45) 21:06:03.321701 IP (tos 0x0, ttl 64, id 44762, offset 0, flags [DF], = proto UDP (17), length 113) amdci6.domain > 192.168.122.35.18137: 61699* 1/0/0 = 35.122.168.192.in-addr.arpa. PTR freebsd-debugging-amdci6-0. (85) 21:06:03.321707 IP (tos 0x0, ttl 64, id 44762, offset 0, flags [DF], = proto UDP (17), length 113) amdci6.domain > 192.168.122.35.18137: 61699* 1/0/0 = 35.122.168.192.in-addr.arpa. PTR freebsd-debugging-amdci6-0. (85) 21:06:03.322188 IP (tos 0x0, ttl 64, id 27799, offset 0, flags [none], = proto UDP (17), length 72) 192.168.122.35.37631 > amdci6.domain: 23871+ PTR? = 1.122.168.192.in-addr.arpa. (44) 21:06:03.322188 IP (tos 0x0, ttl 64, id 27799, offset 0, flags [none], = proto UDP (17), length 72) 192.168.122.35.37631 > amdci6.domain: 23871+ PTR? = 1.122.168.192.in-addr.arpa. (44) 21:06:08.446737 IP (tos 0x0, ttl 64, id 29049, offset 0, flags [none], = proto UDP (17), length 60) 192.168.122.35.24119 > amdci6.domain: 23532+ A? domain.invalid. (32) 21:06:08.446737 IP (tos 0x0, ttl 64, id 29049, offset 0, flags [none], = proto UDP (17), length 60) 192.168.122.35.24119 > amdci6.domain: 23532+ A? domain.invalid. (32) 21:06:18.567376 IP (tos 0x0, ttl 64, id 29050, offset 0, flags [none], = proto UDP (17), length 60) 192.168.122.35.37009 > amdci6.domain: 36459+ AAAA? domain.invalid. = (32) 21:06:18.567376 IP (tos 0x0, ttl 64, id 29050, offset 0, flags [none], = proto UDP (17), length 60) 192.168.122.35.37009 > amdci6.domain: 36459+ AAAA? domain.invalid. = (32) 21:06:23.671046 IP (tos 0x0, ttl 64, id 29051, offset 0, flags [none], = proto UDP (17), length 60) 192.168.122.35.37009 > amdci6.domain: 36459+ AAAA? domain.invalid. = (32) 21:06:23.671046 IP (tos 0x0, ttl 64, id 29051, offset 0, flags [none], = proto UDP (17), length 60) 192.168.122.35.37009 > amdci6.domain: 36459+ AAAA? domain.invalid. = (32) ^C 14 packets captured 20 packets received by filter 2 packets dropped by kernel FreeBSD VM: $ sudo tcpdump -v port 53 tcpdump: listening on vtnet0, link-type EN10MB (Ethernet), capture size = 262144 bytes 21:06:06.179751 IP (tos 0x0, ttl 64, id 29048, offset 0, flags [none], = proto UDP (17), length 60) freebsd-debugging-amdci6-0.24119 > amdci6.domain: 23532+ A? = domain.invalid. (32) 21:06:06.180634 IP (tos 0x0, ttl 64, id 27798, offset 0, flags [none], = proto UDP (17), length 73) freebsd-debugging-amdci6-0.18137 > amdci6.domain: 61699+ PTR? = 35.122.168.192.in-addr.arpa. (45) 21:06:06.180826 IP (tos 0x0, ttl 64, id 44762, offset 0, flags [DF], = proto UDP (17), length 113) amdci6.domain > freebsd-debugging-amdci6-0.18137: 61699* 1/0/0 = 35.122.168.192.in-addr.arpa. PTR freebsd-debugging-amdci6-0. (85) 21:06:06.181193 IP (tos 0x0, ttl 64, id 27799, offset 0, flags [none], = proto UDP (17), length 72) freebsd-debugging-amdci6-0.37631 > amdci6.domain: 23871+ PTR? = 1.122.168.192.in-addr.arpa. (44) 21:06:06.194107 IP (tos 0x0, ttl 64, id 44764, offset 0, flags [DF], = proto UDP (17), length 118) amdci6.domain > freebsd-debugging-amdci6-0.37631: 23871 2/0/0 = 1.122.168.192.in-addr.arpa. PTR amdci6., 1.122.168.192.in-addr.arpa. PTR = amdci6.local. (90) 21:06:11.305743 IP (tos 0x0, ttl 64, id 29049, offset 0, flags [none], = proto UDP (17), length 60) freebsd-debugging-amdci6-0.24119 > amdci6.domain: 23532+ A? = domain.invalid. (32) 21:06:21.426439 IP (tos 0x0, ttl 64, id 29050, offset 0, flags [none], = proto UDP (17), length 60) freebsd-debugging-amdci6-0.37009 > amdci6.domain: 36459+ AAAA? = domain.invalid. (32) 21:06:26.530138 IP (tos 0x0, ttl 64, id 29051, offset 0, flags [none], = proto UDP (17), length 60) freebsd-debugging-amdci6-0.37009 > amdci6.domain: 36459+ AAAA? = domain.invalid. (32) ^C 8 packets captured 427 packets received by filter 0 packets dropped by kernel