Date: Wed, 14 Aug 2024 09:15:22 -0700 From: Alex Arslan <ararslan@comcast.net> To: Bakul Shah <bakul@iitbombay.org> Cc: "Rodney W. Grimes" <freebsd-rwg@gndrsh.dnsmgr.net>, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: Diagnosing virtual machine network issues Message-ID: <B830418F-C9A4-4B59-A5E9-0A1E7BE1CC78@comcast.net> In-Reply-To: <C2D17751-17A9-4429-91ED-5E60B471B22F@iitbombay.org> References: <607068B0-E531-4D7F-8B61-923EE5DC443D@comcast.net> <C2D17751-17A9-4429-91ED-5E60B471B22F@iitbombay.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_69A9E170-4852-4398-9205-07FFF645B079 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 > On Aug 13, 2024, at 9:15=E2=80=AFAM, Bakul Shah <bakul@iitbombay.org> = wrote: >=20 > This weird 127. address seems like a systemd feature/bug thing: = https://unix.stackexchange.com/questions/612416/why-does-etc-resolv-conf-p= oint-at-127-0-0-53 >=20 > This behavior seems like some strange interaction between systemd = assumptions and freebsd=E2=80=99s, or something not being set up quite = right on the linux side when the vm is running freebsd.=20 Could libvirt be a factor here, do you think? For example, perhaps the network should be configured differently than the default when the host is using systemd-resolved and/or when the guest is FreeBSD. In the = network XML format for libvirt (https://libvirt.org/formatnetwork.html), there = is a `domain` element with a `localOnly` attribute that I have seen set by some virtualization projects. As far as I can tell, our setup isn't = using the `domain` element at all. >=20 >> On Aug 13, 2024, at 8:46 AM, Alex Arslan <ararslan@comcast.net> = wrote: >>=20 >> =EF=BB=BF >> Hi Rodney, >>=20 >>> On Aug 10, 2024, at 9:11=E2=80=AFAM, Rodney W. Grimes = <freebsd-rwg@gndrsh.dnsmgr.net> wrote: >>>=20 >>>>=20 >>>>=20 >>>>> On Aug 2, 2024, at 5:58?PM, Bakul Shah <bakul@iitbombay.org> = wrote: >>>>>=20 >>>>> On Aug 2, 2024, at 3:52?PM, Alex Arslan <ararslan@comcast.net> = wrote: >>>>>>=20 >>>>>>> Just a comment and a name server line: >>>>>>>=20 >>>>>>> $ cat /etc/resolv.conf >>>>>>> # Generated by resolvconf >>>>>>> nameserver 192.168.122.1 >>>>>>=20 >>>>>> I believe that is the host IP, so I guess the VM is using the = host for DNS >>>>>> resolution? Interestingly, if I add `nameserver 8.8.8.8` below = the line >>>>>> with the host IP, it takes 10 seconds rather than 30 to reach the = expected >>>>>> domain resolution failure. If I put 8.8.8.8 above the host IP, = the domain >>>>>> resolution failure is instantaneous. >>>>>=20 >>>>> What does your host use as a namesever? >>>>=20 >>>> The nameserver is 127.0.0.53. It sets options edns0 and trust-ad, = and >>>> includes a search entry as well. >>>=20 >>> First, is that a typo and you mean 127.0.0.1:53? >>=20 >> No, the host's /etc/resolv.conf has `nameserver 127.0.0.53`, I just = went >> back and rechecked to be sure. >>=20 >>> Second, is that name server locked to 127.0.0.1, or is it >>> actually listinging on *:53? If it is LOCKED you have no name = server >>> running on 192.168.122.1 to be reached by the VM, if it is NOT = locked >>> can the guest ping 192.168.122.1, and can it reach dns at that IP on >>> port 53? Can the host send a packet BACK to the guest? >>=20 >> I apologize but I don't really know enough about these things to know = how >> to answer your question. I did post the output of tcpdump on the VM = and >> the host a while back but that was for the invalid request, so that >> probably doesn't capture what you're describing. >>=20 >>> Third you can "fix" the "nameserver 192.168.122.1" entry in = /etc/resolv.conf >>> by configuring the DHCP server that handed out the lease to the VM = to send >>> a namserver entry of 8.8.8.8. >>=20 >> If I understand correctly, that is indeed what we've done as a = Band-Aid fix >> for the time being: I added the line `prepend_nameservers=3D8.8.8.8` = to >> /etc/resolvconf.conf. >>=20 >>>>=20 >>>>>=20 >>>>>> Not a particularly satisfying conclusion to this saga as I don't = understand >>>>>> why it's happening but at least I have a workaround that should = hopefully >>>>>> do the job. I really appreciate everyone's help and input thus = far! >>>>>>=20 >>>>>> What's the best way to add `nameserver 8.8.8.8` to = /etc/resolv.conf as >>>>>> part of the VM's configuration? >>>>>=20 >>>>> You should diagnose the problem of the nameserver at 192.168.122.1 >>>>> and fix it to act properly. I don't use vm (just bhyve) so can't = help >>>>> you with its config. >>>>=20 >>>> I do still plan to try to figure out what the actual issue is, but = I also >>>> now have a path forward in the meantime. :) >>>>=20 >>>>=20 >>>=20 >>> --=20 >>> Rod Grimes = rgrimes@freebsd.org <mailto:rgrimes@freebsd.org> --Apple-Mail=_69A9E170-4852-4398-9205-07FFF645B079 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dutf-8"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: = after-white-space;"><br><div><blockquote type=3D"cite"><div>On Aug 13, = 2024, at 9:15=E2=80=AFAM, Bakul Shah <bakul@iitbombay.org> = wrote:</div><br class=3D"Apple-interchange-newline"><div><meta = http-equiv=3D"content-type" content=3D"text/html; charset=3Dutf-8"><div = dir=3D"auto"><div dir=3D"ltr"></div><div dir=3D"ltr"><div = style=3D"display: block;">This weird 127. address seems like a systemd = feature/bug thing: <a = href=3D"https://unix.stackexchange.com/questions/612416/why-does-etc-resol= v-conf-point-at-127-0-0-53">https://unix.stackexchange.com/questions/61241= 6/why-does-etc-resolv-conf-point-at-127-0-0-53</a></div><div = style=3D"display: block;"><br></div><div style=3D"display: block;">This = behavior seems like some strange interaction between systemd assumptions = and freebsd=E2=80=99s, or something not being set up quite right on the = linux side when the vm is running = freebsd. </div></div></div></div></blockquote><div><br></div><div><di= v>Could libvirt be a factor here, do you think? For example, perhaps = the</div><div>network should be configured differently than the default = when the host</div><div>is using systemd-resolved and/or when the guest = is FreeBSD. In the network</div><div>XML format for libvirt = (https://libvirt.org/formatnetwork.html), there is</div><div>a `domain` = element with a `localOnly` attribute that I have seen set = by</div><div>some virtualization projects. As far as I can tell, our = setup isn't using</div><div>the `domain` element at = all.</div></div><br><blockquote type=3D"cite"><div><div dir=3D"auto"><div = dir=3D"ltr"><div style=3D"display: block;"><br></div></div><div = dir=3D"ltr"><blockquote type=3D"cite">On Aug 13, 2024, at 8:46 AM, Alex = Arslan <ararslan@comcast.net> = wrote:<br><br></blockquote></div><blockquote type=3D"cite"><div = dir=3D"ltr">=EF=BB=BF<meta http-equiv=3D"content-type" = content=3D"text/html; charset=3Dutf-8"><div dir=3D"auto" = style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; = line-break: after-white-space;">Hi Rodney,<br = id=3D"lineBreakAtBeginningOfMessage"><div><br><blockquote = type=3D"cite"><div>On Aug 10, 2024, at 9:11=E2=80=AFAM, Rodney W. Grimes = <freebsd-rwg@gndrsh.dnsmgr.net> wrote:</div><br = class=3D"Apple-interchange-newline"><div><meta = charset=3D"UTF-8"><blockquote type=3D"cite" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br><br><blockquote type=3D"cite">On Aug 2, = 2024, at 5:58?PM, Bakul Shah <bakul@iitbombay.org> = wrote:<br><br>On Aug 2, 2024, at 3:52?PM, Alex Arslan = <ararslan@comcast.net> wrote:<br><blockquote = type=3D"cite"><br><blockquote type=3D"cite">Just a comment and a name = server line:<br><br>$ cat /etc/resolv.conf<br># Generated by = resolvconf<br>nameserver 192.168.122.1<br></blockquote><br>I believe = that is the host IP, so I guess the VM is using the host for = DNS<br>resolution? Interestingly, if I add `nameserver 8.8.8.8` below = the line<br>with the host IP, it takes 10 seconds rather than 30 to = reach the expected<br>domain resolution failure. If I put 8.8.8.8 above = the host IP, the domain<br>resolution failure is = instantaneous.<br></blockquote><br>What does your host use as a = namesever?<br></blockquote><br>The nameserver is 127.0.0.53. It sets = options edns0 and trust-ad, and<br>includes a search entry as = well.<br></blockquote><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">First, = is that a typo and you mean 127.0.0.1:53?</span><br style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"></div></blockquote><div><br></div><div><div>No, = the host's /etc/resolv.conf has `nameserver 127.0.0.53`, I just = went</div><div>back and rechecked to be sure.</div></div><br><blockquote = type=3D"cite"><div><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">Second, = is that name server locked to 127.0.0.1, or is it</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">actually listinging on *:53? If it is = LOCKED you have no name server</span><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">running = on 192.168.122.1 to be reached by the VM, if it is NOT locked</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">can the guest ping 192.168.122.1, and can = it reach dns at that IP on</span><br style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">port = 53? Can the host send a packet BACK to the guest?</span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: = none;"></div></blockquote><div><div><br></div><div>I apologize but I = don't really know enough about these things to know how</div><div>to = answer your question. I did post the output of tcpdump on the VM = and</div><div>the host a while back but that was for the invalid = request, so that</div><div>probably doesn't capture what you're = describing.</div></div><br><blockquote type=3D"cite"><div><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">Third you can "fix" the "nameserver = 192.168.122.1" entry in /etc/resolv.conf</span><br style=3D"caret-color: = rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: = normal; font-variant-caps: normal; font-weight: 400; letter-spacing: = normal; text-align: start; text-indent: 0px; text-transform: none; = white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">by = configuring the DHCP server that handed out the lease to the VM to = send</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"><span style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none; float: none; display: inline !important;">a namserver entry of = 8.8.8.8.</span><br style=3D"caret-color: rgb(0, 0, 0); font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; text-align: start; = text-indent: 0px; text-transform: none; white-space: normal; = word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: = none;"></div></blockquote><div><br></div><div>If I understand correctly, = that is indeed what we've done as a Band-Aid fix</div><div>for the time = being: I added the line `prepend_nameservers=3D8.8.8.8` = to</div><div>/etc/resolvconf.conf.</div><br><blockquote = type=3D"cite"><div><blockquote type=3D"cite" style=3D"font-family: = Helvetica; font-size: 12px; font-style: normal; font-variant-caps: = normal; font-weight: 400; letter-spacing: normal; orphans: auto; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><br><blockquote type=3D"cite"><br><blockquote = type=3D"cite">Not a particularly satisfying conclusion to this saga as I = don't understand<br>why it's happening but at least I have a workaround = that should hopefully<br>do the job. I really appreciate everyone's help = and input thus far!<br><br>What's the best way to add `nameserver = 8.8.8.8` to /etc/resolv.conf as<br>part of the VM's = configuration?<br></blockquote><br>You should diagnose the problem of = the nameserver at 192.168.122.1<br>and fix it to act properly. I don't = use vm (just bhyve) so can't help<br>you with its = config.<br></blockquote><br>I do still plan to try to figure out what = the actual issue is, but I also<br>now have a path forward in the = meantime. :)<br><br><br></blockquote><br style=3D"caret-color: rgb(0, 0, = 0); font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none;"><span style=3D"caret-color: rgb(0, 0, 0); = font-family: Helvetica; font-size: 12px; font-style: normal; = font-variant-caps: normal; font-weight: 400; letter-spacing: normal; = text-align: start; text-indent: 0px; text-transform: none; white-space: = normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; = text-decoration: none; float: none; display: inline !important;">--<span = class=3D"Apple-converted-space"> </span></span><br = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;"><span = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none; float: none; = display: inline !important;">Rod Grimes = &n= bsp; &nbs= p; = </span><= a href=3D"mailto:rgrimes@freebsd.org" style=3D"font-family: Helvetica; = font-size: 12px; font-style: normal; font-variant-caps: normal; = font-weight: 400; letter-spacing: normal; orphans: auto; text-align: = start; text-indent: 0px; text-transform: none; white-space: normal; = widows: auto; word-spacing: 0px; -webkit-text-stroke-width: = 0px;">rgrimes@freebsd.org</a></div></blockquote></div><br></div></div></bl= ockquote></div></div></blockquote></div><br></body></html>= --Apple-Mail=_69A9E170-4852-4398-9205-07FFF645B079--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B830418F-C9A4-4B59-A5E9-0A1E7BE1CC78>