From nobody Wed Aug 14 18:29:38 2024 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4WkcHX2Rn2z5SbxL for ; Wed, 14 Aug 2024 18:30:00 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (pdx.rh.CN85.dnsmgr.net [65.75.216.6]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4WkcHW5wCFz41Px for ; Wed, 14 Aug 2024 18:29:59 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Authentication-Results: mx1.freebsd.org; none Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id 47EITcTh080533; Wed, 14 Aug 2024 11:29:38 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id 47EITc7B080532; Wed, 14 Aug 2024 11:29:38 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <202408141829.47EITc7B080532@gndrsh.dnsmgr.net> Subject: Re: Diagnosing virtual machine network issues In-Reply-To: To: Alex Arslan Date: Wed, 14 Aug 2024 11:29:38 -0700 (PDT) CC: Bakul Shah , "Rodney W. Grimes" , FreeBSD Hackers X-Mailer: ELM [version 2.4ME+ PL121h (25)] List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@FreeBSD.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Spamd-Bar: ---- X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:10494, ipnet:65.75.216.0/23, country:US] X-Rspamd-Queue-Id: 4WkcHW5wCFz41Px > > > On Aug 13, 2024, at 9:15?AM, Bakul Shah wrote: > > > > This weird 127. address seems like a systemd feature/bug thing: https://unix.stackexchange.com/questions/612416/why-does-etc-resolv-conf-point-at-127-0-0-53 > > > > This behavior seems like some strange interaction between systemd assumptions and freebsd?s, or something not being set up quite right on the linux side when the vm is running freebsd. > > Could libvirt be a factor here, do you think? For example, perhaps the > network should be configured differently than the default when the host > is using systemd-resolved and/or when the guest is FreeBSD. In the network > XML format for libvirt (https://libvirt.org/formatnetwork.html), there is > a `domain` element with a `localOnly` attribute that I have seen set by > some virtualization projects. As far as I can tell, our setup isn't using > the `domain` element at all. Having a /etc/resolv.conf entry of 127.0.0.53 is indeed something out of the normal on a freebsd box. You need to find where that is coming from and why that value is used. > > > > >> On Aug 13, 2024, at 8:46 AM, Alex Arslan wrote: > >> > >> ? > >> Hi Rodney, > >> > >>> On Aug 10, 2024, at 9:11?AM, Rodney W. Grimes wrote: > >>> > >>>> > >>>> > >>>>> On Aug 2, 2024, at 5:58?PM, Bakul Shah wrote: > >>>>> > >>>>> On Aug 2, 2024, at 3:52?PM, Alex Arslan wrote: > >>>>>> > >>>>>>> Just a comment and a name server line: > >>>>>>> > >>>>>>> $ cat /etc/resolv.conf > >>>>>>> # Generated by resolvconf > >>>>>>> nameserver 192.168.122.1 > >>>>>> > >>>>>> I believe that is the host IP, so I guess the VM is using the host for DNS > >>>>>> resolution? Interestingly, if I add `nameserver 8.8.8.8` below the line > >>>>>> with the host IP, it takes 10 seconds rather than 30 to reach the expected > >>>>>> domain resolution failure. If I put 8.8.8.8 above the host IP, the domain > >>>>>> resolution failure is instantaneous. > >>>>> > >>>>> What does your host use as a namesever? > >>>> > >>>> The nameserver is 127.0.0.53. It sets options edns0 and trust-ad, and > >>>> includes a search entry as well. > >>> > >>> First, is that a typo and you mean 127.0.0.1:53? > >> > >> No, the host's /etc/resolv.conf has `nameserver 127.0.0.53`, I just went > >> back and rechecked to be sure. > >> > >>> Second, is that name server locked to 127.0.0.1, or is it > >>> actually listinging on *:53? If it is LOCKED you have no name server > >>> running on 192.168.122.1 to be reached by the VM, if it is NOT locked > >>> can the guest ping 192.168.122.1, and can it reach dns at that IP on > >>> port 53? Can the host send a packet BACK to the guest? > >> > >> I apologize but I don't really know enough about these things to know how > >> to answer your question. I did post the output of tcpdump on the VM and > >> the host a while back but that was for the invalid request, so that > >> probably doesn't capture what you're describing. > >> > >>> Third you can "fix" the "nameserver 192.168.122.1" entry in /etc/resolv.conf > >>> by configuring the DHCP server that handed out the lease to the VM to send > >>> a namserver entry of 8.8.8.8. > >> > >> If I understand correctly, that is indeed what we've done as a Band-Aid fix > >> for the time being: I added the line `prepend_nameservers=8.8.8.8` to > >> /etc/resolvconf.conf. > >> > >>>> > >>>>> > >>>>>> Not a particularly satisfying conclusion to this saga as I don't understand > >>>>>> why it's happening but at least I have a workaround that should hopefully > >>>>>> do the job. I really appreciate everyone's help and input thus far! > >>>>>> > >>>>>> What's the best way to add `nameserver 8.8.8.8` to /etc/resolv.conf as > >>>>>> part of the VM's configuration? > >>>>> > >>>>> You should diagnose the problem of the nameserver at 192.168.122.1 > >>>>> and fix it to act properly. I don't use vm (just bhyve) so can't help > >>>>> you with its config. > >>>> > >>>> I do still plan to try to figure out what the actual issue is, but I also > >>>> now have a path forward in the meantime. :) > >>>> > >>>> > >>> > >>> -- > >>> Rod Grimes rgrimes@freebsd.org > -- Rod Grimes rgrimes@freebsd.org