From owner-freebsd-questions@freebsd.org Tue Mar 20 23:11:45 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41B62F5443B for ; Tue, 20 Mar 2018 23:11:45 +0000 (UTC) (envelope-from alexmiroslav@gmail.com) Received: from mail-wm0-x232.google.com (mail-wm0-x232.google.com [IPv6:2a00:1450:400c:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B54EA6F56A for ; Tue, 20 Mar 2018 23:11:44 +0000 (UTC) (envelope-from alexmiroslav@gmail.com) Received: by mail-wm0-x232.google.com with SMTP id h76so6516311wme.4 for ; Tue, 20 Mar 2018 16:11:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=iQsgJXL2i4DGB8LWVoI3SDXSOwTTbYR1SEQ1Nuqp+8Q=; b=aoU1Fg+ddI2T6rlBbRnvw9CtKHHyA9JbQimyNfk4txLoZgXicW9emNl3IfI8n7k7O4 qDG4y9GfKD2A7Tc7lcv60AZufYP7IW5MAKmc5pWXrMIbYNm1L8D7xS1a7x68txDRlhiH kH+2jQRuEmYkBqoPLbJhu6BOW/HeH2p3xWpS0imSSltYP9kK9lgxDQTLO4fgp8AVAhzO eTrz1/QALo6qBj0DOlk1x6m8qxnZfsUqcWsvuOi3gnFq87hgy6iYpdjJQ7H9lwUnsh6v Qk8+Q4wokiSx3JLDeB91t7wdbOd1yb21TGEudZekiXjAJ2kiG6QBOc6o3V+W7S7M9/kf xqkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=iQsgJXL2i4DGB8LWVoI3SDXSOwTTbYR1SEQ1Nuqp+8Q=; b=a1UmTp+8SQcS+uU0ZQzHbgltQVLbtSeHIUqYS2u5SncaykgNUy3gbF9IdzrbMXJWui 5QndMp04J1ok5wji8bB5QxzAAgLFoW82JCdYdUJJd9BSMQHOnB0otVR1SY5qFggBN944 opUoDRE4WaxUv7PDIzWihEvKzRRug1rQP3wAfZR57jVB0gzH9NuGPoEG/dUuYOMzO0Sc xeggMGaLvyKKxvRTz9M7WuEpGnFjgJnY/D273/KTWshwHXP9npvpvwtz/2YT5I18Ycg7 Zhk0jx0XPamK1v6mo2REssOePHPq/P8CT7Ds79nYFimE973u1TTWueitJWbrdYM1fn+f vEIA== X-Gm-Message-State: AElRT7GU54JPx3sofyhkmhwZ/Q1cLOVeQgKkOKwPoTPlCaW2xY+3kTJV p34bU7nz2btMPAC7xJ9uJlg8VenfdKxlTQD8ujwqkQ== X-Google-Smtp-Source: AG47ELtdw2R+OpVY7XRsvS9H7CvMs7QS0hJIMHn/Pyct5zgLYEVwtSiYRnJDDw6rLg7kPStycjhdbD1ZzphhYixY35Q= X-Received: by 10.28.50.69 with SMTP id y66mr1050623wmy.133.1521587503330; Tue, 20 Mar 2018 16:11:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.138.142 with HTTP; Tue, 20 Mar 2018 16:11:42 -0700 (PDT) From: Aleksandr Miroslav Date: Tue, 20 Mar 2018 16:11:42 -0700 Message-ID: Subject: weird network/DNS issues (nsd not returning answer) To: freebsd-questions@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Mar 2018 23:11:45 -0000 I have a number of FreeBSD servers online. The other day, one of them that I setup a month back started exhibiting really weird behavior. It doesn't get answers back to queries made to my two DNS servers, both of which are running nsd. Initially I suspected pf or sshguard to be the issue, but this happens with pf and sshguard turned off on all servers in question. The other weird thing is that all other network traffic between these servers are passing back and forth normally, only nsd replies are not being sent. Here is the issue, roughly: - given multiple servers, labeled, a-z - servers k and z run nsd - with the exception of server b, all other servers can communicate normally with servers k and z - with the exception of DNS queries, server b can communicate normally with server k and z - b can ping, ssh to, rsync, scp, to and from server k and z The only issue is when b makes a DNS query to k or z. I see those two servers get the query, and return the answer, but that answer never reaches b. I have sniffed the network to confirm this. Observe: # in these examples: # b.example.org = 66.66.66.66, the server that is misbehaving # k.example.org = 1.1.1.1, one of my DNS servers # c.example.org = 3.3.3.3, another server of mine, which I am looking up the DNS for # b make initially query to k 14:11:46.912995 IP 66.66.66.66.18394 > 1.1.1.1.53: 22479+ A? c.example.org. (31) # k receives query and immediately returns the answer 14:11:46.931605 IP 66.66.66.66.18394 > 1.1.1.1.53: 22479+ A? c.example.org. (31) 14:11:46.931854 IP 1.1.1.1.53 > 66.66.66.66.18394: 22479*- 1/2/1 A 3.3.3.3 (103) # this second line, the answer, never makes it to b # after a second or two, it makes another query: 14:11:51.969083 IP 66.66.66.66.12645 > 1.1.1.1.53: 22479+ A? c.example.org. (31) # k receives the second query and immediately returns the answer again 14:11:51.991267 IP 66.66.66.66.12645 > 1.1.1.1.53: 22479+ A? c.example.org. (31) 14:11:51.991508 IP 1.1.1.1.53 > 66.66.66.66.12645: 22479*- 1/2/1 A 3.3.3.3 (103) # there still nothing from tcpdump on b's interface that it received the answer # [DNS names and IPs have been changed above.] Here's what it looks like from b's command line $ host c.example.org k.example.org # a few seconds delay ;; connection timed out; no servers could be reached $ b has the same problem with my my other server z, which also runs nsd. All my other servers can query k and z just fine. Only b is exhibiting this problem. All the servers run pf/sshguard. But these rules/configs have not been updated in months. I did do one other thing to debug. I shutdown nsd on k, and setup a listener on b like this nc -l 10000 And on k, I did this: ls /etc | sudo nc -s 1.1.1.1 -p 53 b.example.org 10000 This produced the contents of /etc on b. So that means that without nsd in the picture, k is able to talk to b via port 53 just fine. All the above servers in question are running FreeBSD 11.1-RELEASE-p6. I'm not exactly sure how I can debug this problem further, I'm not sure where the block is happening. Any help appreciated.