From owner-freebsd-pf@freebsd.org Fri Oct 2 17:35:40 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7B71F432998 for ; Fri, 2 Oct 2020 17:35:40 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lf1-x141.google.com (mail-lf1-x141.google.com [IPv6:2a00:1450:4864:20::141]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C2xwW3hJSz4RYs for ; Fri, 2 Oct 2020 17:35:39 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lf1-x141.google.com with SMTP id 77so2848707lfj.0 for ; Fri, 02 Oct 2020 10:35:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=VT3lXANOtM5gKp6EUBNU5rZRmDpqC4GFom33G1MVjms=; b=qwIRwj1HOQQdFwOtmx0Bw3w6FNHhjlpL3RSxs9j9dGC7xwBR4jfhS/xss8oknEC6as hVSZ49w7xX/XQhpnEH+xkdyKfFdK3Y98btjJMp3hZ8EBPOoqnApfjg3l/+SdlqNwLuW2 5x9QlsxTd7SADtmeXgU5iYdsCr0QJXOwy90haQ/huOd0QgmQqJLXKkb9Jr4BSXcXttUX ypglhRJi6U1/MkjQHz61vd0zyL4klqPwyE5ck7FNvAEsUVD/Seha2OJVkB0GlR29mq1H ywuPIgdWyhYjggLc4RBkgiscZUItSFFSfUCb05WgDIKTpwI/j13lnnNRrLeXVXBoeap4 lAcw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=VT3lXANOtM5gKp6EUBNU5rZRmDpqC4GFom33G1MVjms=; b=qLllYzREOsU9AdmY714CG6nQqJLq+I2YqhzT/bZ0AVeVNe+uKb1Tc4hZUwsIQCaPqE B/QP8Bh2KGdNpasZ+BoEV7tGsa1yqEusMiclnAsUa1qTKweSYBNWNu23bfKgl5qeDmd6 /YV3CF6ZEVxRvTkTkvIU3i3EcpqH1LCuMDBwcGQ6UcV3idcwwxm2y5fIwq7cVPbaoMNu 3SygZbprkJwp9xqp1WhpQIUmLgTNRKuxgoAERTMmodDVuNHS5cY4XQpHXS+zdHUIS7HV g9QjiYWKkIwHyIcOX886HZ4cTKUKnSgnGTr+5jxhLJhv+mfNlmZFpyjGVWOXa6xPJfJI trzw== X-Gm-Message-State: AOAM5319OrdSkqUoYhnEfs3GAlxBcTdZQvX+1Bn5NY17ncGdsFX9UZHk K042f4ERRIvc6HZ1n4MhhZoHef37CbixOfk/vIrL2FOg2yQ= X-Google-Smtp-Source: ABdhPJyJ+WMy+vMr7uINchEc8eCHSK+Qil4/YunaI5JZruavl9xf0iAVmaudLzGvXLyGR9Y2llGaNhNLG020xSRfVcA= X-Received: by 2002:ac2:548d:: with SMTP id t13mr1336319lfk.602.1601660137173; Fri, 02 Oct 2020 10:35:37 -0700 (PDT) MIME-Version: 1.0 From: J David Date: Fri, 2 Oct 2020 13:35:26 -0400 Message-ID: Subject: Packets passed by pf don't make it out? To: freebsd-pf@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4C2xwW3hJSz4RYs X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=qwIRwj1H; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2a00:1450:4864:20::141 as permitted sender) smtp.mailfrom=jdavidlists@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.04)[-0.040]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.960]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-pf@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::141:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-pf] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Oct 2020 17:35:40 -0000 We have pf running on a FreeBSD 11.4 system acting as a load balancer, mapping a set of 8 external DNS service IP addresses to a set of internal DNS servers, any of which can handle those requests. When UDP packets from one source IP/port arrive for multiple external IPs in a short period of time, pf claims they all pass, but only the ones for the first IP actually make it out the outbound interface. Redirect rule: rdr inet proto udp from any to { 172.17.53.1, 172.17.53.2, 172.17.53.3, 172.17.53.4, 172.17.53.5, 172.17.53.6, 172.17.53.7, 172.17.53.8 } port 53 -> { 10.53.0.1, 10.53.0.2, 10.53.0.3 } round-robin sticky-address Pass rule: pass in log quick proto udp to { 10.53.0.1, 10.53.0.2, 10.53.0.3 } port 53 (The pass rule isn't technically necessary, it's only there to log the packets to debug this issue.) With tcpdumps running simultaneously on ix1, all packets show up the inbound interface: 16:32:39.183168 IP 149.20.1.48.56246 > 172.17.53.1.53: 3215 SOA? example.com. (29) 16:32:39.183761 IP 149.20.1.48.56246 > 172.17.53.2.53: 2934 SOA? example.com. (29) 16:32:39.184368 IP 149.20.1.48.56246 > 172.17.53.3.53: 52875 SOA? example.com. (29) 16:32:39.185618 IP 149.20.1.48.56246 > 172.17.53.4.53: 36289 SOA? example.com. (29) 16:32:39.186067 IP 149.20.1.48.56246 > 172.17.53.5.53: 44049 SOA? example.com. (29) 16:32:39.186422 IP 149.20.1.48.56246 > 172.17.53.6.53: 34410 SOA? example.com. (29) 16:32:39.186494 IP 149.20.1.48.56246 > 172.17.53.7.53: 30923 SOA? example.com. (29) 16:32:39.188541 IP 149.20.1.48.56246 > 172.17.53.8.53: 48814 SOA? example.com. (29) and on pflog0: 16:32:39.183189 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > 10.53.0.1.53: 3215 SOA? example.com. (29) 16:32:39.183780 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > 10.53.0.1.53: 2934 SOA? example.com. (29) 16:32:39.184375 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > 10.53.0.1.53: 52875 SOA? example.com. (29) 16:32:39.185625 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > 10.53.0.1.53: 36289 SOA? example.com. (29) 16:32:39.186074 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > 10.53.0.1.53: 44049 SOA? example.com. (29) 16:32:39.186425 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > 10.53.0.1.53: 34410 SOA? example.com. (29) 16:32:39.186499 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > 10.53.0.1.53: 30923 SOA? example.com. (29) 16:32:39.188548 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > 10.53.0.1.53: 48814 SOA? example.com. (29) but only the first one appears on ix0, the outbound interface: 16:32:39.183211 IP 149.20.1.48.56246 > 10.53.0.1.53: 3215 SOA? example.com. (29) The actual query order is random, so if the test is repeated a minute later, then 172.17.53.3 might be hit first, and then that one will make it through and the rest will disappear. So it is not specific to any destination IP. It also only appears to occur when the UDP source port is the same across the connections. (This is probably why TCP is not affected.) It does not appear related to state entries ("no state") doesn't help. If "sticky-address" is removed from the rdr, then one packet will make it through for each backend IP, instead of one total. What could be causing this? It seems somehow related to 5-tuple non-uniqueness after the rdr, but that shouldn't be an issue for UDP; it should be treated as two connectionless packets from the same source for the same destination. (The query test source, 149.20.1.48 is an EDNS checker found at https://dnsflagday.net/2020/ .) Thanks for any advice!