Date: Fri, 9 Oct 2020 20:24:54 -0400 From: J David <j.david.lists@gmail.com> To: freebsd-pf@freebsd.org Cc: freebsd-net@freebsd.org Subject: Re: Packets passed by pf don't make it out? Message-ID: <CABXB=RQE74yggCj6=Zizb2rQjtCi=hg155J0_u=NRK2Q3QHmqg@mail.gmail.com> In-Reply-To: <CABXB=RSO2UDx2=LWx7W5SigYgJcaZ3vUTR0%2BVTDJUx2QezHK1Q@mail.gmail.com> References: <CABXB=RSO2UDx2=LWx7W5SigYgJcaZ3vUTR0%2BVTDJUx2QezHK1Q@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
To investigate this issue, I've created a greatly simplified and reproducible test case. The code is available at: https://github.com/jdavidlists/pfudpbug It includes the pf.conf, the source code for client and server, and the rc.conf from all three machines. The test uses three FreeBSD systems (client, gateway, and server) to demonstrate that if a client with a bound UDP source port sends UDP packets to multiple server addresses that a gateway running pf redirects to one backend server, only the first such packet will be delivered. The remaining packets never leave the gateway; they get lost somewhere after being logged as passing pf via pflog0 but before a tcpdump running on the gateway's server-facing interface. They also do not increase the outbound packet count on the server-facing interface. (More detail is available in the repository's README.md.) This may indicate a bug, but I'm not sure whether it is happening inside pf or farther along the output path. Nor do I know how to explore this further. Is anyone able to point me in the right direction here? Thanks! On Fri, Oct 2, 2020 at 1:35 PM J David <j.david.lists@gmail.com> wrote: > > We have pf running on a FreeBSD 11.4 system acting as a load balancer, > mapping a set of 8 external DNS service IP addresses to a set of > internal DNS servers, any of which can handle those requests. > > When UDP packets from one source IP/port arrive for multiple external > IPs in a short period of time, pf claims they all pass, but only the > ones for the first IP actually make it out the outbound interface. > > Redirect rule: > > rdr inet proto udp from any to { 172.17.53.1, 172.17.53.2, > 172.17.53.3, 172.17.53.4, 172.17.53.5, 172.17.53.6, 172.17.53.7, > 172.17.53.8 } port 53 -> { 10.53.0.1, 10.53.0.2, 10.53.0.3 } > round-robin sticky-address > > Pass rule: > > pass in log quick proto udp to { 10.53.0.1, 10.53.0.2, 10.53.0.3 } port 53 > > (The pass rule isn't technically necessary, it's only there to log the > packets to debug this issue.) > > With tcpdumps running simultaneously on ix1, all packets show up the > inbound interface: > > 16:32:39.183168 IP 149.20.1.48.56246 > 172.17.53.1.53: 3215 SOA? > example.com. (29) > 16:32:39.183761 IP 149.20.1.48.56246 > 172.17.53.2.53: 2934 SOA? > example.com. (29) > 16:32:39.184368 IP 149.20.1.48.56246 > 172.17.53.3.53: 52875 SOA? > example.com. (29) > 16:32:39.185618 IP 149.20.1.48.56246 > 172.17.53.4.53: 36289 SOA? > example.com. (29) > 16:32:39.186067 IP 149.20.1.48.56246 > 172.17.53.5.53: 44049 SOA? > example.com. (29) > 16:32:39.186422 IP 149.20.1.48.56246 > 172.17.53.6.53: 34410 SOA? > example.com. (29) > 16:32:39.186494 IP 149.20.1.48.56246 > 172.17.53.7.53: 30923 SOA? > example.com. (29) > 16:32:39.188541 IP 149.20.1.48.56246 > 172.17.53.8.53: 48814 SOA? > example.com. (29) > > and on pflog0: > > 16:32:39.183189 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 3215 SOA? example.com. (29) > 16:32:39.183780 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 2934 SOA? example.com. (29) > 16:32:39.184375 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 52875 SOA? example.com. (29) > 16:32:39.185625 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 36289 SOA? example.com. (29) > 16:32:39.186074 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 44049 SOA? example.com. (29) > 16:32:39.186425 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 34410 SOA? example.com. (29) > 16:32:39.186499 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 30923 SOA? example.com. (29) > 16:32:39.188548 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 48814 SOA? example.com. (29) > > but only the first one appears on ix0, the outbound interface: > > 16:32:39.183211 IP 149.20.1.48.56246 > 10.53.0.1.53: 3215 SOA? example.com. (29) > > The actual query order is random, so if the test is repeated a minute > later, then 172.17.53.3 might be hit first, and then that one will > make it through and the rest will disappear. So it is not specific to > any destination IP. > > It also only appears to occur when the UDP source port is the same > across the connections. (This is probably why TCP is not affected.) > > It does not appear related to state entries ("no state") doesn't help. > > If "sticky-address" is removed from the rdr, then one packet will make > it through for each backend IP, instead of one total. > > What could be causing this? It seems somehow related to 5-tuple > non-uniqueness after the rdr, but that shouldn't be an issue for UDP; > it should be treated as two connectionless packets from the same > source for the same destination. > > (The query test source, 149.20.1.48 is an EDNS checker found at > https://dnsflagday.net/2020/ .) > > Thanks for any advice!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABXB=RQE74yggCj6=Zizb2rQjtCi=hg155J0_u=NRK2Q3QHmqg>