Date: Fri, 2 Oct 2020 13:35:26 -0400 From: J David <j.david.lists@gmail.com> To: freebsd-pf@freebsd.org Subject: Packets passed by pf don't make it out? Message-ID: <CABXB=RSO2UDx2=LWx7W5SigYgJcaZ3vUTR0%2BVTDJUx2QezHK1Q@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
We have pf running on a FreeBSD 11.4 system acting as a load balancer,
mapping a set of 8 external DNS service IP addresses to a set of
internal DNS servers, any of which can handle those requests.
When UDP packets from one source IP/port arrive for multiple external
IPs in a short period of time, pf claims they all pass, but only the
ones for the first IP actually make it out the outbound interface.
Redirect rule:
rdr inet proto udp from any to { 172.17.53.1, 172.17.53.2,
172.17.53.3, 172.17.53.4, 172.17.53.5, 172.17.53.6, 172.17.53.7,
172.17.53.8 } port 53 -> { 10.53.0.1, 10.53.0.2, 10.53.0.3 }
round-robin sticky-address
Pass rule:
pass in log quick proto udp to { 10.53.0.1, 10.53.0.2, 10.53.0.3 } port 53
(The pass rule isn't technically necessary, it's only there to log the
packets to debug this issue.)
With tcpdumps running simultaneously on ix1, all packets show up the
inbound interface:
16:32:39.183168 IP 149.20.1.48.56246 > 172.17.53.1.53: 3215 SOA?
example.com. (29)
16:32:39.183761 IP 149.20.1.48.56246 > 172.17.53.2.53: 2934 SOA?
example.com. (29)
16:32:39.184368 IP 149.20.1.48.56246 > 172.17.53.3.53: 52875 SOA?
example.com. (29)
16:32:39.185618 IP 149.20.1.48.56246 > 172.17.53.4.53: 36289 SOA?
example.com. (29)
16:32:39.186067 IP 149.20.1.48.56246 > 172.17.53.5.53: 44049 SOA?
example.com. (29)
16:32:39.186422 IP 149.20.1.48.56246 > 172.17.53.6.53: 34410 SOA?
example.com. (29)
16:32:39.186494 IP 149.20.1.48.56246 > 172.17.53.7.53: 30923 SOA?
example.com. (29)
16:32:39.188541 IP 149.20.1.48.56246 > 172.17.53.8.53: 48814 SOA?
example.com. (29)
and on pflog0:
16:32:39.183189 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 >
10.53.0.1.53: 3215 SOA? example.com. (29)
16:32:39.183780 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 >
10.53.0.1.53: 2934 SOA? example.com. (29)
16:32:39.184375 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 >
10.53.0.1.53: 52875 SOA? example.com. (29)
16:32:39.185625 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 >
10.53.0.1.53: 36289 SOA? example.com. (29)
16:32:39.186074 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 >
10.53.0.1.53: 44049 SOA? example.com. (29)
16:32:39.186425 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 >
10.53.0.1.53: 34410 SOA? example.com. (29)
16:32:39.186499 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 >
10.53.0.1.53: 30923 SOA? example.com. (29)
16:32:39.188548 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 >
10.53.0.1.53: 48814 SOA? example.com. (29)
but only the first one appears on ix0, the outbound interface:
16:32:39.183211 IP 149.20.1.48.56246 > 10.53.0.1.53: 3215 SOA? example.com. (29)
The actual query order is random, so if the test is repeated a minute
later, then 172.17.53.3 might be hit first, and then that one will
make it through and the rest will disappear. So it is not specific to
any destination IP.
It also only appears to occur when the UDP source port is the same
across the connections. (This is probably why TCP is not affected.)
It does not appear related to state entries ("no state") doesn't help.
If "sticky-address" is removed from the rdr, then one packet will make
it through for each backend IP, instead of one total.
What could be causing this? It seems somehow related to 5-tuple
non-uniqueness after the rdr, but that shouldn't be an issue for UDP;
it should be treated as two connectionless packets from the same
source for the same destination.
(The query test source, 149.20.1.48 is an EDNS checker found at
https://dnsflagday.net/2020/ .)
Thanks for any advice!
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABXB=RSO2UDx2=LWx7W5SigYgJcaZ3vUTR0%2BVTDJUx2QezHK1Q>