Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 11 Oct 2020 13:59:58 -0400
From:      J David <j.david.lists@gmail.com>
To:        Andreas Longwitz <longwitz@incore.de>
Cc:        freebsd-pf@freebsd.org, freebsd-net@freebsd.org
Subject:   Re: Packets passed by pf don't make it out?
Message-ID:  <CABXB=RRdbDYyKfXUtyc9eW-P8eoX2nUb1A1Tn46MHWv5YNjT0g@mail.gmail.com>
In-Reply-To: <5F8336C7.5020709@incore.de>
References:  <CABXB=RSO2UDx2=LWx7W5SigYgJcaZ3vUTR0%2BVTDJUx2QezHK1Q@mail.gmail.com> <CABXB=RQE74yggCj6=Zizb2rQjtCi=hg155J0_u=NRK2Q3QHmqg@mail.gmail.com> <5F8336C7.5020709@incore.de>

next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Oct 11, 2020 at 12:46 PM Andreas Longwitz <longwitz@incore.de> wrote:
> Please look at the output of "pfctl -vsn" on fb2 during your test.
> With "netstat -ss | grep drop" you can check for packets dropped by the
> kernel for what reason ever.

Here's the complete diff of the output from netstat -ss from before to
after running the test:

--- nss.pre 2020-10-11 17:10:19.932002000 +0000
+++ nss.post 2020-10-11 17:10:21.999823000 +0000
@@ -48,9 +48,9 @@
  Packet drop statistics:
  Timeouts:
 ip:
- 66578 total packets received
+ 66582 total packets received
  66531 packets for this host
- 16 packets forwarded
+ 17 packets forwarded
  1 packet not forwardable
  31675 packets sent from this host
  10 packets sent with fabricated ip header

No drops of any kind (nor anything else) recorded during the test.

4 packets in, 1 packet forwarded, which exactly matches the observed
behavior of only one packet reaching the server.

The results of "pfctl -vsn" are a bit more interesting, and also inconsistent.

Before, after a full flush to zero states and counters:

rdr inet proto udp from any to 172.16.0.0/12 port = 12345 -> 10.255.255.3
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 1044 State Creations: 0     ]

After:

rdr inet proto udp from any to 172.16.0.0/12 port = 12345 -> 10.255.255.3
  [ Evaluations: 4         Packets: 1         Bytes: 44          States: 1     ]
  [ Inserted: uid 0 pid 1044 State Creations: 4     ]

So it says it created four states, but only matched one packet out of
the four it evaluated.  And it didn't create 4 states, either. "pfctl
-s state" shows only 1:

all udp 10.255.255.3:12345 (172.16.0.1:12345) <- 10.0.0.1:23456
NO_TRAFFIC:SINGLE

and pflog0 reported all four packets as matching the pass rule which,
important, is based on the destination address after redirection:

17:23:39.039641 rule 0/0(match): pass in on em1: 10.0.0.1.23456 >
10.255.255.3.12345: UDP, length 16
17:23:39.039751 rule 0/0(match): pass in on em1: 10.0.0.1.23456 >
10.255.255.3.12345: UDP, length 16
17:23:39.039769 rule 0/0(match): pass in on em1: 10.0.0.1.23456 >
10.255.255.3.12345: UDP, length 16
17:23:39.039780 rule 0/0(match): pass in on em1: 10.0.0.1.23456 >
10.255.255.3.12345: UDP, length 16

If I repeat the test, this happens:

rdr inet proto udp from any to 172.16.0.0/12 port = 12345 -> 10.255.255.3
  [ Evaluations: 7         Packets: 2         Bytes: 88          States: 1     ]
  [ Inserted: uid 0 pid 1044 State Creations: 7     ]

But still just the one state:

all udp 10.255.255.3:12345 (172.16.0.1:12345) <- 10.0.0.1:23456
NO_TRAFFIC:SINGLE

But only three passes appear in pflog0:

17:29:19.857174 rule 0/0(match): pass in on em1: 10.0.0.1.23456 >
10.255.255.3.12345: UDP, length 16
17:29:19.857193 rule 0/0(match): pass in on em1: 10.0.0.1.23456 >
10.255.255.3.12345: UDP, length 16
17:29:19.857226 rule 0/0(match): pass in on em1: 10.0.0.1.23456 >
10.255.255.3.12345: UDP, length 16

And, to confirm, the first packet from each of these tests did reach
the server, the remaining three from each test did not.

> A similar setup works for me without any problems, so there may be
> something special in your environment.

This has been tested on fresh installs of both FreeBSD 12.1 and 11.4
on both physical hardware and virtual machines including both Xeons
and AMD Epyc.  So it seems like most environmental factors have been
controlled for.

> It seems your routing table on fb2 is empty, please try to set a
> defaultroute, e.g.: "route add default 10.0.0.NN" with any NN.

fb2 does have a default route, it is obtained from DHCP on the first
interface.  But that is not relevant; the client machine (fb1) is
directly connected to fb2's second interface, and the server (fb3) is
directly connected to fb2's third interface.

No additional routes are necessary for this test, and the default
route is never consulted.

Perhaps there's some detail of the scenario that I have omitted
without which it's not clear?

Thanks!



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CABXB=RRdbDYyKfXUtyc9eW-P8eoX2nUb1A1Tn46MHWv5YNjT0g>