Date: Thu, 21 Jan 2016 09:04:27 -0800 From: Nick Rogers <ncrogers@gmail.com> To: Matthew Grooms <mgrooms@shrew.net> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: pf state disappearing Message-ID: <CAKOb=YakqYqeGYUh3PKm-PGQma7E69ZPAtAe7og3byN7s5d4SA@mail.gmail.com> In-Reply-To: <56A003B8.9090104@shrew.net> References: <56A003B8.9090104@shrew.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jan 20, 2016 at 2:01 PM, Matthew Grooms <mgrooms@shrew.net> wrote: > All, > > I have a curious problem with a lightly loaded pair of pf firewall running > on FreeBSD 10.2-RELEASE. I'm noticing TCP entries are disappearing from > the state table for no good reason that I can see. The entry limit is set > to 100000 and I never see the system go over about 70000 entries, so we > shouldn't be hitting the configured limit ... > In my experience if you hit the state limit, new connections/states are dropped and existing states are unaffected. > > # pfctl -sm > states hard limit 100000 > src-nodes hard limit 100000 > frags hard limit 50000 > table-entries hard limit 200000 > > # pfctl -si > Status: Enabled for 78 days 14:24:18 Debug: Urgent > > State Table Total Rate > current entries 67829 > searches 113412118733 16700.2/s > inserts 386313496 56.9/s > removals 386245667 56.9/s > Counters > match 441731678 65.0/s > bad-offset 0 0.0/s > fragment 1090 0.0/s > short 220 0.0/s > normalize 761 0.0/s > memory 0 0.0/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 4366487 0.6/s > proto-cksum 0 0.0/s > state-mismatch 50334 0.0/s > state-insert 10 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > This problem is easy to reproduce by establishing an SSH connection to the > firewall itself, letting it sit for a while and then examining the state > table. After a connection is made, I can see the entry with an > established:established state ... > > # pfctl -ss | grep X.X.X.X | grep 63446 > all tcp Y.Y.Y.Y:22 <- X.X.X.X:63446 ESTABLISHED:ESTABLISHED > > If I let the SSH session sit for a while and then try to type into the > terminal on the client end, the connection stalls and produces a network > error message. When I look at the pf state table again, the state entry for > the connection is no longer visible. However, the ssh process is still > running and I still see the TCP connection established in the output of > netstat ... > > # netstat -na | grep 63446 > tcp4 0 0 Y.Y.Y.Y.22 X.X.X.X.63446 ESTABLISHED > > When I observe the packet flow in TCP dump when a connection stalls, > packets being sent from the client are visible on the physical interface > but are shown as blocked on the pflog0 interface. > Does this happen with non-SSH connections? It sounds like your SSH client/server interaction is not performing a keep-alive frequently enough to keep the PF state established. If no packets are sent over the connection (state) for some time, then PF will timeout (remove) the state. At this point your SSH client still believes it has a successful connection, so it tries to send packets when you resume typing, but they are blocked by your PF rules which likely specify "flags S/SA keep state", either explicitly or implicitly (it is the filter rule default), which means block packets that don't match an existing state or are not part of the initial SYN handshake of the TCP connection. Look at your settings in pf.conf for "timeout tcp.established", which affects how long before an idle ESTABLISHED state will timeout. Also look into ClientAliveInterval in sshd configuration, which I believe is 0 (disabled) by default, which means it will let the client timeout without sending a keep-alive. If you don't want PF to force timeout an idle SSH connection, then ideally ClientAliveInterval is less than or equal (i.e., more-frequent) to PF's tcp.established timeout value. > All this points to a state table entry being evicted from the state table > for a healthy TCP connection, but I have no idea why. Is there a secondary > resource limit I could be hitting that would cause the state entry to be > removed? Maybe there was a bug has been fixed recently that would cause > this behavior? I'd be very grateful for any input that would help me track > down or resolve this problem. > > Thanks in advance, > > -Matthew > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAKOb=YakqYqeGYUh3PKm-PGQma7E69ZPAtAe7og3byN7s5d4SA>