From owner-freebsd-net@freebsd.org Thu Jan 21 17:04:28 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C565A8CDAF for ; Thu, 21 Jan 2016 17:04:28 +0000 (UTC) (envelope-from ncrogers@gmail.com) Received: from mail-oi0-x234.google.com (mail-oi0-x234.google.com [IPv6:2607:f8b0:4003:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C795A13CE for ; Thu, 21 Jan 2016 17:04:27 +0000 (UTC) (envelope-from ncrogers@gmail.com) Received: by mail-oi0-x234.google.com with SMTP id o124so30226100oia.3 for ; Thu, 21 Jan 2016 09:04:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=kCIiy9Y9OBLldJAslUowB1x6CtdBhXP5KQsscw35dLM=; b=gyi4lq6iNa3+yStm4iXYVsAEft/6TUfDhe2YsRXZUA1rBfvSE76dzC2FnW4niHrSfv b3jMbcl2IgpoUF/QzqCjUQjQy6mli6VNGDLZU0HiuV9aOR39pX28Z9b2sGS6wq09ToWD WLjz5Y9LeKaw5Vf3CqOjX5KD8HwBeSRb7KGEkPXQqJfIPkROUsTg7kR9nq98HE8dQVtk HXMFtTl4hsbddDrgvBBwhN1B4YpdzelAt0xc1zd+fqgGpzqyWsMVx5aO8UVSwa3aBo+9 FlJG8hK1JHwmF7EgP3p6NJuMXCB0bLGK1e9jCPGp9lxr9y5+5VrL26SKmdoA4NqEzT8M 8xkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=kCIiy9Y9OBLldJAslUowB1x6CtdBhXP5KQsscw35dLM=; b=fWaVQi6Jbytt/Ytcw5WA8EZpuyi9IbKtCsLY9VEsOWi1nhEBAdamu0IjegjcMcNsDw BfyX1zngO2+D471tq0EIcv6qK0dwKdUmUgC2llpzR0jGMn/quFDiYLtCeTHNscdGhTSL Q86IuMUgkrl7YZJhIcxBKsa/0qV79jsIxbS+LOlbfzlbukcA8HDuxeYp9wpf/9Yfvk/4 kIvnCi/1A+A378EdLNBaxqVNq0RZKdphQ6K+NGJHIYlRQK7IZrrrJofEWUXOg82fJBcF LG7edtGK3HPhTQdwYIVikZBR8Lq2MtA1TYscba+wUYlaA2CgLRcTp8wct2xN0JJitFcr mP+g== X-Gm-Message-State: ALoCoQlfz29+TaRM16ot4+/1a20sNLAXT78G59D8RdYZcFtMcylVf9+Y2Q5IyS25E95U2ElxxiLWH7QBBL6Lx7lxCm56qc57Bg== MIME-Version: 1.0 X-Received: by 10.202.217.5 with SMTP id q5mr29845360oig.29.1453395867152; Thu, 21 Jan 2016 09:04:27 -0800 (PST) Received: by 10.202.242.132 with HTTP; Thu, 21 Jan 2016 09:04:27 -0800 (PST) In-Reply-To: <56A003B8.9090104@shrew.net> References: <56A003B8.9090104@shrew.net> Date: Thu, 21 Jan 2016 09:04:27 -0800 Message-ID: Subject: Re: pf state disappearing From: Nick Rogers To: Matthew Grooms Cc: "freebsd-net@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Jan 2016 17:04:28 -0000 On Wed, Jan 20, 2016 at 2:01 PM, Matthew Grooms wrote: > All, > > I have a curious problem with a lightly loaded pair of pf firewall running > on FreeBSD 10.2-RELEASE. I'm noticing TCP entries are disappearing from > the state table for no good reason that I can see. The entry limit is set > to 100000 and I never see the system go over about 70000 entries, so we > shouldn't be hitting the configured limit ... > In my experience if you hit the state limit, new connections/states are dropped and existing states are unaffected. > > # pfctl -sm > states hard limit 100000 > src-nodes hard limit 100000 > frags hard limit 50000 > table-entries hard limit 200000 > > # pfctl -si > Status: Enabled for 78 days 14:24:18 Debug: Urgent > > State Table Total Rate > current entries 67829 > searches 113412118733 16700.2/s > inserts 386313496 56.9/s > removals 386245667 56.9/s > Counters > match 441731678 65.0/s > bad-offset 0 0.0/s > fragment 1090 0.0/s > short 220 0.0/s > normalize 761 0.0/s > memory 0 0.0/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 4366487 0.6/s > proto-cksum 0 0.0/s > state-mismatch 50334 0.0/s > state-insert 10 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > This problem is easy to reproduce by establishing an SSH connection to the > firewall itself, letting it sit for a while and then examining the state > table. After a connection is made, I can see the entry with an > established:established state ... > > # pfctl -ss | grep X.X.X.X | grep 63446 > all tcp Y.Y.Y.Y:22 <- X.X.X.X:63446 ESTABLISHED:ESTABLISHED > > If I let the SSH session sit for a while and then try to type into the > terminal on the client end, the connection stalls and produces a network > error message. When I look at the pf state table again, the state entry for > the connection is no longer visible. However, the ssh process is still > running and I still see the TCP connection established in the output of > netstat ... > > # netstat -na | grep 63446 > tcp4 0 0 Y.Y.Y.Y.22 X.X.X.X.63446 ESTABLISHED > > When I observe the packet flow in TCP dump when a connection stalls, > packets being sent from the client are visible on the physical interface > but are shown as blocked on the pflog0 interface. > Does this happen with non-SSH connections? It sounds like your SSH client/server interaction is not performing a keep-alive frequently enough to keep the PF state established. If no packets are sent over the connection (state) for some time, then PF will timeout (remove) the state. At this point your SSH client still believes it has a successful connection, so it tries to send packets when you resume typing, but they are blocked by your PF rules which likely specify "flags S/SA keep state", either explicitly or implicitly (it is the filter rule default), which means block packets that don't match an existing state or are not part of the initial SYN handshake of the TCP connection. Look at your settings in pf.conf for "timeout tcp.established", which affects how long before an idle ESTABLISHED state will timeout. Also look into ClientAliveInterval in sshd configuration, which I believe is 0 (disabled) by default, which means it will let the client timeout without sending a keep-alive. If you don't want PF to force timeout an idle SSH connection, then ideally ClientAliveInterval is less than or equal (i.e., more-frequent) to PF's tcp.established timeout value. > All this points to a state table entry being evicted from the state table > for a healthy TCP connection, but I have no idea why. Is there a secondary > resource limit I could be hitting that would cause the state entry to be > removed? Maybe there was a bug has been fixed recently that would cause > this behavior? I'd be very grateful for any input that would help me track > down or resolve this problem. > > Thanks in advance, > > -Matthew > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >