Date: Thu, 06 Feb 2014 11:14:53 +0400 From: Dennis Yusupoff <dyr@smartspb.net> To: freebsd-net@freebsd.org Subject: PF states degrade? Message-ID: <52F3366D.3030202@smartspb.net>
next in thread | raw e-mail | index | archive | help
Good day. We had started to testing FreeBSD 10.0 in production (pf nat, ipfw pipes, ng_netflow) with setting (sysctl,pf.conf,ipfw.conf etc) from similar rocksolid 9.0-STABLE. Server has worked fine for a ~5 days and then suddenly stop forwarding traffic from clients. What was a quite unexpecting is how it had happening. Traffic from customers...dissappear (seen in tcpdump) from LAN interface in ~10 seconds after _connection_ (NAT translation state has been created?) has been started, with pf log (when set "log debug loud" in pf.conf) strange record appears in that moment, like that: 10.53.80.224 nat'ed in 109.71.177.147, http connection to 213.180.204.183: --- Feb 5 20:41:21 nata2 kernel: pf: State failure on: 1 | 5 Feb 5 20:41:21 nata2 kernel: pf: BAD state: TCP out wire: 213.180.204.183:80 Feb 5 20:41:21 nata2 kernel: 109.71.177.147:50114 stack: 213.180.204.183:80 10.53.80.224:50114 [lo=1997798965 high=1997799354 win=2772 modulator=0] Feb 5 20:41:21 nata2 kernel: [lo=864623348 high=864624718 win=389 modulator=0] 4:4 A seq=864739382 (864739382) ack=1997798965 len=1398 ackskew=0 pkts=3:2 dir=in,rev --- Full log there: http://pastebin.com/CQ78JyJe Disabling/enabling PF - no difference (except, indeed, nat stop working). After all attempts we did "pfctl -d" and setup ipfw nat for that customer. All has work fine! So we believe in uknown (for us) problem related to PF and it state work. PF rules and settings: --- ext_if="lagg0" int_if_1="vlan22" int_if_2="vlan21" dst_nat1="109.71.177.128/25" dst_nat2="109.71.177.0/25" table <src-nat> persist file "/etc/pf.src-nat" table <our-nets> const { 80.249.176.0/20, 93.92.192.0/21, 109.71.176.0/21, 217.119.16.0/20 } table <allowed-spammers> persist { 10.52.249.24 } table <always_allowed_dst> persist { 84.204.97.154, 213.180.204.32, 195.95.218.31, 195.95.218.30 } set limit { states 1000000, frags 80000, src-nodes 100000, table-entries 500000} set state-policy if-bound set optimization aggressive set debug urgent set ruleset-optimization profile set timeout { frag 10, tcp.established 3600, src.track 30 } set block-policy drop set require-order no set skip on {lo0, em0, pfsync0} table <spammers> persist pass in quick on $int_if_1 proto tcp from <allowed-spammers> to any port smtp flags S/SAFR keep state pass in quick on $int_if_2 proto tcp from <allowed-spammers> to any port smtp flags S/SAFR keep state pass in on $int_if_1 proto tcp from any to any port smtp flags S/SAFR keep state \ (max-src-conn 15, max-src-conn-rate 15/30, overload <spammers> flush global) block return-icmp (host-prohib) log quick proto tcp from <spammers> to any port smtp pass in on $int_if_2 proto tcp from any to any port smtp flags S/SAFR keep state \ (max-src-conn 15, max-src-conn-rate 15/30, overload <spammers> flush global) block return-icmp (host-prohib) log quick proto tcp from <spammers> to any port smtp pass in quick on $int_if_1 all no state allow-opts tag NAT1 label "$nr:NAT1" pass in quick on $int_if_2 all no state allow-opts tag NAT2 label "$nr:NAT2" binat-anchor "binat" load anchor "binat" from "/etc/pf.anchor.binat" nat-anchor "ftp-proxy/*" rdr-anchor "ftp-proxy/*" rdr pass on $int_if_1 proto tcp from <src-nat> to any port 21 -> 127.0.0.1 port 8021 rdr pass on $int_if_2 proto tcp from <src-nat> to any port 21 -> 127.0.0.1 port 8021 rdr pass on $ext_if proto udp from 109.71.176.3 to 109.71.176.2 port 4784 -> 10.78.76.2 port 4784 nat on $ext_if from <src-nat> to any tagged NAT1 -> $dst_nat1 static-port source-hash #sticky-address nat on $ext_if from <src-nat> to any tagged NAT2 -> $dst_nat2 static-port source-hash #sticky-address nat on $ext_if from any to <always_allowed_dst> -> $dst_nat1 static-port source-hash #sticky-address binat on $ext_if from 10.78.78.2 to any -> 93.92.199.252 nat on $ext_if from 10.78.76.0/24 to any -> 109.71.176.2 static-port source-hash nat on $ext_if from 10.78.77.0/24 to any -> 93.92.199.254 nat on $ext_if from 10.78.78.0/24 to any -> $dst_nat1 static-port source-hash anchor "ftp-proxy/*" pass out quick proto tcp from any to any port 21 no state pass quick on $ext_if proto gre all no state --- *P. S. Traffic start forwarding with pf only after server has been rebooted.* -- Best regards, Dennis Yusupoff, network engineer of Smart-Telecom ISP Russia, Saint-Petersburg
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52F3366D.3030202>