Date: Sat, 20 Jun 2015 18:24:32 +0200 From: Milan Obuch <freebsd-pf@dino.sk> To: Ian FREISLICH <ian.freislich@capeaugusta.com> Cc: freebsd-pf@freebsd.org Subject: Re: Large scale NAT with PF - some weird problem Message-ID: <20150620182432.62797ec5@zeta.dino.sk> In-Reply-To: <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com> References: <20150619091857.304b707b@zeta.dino.sk> <14e119e8fa8.2755.abfb21602af57f30a7457738c46ad3ae@capeaugusta.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a MIME-formatted message. If you see this text it means that your E-mail software does not support MIME-formatted messages. --=_mailhost.netlabit.sk-62017-1434817473-0001-2 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline On Sat, 20 Jun 2015 17:38:05 +0200 Ian FREISLICH <ian.freislich@capeaugusta.com> wrote: > Hi, > > How many NAT states in your table? > How can I find out? Is there another statistics collected I can gert out of pfctl? Full output of pfctl -vs info is attached (hopefully it will come unmangled through), is there anything interesting? > I had a router translating a /20 and a /22 to a /24 and doing > transparent interception of those and a /16 to a proxy pool and I > never saw this. My state table was about 380000 to 850000 with a > search rate about quadruple yours. > Did you do any pf tuning? What about limits in your case? > If you can, give 10-STABLE a try. I ran the above router pair as > 10-CURRENT for a long time. There are some significant performance > improvements. > It is possible, but not easy (read: needs some planning before switch) so as not interrupt the operation. > Ian > Thanks, Milan > On 19 June 2015 09:24:22 Milan Obuch <freebsd-pf@dino.sk> wrote: > > > Hi, > > > > I am managing FreeBSD 9 based router for a network using PF for > > NAT. I think I can call it large scale - there is approximately > > 3000 customers' devices (home routers and similar) with private IPs > > in segment 172.16.0.0/12 translated to /23 public address block. > > Basically, in pf.conf, there is > > > > nat on $if_ext from $net_int to any -> $pool_ext round-robin > > sticky-address > > > > and handful of > > > > binat on $if_ext from 172.16.x.y to any -> a.b.c.d > > > > statements. It works, basically, but for some time now there are > > some intermitent outages. When it occurs, customer's device loses > > access to internet. I can verify it with simple ping to any address > > outside of the network. > > > > The weird thing is, I can see icmp request packets coming out of > > external interface, but no icmp echo packets coming back. While I > > can't verify on uplink router that these replies are actually > > coming in on interface, I am pretty sure it does, but they are not > > visible in tcpdump's output. (When I am pinging some device outside > > of the network, which is under my control, I can see there both > > icmp requests and icmp echo packets. Also, if I ping address to > > which thich ping is translated from outside, I see it on external > > interface coming in.) > > > > I think I have a problem with same table being too small, but no > > idea where it is. It is not state table, I have > > > > set limit states 500000 > > > > in my pf.conf, and pfctl -vs info tells > > > > State Table Total Rate > > current entries 36668 > > searches 1996138369 29280.5/s > > inserts 15757727 231.1/s > > removals 15770004 231.3/s > > > > so I think I have plenty of room here. It was set in past when > > issue a bit similar occured and using bigger state table solved it. > > > > Also, pfctl -vs state | grep <ip.address.with.problem> shows states > > for not working ping as > > > > all icmp a.b.c.d:538 <- 172.16.x.y:538 0:0 > > all icmp e.f.g.h:40011 (172.16.x.y:538) -> a.b.c.d:40011 0:0 > > > > where a.b.c.d is address being used as ping target (outside of > > network), 172.16.x.y is address of device with trouble access to > > internet, and e.f.g.h is translated address for this device, > > allocated dynamically. > > > > After doing /etc/rc.d/pf restart if works again, so I think, again, > > issue is with some table being too small. Restart empties it and > > things begin to work. > > > > Does this sound familiar to anybody? I was trying to find some > > tuning guide for pf and large scale nat, but no success yet. I > > would be gratefull for any help. > > > > Regards, > > Milan > > > --=_mailhost.netlabit.sk-62017-1434817473-0001-2 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename=pfctl-vsinfo U3RhdHVzOiBFbmFibGVkIGZvciAxIGRheXMgMDg6NDA6MDMgICAgICAgICAgIERlYnVnOiBVcmdl bnQNCg0KSG9zdGlkOiAgIDB4ZGM5NmMwNWENCkNoZWNrc3VtOiAweGViMjA3YmU3YjAwN2MxMWRh M2Y5YmUzYzU0MWQ5MGQwDQoNClN0YXRlIFRhYmxlICAgICAgICAgICAgICAgICAgICAgICAgICBU b3RhbCAgICAgICAgICAgICBSYXRlDQogIGN1cnJlbnQgZW50cmllcyAgICAgICAgICAgICAgICAg ICAgNDk1MTggICAgICAgICAgICAgICANCiAgc2VhcmNoZXMgICAgICAgICAgICAgICAgICAgICAg NDIwMzExNjUyNCAgICAgICAgMzU3MzkuOS9zDQogIGluc2VydHMgICAgICAgICAgICAgICAgICAg ICAgICAgMzEwMDk1NTYgICAgICAgICAgMjYzLjcvcw0KICByZW1vdmFscyAgICAgICAgICAgICAg ICAgICAgICAgIDMxMDAwMzExICAgICAgICAgIDI2My42L3MNClNvdXJjZSBUcmFja2luZyBUYWJs ZQ0KICBjdXJyZW50IGVudHJpZXMgICAgICAgICAgICAgICAgICAgICAgNTQ4ICAgICAgICAgICAg ICAgDQogIHNlYXJjaGVzICAgICAgICAgICAgICAgICAgICAgICAgIDg2NzI2OTYgICAgICAgICAg IDczLjcvcw0KICBpbnNlcnRzICAgICAgICAgICAgICAgICAgICAgICAgICAgIDExMzQ4ICAgICAg ICAgICAgMC4xL3MNCiAgcmVtb3ZhbHMgICAgICAgICAgICAgICAgICAgICAgICAgICAxMDgwMCAg ICAgICAgICAgIDAuMS9zDQpDb3VudGVycw0KICBtYXRjaCAgICAgICAgICAgICAgICAgICAgICAg ICAgIDk4NzQ2MTg2ICAgICAgICAgIDgzOS43L3MNCiAgYmFkLW9mZnNldCAgICAgICAgICAgICAg ICAgICAgICAgICAgICAgMCAgICAgICAgICAgIDAuMC9zDQogIGZyYWdtZW50ICAgICAgICAgICAg ICAgICAgICAgICAgICAgICAyMDggICAgICAgICAgICAwLjAvcw0KICBzaG9ydCAgICAgICAgICAg ICAgICAgICAgICAgICAgICAgICAgMzQxICAgICAgICAgICAgMC4wL3MNCiAgbm9ybWFsaXplICAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgMCAgICAgICAgICAgIDAuMC9zDQogIG1lbW9yeSAg ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDAgICAgICAgICAgICAwLjAvcw0KICBiYWQt dGltZXN0YW1wICAgICAgICAgICAgICAgICAgICAgICAgICAwICAgICAgICAgICAgMC4wL3MNCiAg Y29uZ2VzdGlvbiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgMCAgICAgICAgICAgIDAuMC9z DQogIGlwLW9wdGlvbiAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDAgICAgICAgICAgICAw LjAvcw0KICBwcm90by1ja3N1bSAgICAgICAgICAgICAgICAgICAgICAgICAgICAwICAgICAgICAg ICAgMC4wL3MNCiAgc3RhdGUtbWlzbWF0Y2ggICAgICAgICAgICAgICAgICAgICA0NDMxMSAgICAg ICAgICAgIDAuNC9zDQogIHN0YXRlLWluc2VydCAgICAgICAgICAgICAgICAgICAgICAgICAgMTIg ICAgICAgICAgICAwLjAvcw0KICBzdGF0ZS1saW1pdCAgICAgICAgICAgICAgICAgICAgICAgICAg ICAwICAgICAgICAgICAgMC4wL3MNCiAgc3JjLWxpbWl0ICAgICAgICAgICAgICAgICAgICAgICAg ICAgICA2NiAgICAgICAgICAgIDAuMC9zDQogIHN5bnByb3h5ICAgICAgICAgICAgICAgICAgICAg ICAgICAgICAgIDAgICAgICAgICAgICAwLjAvcw0KTGltaXQgQ291bnRlcnMNCiAgbWF4IHN0YXRl cyBwZXIgcnVsZSAgICAgICAgICAgICAgICAgICAgMCAgICAgICAgICAgIDAuMC9zDQogIG1heC1z cmMtc3RhdGVzICAgICAgICAgICAgICAgICAgICAgICAgIDAgICAgICAgICAgICAwLjAvcw0KICBt YXgtc3JjLW5vZGVzICAgICAgICAgICAgICAgICAgICAgICAgICAwICAgICAgICAgICAgMC4wL3MN CiAgbWF4LXNyYy1jb25uICAgICAgICAgICAgICAgICAgICAgICAgIDM3NyAgICAgICAgICAgIDAu MC9zDQogIG1heC1zcmMtY29ubi1yYXRlICAgICAgICAgICAgICAgICAgICAyNzYgICAgICAgICAg ICAwLjAvcw0KICBvdmVybG9hZCB0YWJsZSBpbnNlcnRpb24gICAgICAgICAgICAgNjQ5ICAgICAg ICAgICAgMC4wL3MNCiAgb3ZlcmxvYWQgZmx1c2ggc3RhdGVzICAgICAgICAgICAgICAgIDY0OSAg ICAgICAgICAgIDAuMC9zDQo= --=_mailhost.netlabit.sk-62017-1434817473-0001-2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20150620182432.62797ec5>