Date: Wed, 01 Apr 2009 11:45:43 +0200 From: Sebastiaan van Erk <sebster@sebster.com> To: freebsd-pf@freebsd.org Subject: Re: state mismatch/connection issues Message-ID: <49D337C7.9020707@sebster.com> In-Reply-To: <49C9F27F.3010505@sebster.com> References: <49C9F27F.3010505@sebster.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is a cryptographically signed message in MIME format. --------------ms060201060408070404070806 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi, I upgrade to the latest FreeBSD-7.0 release using freebsd-update, with kernel 7.0-RELEASE-p11. I still get massive amounts of state mismatches and intermittent connection problems (connection refused, operation not permitted) with outging connections.... My firewall rules are unchanged (see below), the stats are now: Status: Enabled for 3 days 21:29:15 Debug: Urgent State Table Total Rate current entries 1994 searches 33567431 99.7/s inserts 4611322 13.7/s removals 4609328 13.7/s Counters match 6170429 18.3/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 1 0.0/s memory 1516667 4.5/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 247 0.0/s state-mismatch 1438892 4.3/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s Does anybody have *any* clue what's going on, and how I can go about fixing it? Thanks in advance, Sebastiaan Sebastiaan van Erk wrote: > Hi, > > I'm running FreeBSD-7.0 RELEASE with the following patch to the kernel > (I know it's integrated in the latest patchlevels which you get when you > do freebsd-update, but since I'm still getting state-mismatches WITH the > patch I'm holding off on the upgrade until I have more information as to > the nature of the problem): > > *** net/pf.c 2007/09/07 21:34:10 1.558 > --- net/pf.c 2007/09/18 19:45:59 1.559 > *************** pf_test_state_tcp(struct pf_state **state, int directi > *** 3730,3735 **** > --- 3730,3751 ---- > REASON_SET(reason, PFRES_SYNPROXY); > return (PF_SYNPROXY_DROP); > } > + } > + > + if (((th->th_flags & (TH_SYN|TH_ACK)) == TH_SYN) && > + dst->state >= TCPS_FIN_WAIT_2 && > + src->state >= TCPS_FIN_WAIT_2) { > + if (pf_status.debug >= PF_DEBUG_MISC) { > + printf("pf: state reuse "); > + pf_print_state(*state); > + pf_print_flags(th->th_flags); > + printf("\n"); > + } > + /* XXX make sure it's the same direction ?? */ > + (*state)->src.state = (*state)->dst.state = TCPS_CLOSED; > + pf_unlink_state(*state); > + *state = NULL; > + return (PF_DROP); > } > > if (src->wscale && dst->wscale && !(th->th_flags & TH_SYN)) { > > > The problem I'm having is that I get intermittent connection > refused/operation not permitted to another machine on the local network. > When I do pfctl -s info I see *huge* numbers of state mismatches: > > Status: Enabled for 94 days 01:27:40 Debug: Urgent > > State Table Total Rate > current entries 398 > searches 986228319 121.4/s > inserts 104049508 12.8/s > removals 104049110 12.8/s > Counters > match 107482262 13.2/s > bad-offset 0 0.0/s > fragment 0 0.0/s > short 0 0.0/s > normalize 42 0.0/s > memory 3125235 0.4/s > bad-timestamp 0 0.0/s > congestion 0 0.0/s > ip-option 0 0.0/s > proto-cksum 13919 0.0/s > state-mismatch 3039814 0.4/s > state-insert 0 0.0/s > state-limit 0 0.0/s > src-limit 0 0.0/s > synproxy 0 0.0/s > > This is causing serious problems at them moment. It seems that the state > problems occur in certain small time windows (my nagios starts reporting > that every service is connection refused/operation not permitted, which > is about 20 services). Then I get 20 recovery messages. > > The firewall rules are trivially simple, $ext_if has 2 ips and $int_if > has one: > > interfaces = "{" $ext_if "," $int_if "}" > > scrub in all > set skip on lo0 > antispoof for $interfaces inet > block out log quick on $ext_if from !$ext_ip1 to any > block in quick on $ext_if from any to 255.255.255.255 > block log all > > pass in quick inet proto icmp all icmp-type $icmp_types > > pass in quick on $int_if from $int_net to any > pass out quick on $int_if from any to $int_net > > pass out on $ext_if proto tcp all > pass out on $ext_if proto { udp, icmp } all > pass in on $ext_if proto tcp from any to $ext_ip1 port $tcp_services1 > pass in on $ext_if proto tcp from any to $ext_ip2 port $tcp_services2 > > Does anybody have any idea what's going on and where I can look? This is > a production server so it's seriously influencing the quality of the > hosted services. :-( > > > Regards, > Sebastiaan --------------ms060201060408070404070806 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIJUTCC AwMwggJsoAMCAQICEFN8DarMNuuKJDEtfs0UaqUwDQYJKoZIhvcNAQEFBQAwYjELMAkGA1UE BhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMT I1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMB4XDTA4MDYzMDEzNTE1N1oX DTA5MDYzMDEzNTE1N1owaDEQMA4GA1UEBBMHdmFuIEVyazETMBEGA1UEKhMKU2ViYXN0aWFh bjEbMBkGA1UEAxMSU2ViYXN0aWFhbiB2YW4gRXJrMSIwIAYJKoZIhvcNAQkBFhNzZWJzdGVy QHNlYnN0ZXIuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAsJDDAeYHVmH/ GVxi+bhFx27dmg++9BdhPJfk8k041sqEqq7oXnR2GT54quY3Ac7A1BuOM2JvoICraGmjud4y b3EanRnqGIK6iH+VAhhTlV/Owrb2Qm1e13DLxwLp1SocSQl4IrEbF9Y5H3ASdIrE0iFqkpju nPiiHeNhz3LaI5ipjiluKYoH+F6gPx8njHoaDxPePCkSLg4r0IA0afLM74LVZxCRBZEfyRZS J6VVUJefKlz91dWSzR/3xSw/rO4u9Ds/Zh7VBUKy3K+YFryHxRpUek0gSepE1b70Q39L9Sqd M/NZqMvFpwrqgW2Zh2Nh8nqRge90maR4ypBzz3GzLwIDAQABozAwLjAeBgNVHREEFzAVgRNz ZWJzdGVyQHNlYnN0ZXIuY29tMAwGA1UdEwEB/wQCMAAwDQYJKoZIhvcNAQEFBQADgYEAS1Sk NMgDVzb0ktO9tPPacV0KdKhTYOHcICVmuDEe2sFHOkjLAI1iAKp640pqJEVqvRnfRcCFJ9hK koPjjVZ+ui2rVmJWBG6FSloLRS/YYED4tUAw6DQhK61UOpjkpQxjCdm+5bHG/2ZgJAda1j0x uiN822+xFkcaW/5PQgxSRxcwggMDMIICbKADAgECAhBTfA2qzDbriiQxLX7NFGqlMA0GCSqG SIb3DQEBBQUAMGIxCzAJBgNVBAYTAlpBMSUwIwYDVQQKExxUaGF3dGUgQ29uc3VsdGluZyAo UHR5KSBMdGQuMSwwKgYDVQQDEyNUaGF3dGUgUGVyc29uYWwgRnJlZW1haWwgSXNzdWluZyBD QTAeFw0wODA2MzAxMzUxNTdaFw0wOTA2MzAxMzUxNTdaMGgxEDAOBgNVBAQTB3ZhbiBFcmsx EzARBgNVBCoTClNlYmFzdGlhYW4xGzAZBgNVBAMTElNlYmFzdGlhYW4gdmFuIEVyazEiMCAG CSqGSIb3DQEJARYTc2Vic3RlckBzZWJzdGVyLmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEP ADCCAQoCggEBALCQwwHmB1Zh/xlcYvm4Rcdu3ZoPvvQXYTyX5PJNONbKhKqu6F50dhk+eKrm NwHOwNQbjjNib6CAq2hpo7neMm9xGp0Z6hiCuoh/lQIYU5VfzsK29kJtXtdwy8cC6dUqHEkJ eCKxGxfWOR9wEnSKxNIhapKY7pz4oh3jYc9y2iOYqY4pbimKB/heoD8fJ4x6Gg8T3jwpEi4O K9CANGnyzO+C1WcQkQWRH8kWUielVVCXnypc/dXVks0f98UsP6zuLvQ7P2Ye1QVCstyvmBa8 h8UaVHpNIEnqRNW+9EN/S/UqnTPzWajLxacK6oFtmYdjYfJ6kYHvdJmkeMqQc89xsy8CAwEA AaMwMC4wHgYDVR0RBBcwFYETc2Vic3RlckBzZWJzdGVyLmNvbTAMBgNVHRMBAf8EAjAAMA0G CSqGSIb3DQEBBQUAA4GBAEtUpDTIA1c29JLTvbTz2nFdCnSoU2Dh3CAlZrgxHtrBRzpIywCN YgCqeuNKaiRFar0Z30XAhSfYSpKD441Wfrotq1ZiVgRuhUpaC0Uv2GBA+LVAMOg0ISutVDqY 5KUMYwnZvuWxxv9mYCQHWtY9MbojfNtvsRZHGlv+T0IMUkcXMIIDPzCCAqigAwIBAgIBDTAN BgkqhkiG9w0BAQUFADCB0TELMAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTES MBAGA1UEBxMJQ2FwZSBUb3duMRowGAYDVQQKExFUaGF3dGUgQ29uc3VsdGluZzEoMCYGA1UE CxMfQ2VydGlmaWNhdGlvbiBTZXJ2aWNlcyBEaXZpc2lvbjEkMCIGA1UEAxMbVGhhd3RlIFBl cnNvbmFsIEZyZWVtYWlsIENBMSswKQYJKoZIhvcNAQkBFhxwZXJzb25hbC1mcmVlbWFpbEB0 aGF3dGUuY29tMB4XDTAzMDcxNzAwMDAwMFoXDTEzMDcxNjIzNTk1OVowYjELMAkGA1UEBhMC WkExJTAjBgNVBAoTHFRoYXd0ZSBDb25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1Ro YXd0ZSBQZXJzb25hbCBGcmVlbWFpbCBJc3N1aW5nIENBMIGfMA0GCSqGSIb3DQEBAQUAA4GN ADCBiQKBgQDEpjxVc1X7TrnKmVoeaMB1BHCd3+n/ox7svc31W/Iadr1/DDph8r9RzgHU5VAK MNcCY1osiRVwjt3J8CuFWqo/cVbLrzwLB+fxH5E2JCoTzyvV84J3PQO+K/67GD4Hv0CAAmTX p6a7n2XRxSpUhQ9IBH+nttE8YQRAHmQZcmC3+wIDAQABo4GUMIGRMBIGA1UdEwEB/wQIMAYB Af8CAQAwQwYDVR0fBDwwOjA4oDagNIYyaHR0cDovL2NybC50aGF3dGUuY29tL1RoYXd0ZVBl cnNvbmFsRnJlZW1haWxDQS5jcmwwCwYDVR0PBAQDAgEGMCkGA1UdEQQiMCCkHjAcMRowGAYD VQQDExFQcml2YXRlTGFiZWwyLTEzODANBgkqhkiG9w0BAQUFAAOBgQBIjNFQg+oLLswNo2as Zw9/r6y+whehQ5aUnX9MIbj4Nh+qLZ82L8D0HFAgk3A8/a3hYWLD2ToZfoSxmRsAxRoLgnSe JVCUYsfbJ3FXJY3dqZw5jowgT2Vfldr394fWxghOrvbqNOUQGls1TXfjViF4gtwhGTXeJLHT HUb/XV9lTzGCA3EwggNtAgEBMHYwYjELMAkGA1UEBhMCWkExJTAjBgNVBAoTHFRoYXd0ZSBD b25zdWx0aW5nIChQdHkpIEx0ZC4xLDAqBgNVBAMTI1RoYXd0ZSBQZXJzb25hbCBGcmVlbWFp bCBJc3N1aW5nIENBAhBTfA2qzDbriiQxLX7NFGqlMAkGBSsOAwIaBQCgggHQMBgGCSqGSIb3 DQEJAzELBgkqhkiG9w0BBwEwHAYJKoZIhvcNAQkFMQ8XDTA5MDQwMTA5NDU0M1owIwYJKoZI hvcNAQkEMRYEFCtrXiyj/rgQ+sdChaa6UguqMBLXMF8GCSqGSIb3DQEJDzFSMFAwCwYJYIZI AWUDBAECMAoGCCqGSIb3DQMHMA4GCCqGSIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUr DgMCBzANBggqhkiG9w0DAgIBKDCBhQYJKwYBBAGCNxAEMXgwdjBiMQswCQYDVQQGEwJaQTEl MCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3Rl IFBlcnNvbmFsIEZyZWVtYWlsIElzc3VpbmcgQ0ECEFN8DarMNuuKJDEtfs0UaqUwgYcGCyqG SIb3DQEJEAILMXigdjBiMQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRp bmcgKFB0eSkgTHRkLjEsMCoGA1UEAxMjVGhhd3RlIFBlcnNvbmFsIEZyZWVtYWlsIElzc3Vp bmcgQ0ECEFN8DarMNuuKJDEtfs0UaqUwDQYJKoZIhvcNAQEBBQAEggEAAoFT+l6gtgpYB70X 6bXph2htHLrDO+u1ywmZtqv0DdCbRQRug5gr605HU/59kCwW7uFq0oPInh/hkCjY97bv4bON IakAybxs9f9d/YtOdDCzdtalP/sqBuRJ6XIfhWN727DsBW5FP/t5ocuJErWe25FredcLSG03 jpzGMTKDzdAR3Wo9MWjxl6w4TJjiTw2CBitkojcrbCe8rZrHxKFr48ll1bEIq/h15ES1quLc lboJtyA9AyQLros03z5CrFmoSlvg0zRTITQj42OCXGK5KLViHCDTz74af3mYypz//HBIT6oD EKYeKngSN1ia0v71jQxWhlYD+ngf1Tddipxe0gAAAAAAAA== --------------ms060201060408070404070806--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?49D337C7.9020707>