Date: Wed, 14 Sep 2011 12:33:18 +0400 From: Vladimir Budnev <vladimir.budnev@gmail.com> To: freebsd-net@freebsd.org, freebsd-ipfw@freebsd.org Subject: IPFW hidden/broken rule? (Free 7.2) Message-ID: <4E7066CE.3070702@gmail.com>
next in thread | raw e-mail | index | archive | help
Hello list I am not sure which list this question must go to, so I am sending to -net and -ipfw lists. We have faced some strange problem with ipfw behavior, which we can't understand ourselves. An it really hurts:( We are running 7.2-RELEASE. I'll try to describe the problem as we observe it and our steps to figure out what is happening. An questions will be at the end. In short there is a situation which looks like some old rule keep on playing but we cant see this in ipfw outputs. I'd say 'hidden' rule So we have ipfw using tables for pipes, and we have e.g. following ipfw output.The pipes configuration does mean nothing, it will be clear at the end of mail, where key moment will be described. [Out1] <...> 04701 pipe tablearg ip from table(2) to any in via em0 04801 pipe tablearg ip from any to table(3) out via em0 04901 allow ip from table(4) to any in via em0 05001 allow ip from any to table(4) out via em0 05101 fwd <example.server> tcp from table(5) to any dst-port 80,443,8828 in via em0 <...> We'v noticed that no packets from specific ip(10.121.241.23) reache 5101 rule with fwd to example.server. This ip is in table 5: # ipfw table 5 list 10.10.122.23/32 0 10.10.122.167/32 0 We’ve parsed all tables and no matches. OK, we started placing debug rules to realize which rule accepts packets. And we found out following.If we place the following debug rule(we used the same fwd rule) such way(added 4602 rule): [Out2] <...> 04602 113 75 fwd <example.server> tcp from table(5) to any dst-port 80,443,8828 in via em0 <-- OK her 04701 107971113 85095893815 pipe tablearg ip from table(2) to any in via em0 04801 102517924 83276945675 pipe tablearg ip from any to table(3) out via em0 04901 4413338 991348968 allow ip from table(4) to any in via em0 05001 7146323 8293221022 allow ip from any to table(4) out via em0 05101 0 0 fwd <example.server> tcp from table(5) to any dst-port 80,443,8828 in via em0 <...> Then packets match the rule.But if we delete 4602 and place debug rule at 4702: [Out3] <...> 04701 108458823 85372134891 pipe tablearg ip from table(2) to any in via em0 04702 0 0 fwd <example.server> tcp from table(5) to any dst-port 80,443,8828 in via em0 <-- NOT WORKING 04801 - - pipe tablearg ip from any to table(3) out via em0 04901 - - allow ip from table(4) to any in via em0 05001 - - allow ip from any to table(4) out via em0 05101 0 0 fwd <example.server> tcp from table(5) to any dst-port 80,443,8828 in via em0 <...> So placing rule after 4701 gives zero counters, that means that our ip packets match 4701 rule :) But we DO NOT HAVE target ip in table 2. It resides only in table 5 as shown before.But it must be noticed that such rule WAS in table 2 before.But then was removed from table 2 and added to table 6. Records in table 2 looks like that: # ipfw table 2 list 10.10.122.20/32 9 10.10.122.21/32 4 10.10.122.25/32 9 10.10.122.28/32 6 10.10.122.30/32 6 We also thought that mb we added missconfigured rule e.g. 10.10.122.0/24 rule in table 2 but no. Here goes the KEY moment! OK crawling through the logs, different tests etc we decide to clear table 2. At first step, we’ve deleted all records manually, with: ipfw table 2 delete <addr> and NO effect at the time table was clear.So there were no records in table 2, but rule 4701 was still catching packets from the target ip. Second, we did flush with ipfw table 2 flush And voila, everything went fine from that moment! Miracle flush? So the problem seems like the rule resides deep in the kernel heart, but we can NOT see this rule with ipfw output. So i think there are at least to questions: 1. Have anyone ever met such situation? Or may be something close to this one with 'hidden' ipfw rules? 2. Is there a way how to get more info about rules and firewall matching decisions? Mb some sysctl tuning or something else? Cause its the first time we met such situation and don’t even imagine how to diagnose such 'hidden' rule:(
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4E7066CE.3070702>