From owner-freebsd-net@FreeBSD.ORG Wed Sep 14 20:50:16 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C7A7A106566B for ; Wed, 14 Sep 2011 20:50:16 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) by mx1.freebsd.org (Postfix) with ESMTP id A85618FC0A for ; Wed, 14 Sep 2011 20:50:16 +0000 (UTC) Received: from julian-mac.elischer.org (home-nat.elischer.org [67.100.89.137]) (authenticated bits=0) by vps1.elischer.org (8.14.4/8.14.4) with ESMTP id p8EKo8Lb032790 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 14 Sep 2011 13:50:15 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <4E7113A3.80605@freebsd.org> Date: Wed, 14 Sep 2011 13:50:43 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.22) Gecko/20110902 Thunderbird/3.1.14 MIME-Version: 1.0 To: Vladimir Budnev References: <4E7066CE.3070702@gmail.com> In-Reply-To: <4E7066CE.3070702@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Cc: freebsd-net@freebsd.org, freebsd-ipfw@freebsd.org Subject: Re: IPFW hidden/broken rule? (Free 7.2) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Sep 2011 20:50:16 -0000 On 9/14/11 1:33 AM, Vladimir Budnev wrote: > Hello list > > I am not sure which list this question must go to, so I am sending > to -net and -ipfw lists. > > We have faced some strange problem with ipfw behavior, which we > can't understand ourselves. An it really hurts:( > > We are running 7.2-RELEASE. > > I'll try to describe the problem as we observe it and our steps to > figure out what is happening. An questions will be at the end. > > In short there is a situation which looks like some old rule keep on > playing but we cant see this in ipfw outputs. I'd say 'hidden' rule > > So we have ipfw using tables for pipes, and we have e.g. following > ipfw output.The pipes configuration does mean nothing, it will be > clear at the end of mail, where key moment will be described. > > [Out1] > <...> > 04701 pipe tablearg ip from table(2) to any in via em0 > 04801 pipe tablearg ip from any to table(3) out via em0 > 04901 allow ip from table(4) to any in via em0 > 05001 allow ip from any to table(4) out via em0 > 05101 fwd tcp from table(5) to any dst-port > 80,443,8828 in via em0 > <...> > > > We'v noticed that no packets from specific ip(10.121.241.23) reache > 5101 rule with fwd to example.server. > > This ip is in table 5: > # ipfw table 5 list > 10.10.122.23/32 0 > 10.10.122.167/32 0 > > We’ve parsed all tables and no matches. > OK, we started placing debug rules to realize which rule accepts > packets. > > And we found out following.If we place the following debug rule(we > used the same fwd rule) such way(added 4602 rule): > [Out2] > <...> > 04602 113 75 fwd tcp from table(5) to any dst-port > 80,443,8828 in via em0 <-- OK her > 04701 107971113 85095893815 pipe tablearg ip from table(2) to any in > via em0 > 04801 102517924 83276945675 pipe tablearg ip from any to table(3) > out via em0 > 04901 4413338 991348968 allow ip from table(4) to any in via em0 > 05001 7146323 8293221022 allow ip from any to table(4) out via em0 > 05101 0 0 fwd tcp from table(5) to any dst-port > 80,443,8828 in via em0 > <...> > > Then packets match the rule.But if we delete 4602 and place debug > rule at 4702: > > [Out3] > <...> > 04701 108458823 85372134891 pipe tablearg ip from table(2) to any in > via em0 > 04702 0 0 fwd tcp from table(5) to any dst-port > 80,443,8828 in via em0 <-- NOT WORKING > 04801 - - pipe tablearg ip from any to table(3) out via em0 > 04901 - - allow ip from table(4) to any in via em0 > 05001 - - allow ip from any to table(4) out via em0 > 05101 0 0 fwd tcp from table(5) to any dst-port > 80,443,8828 in via em0 > <...> > > So placing rule after 4701 gives zero counters, that means that our > ip packets match 4701 rule :) > > But we DO NOT HAVE target ip in table 2. It resides only in table 5 > as shown before.But it must be noticed that such rule WAS in table 2 > before.But then was removed from table 2 and added to table 6. > > Records in table 2 looks like that: > # ipfw table 2 list > 10.10.122.20/32 9 > 10.10.122.21/32 4 > 10.10.122.25/32 9 > 10.10.122.28/32 6 > 10.10.122.30/32 6 > > We also thought that mb we added missconfigured rule e.g. > 10.10.122.0/24 rule in table 2 but no. > > > > Here goes the KEY moment! > OK crawling through the logs, different tests etc we decide to clear > table 2. > At first step, we’ve deleted all records manually, with: > ipfw table 2 delete > > and NO effect at the time table was clear.So there were no records > in table 2, but rule 4701 was still catching packets from the target > ip. > > Second, we did flush with > ipfw table 2 flush > And voila, everything went fine from that moment! Miracle flush? > > So the problem seems like the rule resides deep in the kernel heart, > but we can NOT see this rule with ipfw output. > > > So i think there are at least to questions: > > 1. Have anyone ever met such situation? Or may be something close to > this one with 'hidden' ipfw rules? I have never seen it.. > 2. Is there a way how to get more info about rules and firewall > matching decisions? Mb some sysctl tuning or something else? Cause > its the first time we met such situation and don’t even imagine how > to diagnose such 'hidden' rule:( turn on logging and add 'log' to each rule. (we probably should have an option to turn on logging by default but I don't know of such). > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >