From owner-freebsd-ipfw@FreeBSD.ORG Sun Nov 23 18:02:59 2008 Return-Path: Delivered-To: freebsd-ipfw@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F24381065677; Sun, 23 Nov 2008 18:02:59 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id CB8B28FC0A; Sun, 23 Nov 2008 18:02:59 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id mANI2xk9011355; Sun, 23 Nov 2008 18:02:59 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id mANI2xsD011351; Sun, 23 Nov 2008 18:02:59 GMT (envelope-from linimon) Date: Sun, 23 Nov 2008 18:02:59 GMT Message-Id: <200811231802.mANI2xsD011351@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-ipfw@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/129093: [ipfw] ipfw nat must not drop packets X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Nov 2008 18:03:00 -0000 Old Synopsis: ipfw nat must not drop packets New Synopsis: [ipfw] ipfw nat must not drop packets Responsible-Changed-From-To: freebsd-bugs->freebsd-ipfw Responsible-Changed-By: linimon Responsible-Changed-When: Sun Nov 23 18:02:44 UTC 2008 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=129093 From owner-freebsd-ipfw@FreeBSD.ORG Sun Nov 23 23:42:24 2008 Return-Path: Delivered-To: freebsd-ipfw@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5D3541065673; Sun, 23 Nov 2008 23:42:24 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 332858FC13; Sun, 23 Nov 2008 23:42:24 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id mANNgO72069404; Sun, 23 Nov 2008 23:42:24 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id mANNgOnr069400; Sun, 23 Nov 2008 23:42:24 GMT (envelope-from linimon) Date: Sun, 23 Nov 2008 23:42:24 GMT Message-Id: <200811232342.mANNgOnr069400@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-ipfw@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/129103: [ipfw] IPFW check state does not work =( X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Nov 2008 23:42:24 -0000 Old Synopsis: IPFW check state does not work =( New Synopsis: [ipfw] IPFW check state does not work =( Responsible-Changed-From-To: freebsd-bugs->freebsd-ipfw Responsible-Changed-By: linimon Responsible-Changed-When: Sun Nov 23 23:42:11 UTC 2008 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=129103 From owner-freebsd-ipfw@FreeBSD.ORG Mon Nov 24 11:07:15 2008 Return-Path: Delivered-To: freebsd-ipfw@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7ED251065672 for ; Mon, 24 Nov 2008 11:07:15 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6E6D38FC21 for ; Mon, 24 Nov 2008 11:07:15 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id mAOB7FCJ019936 for ; Mon, 24 Nov 2008 11:07:15 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id mAOB7EeC019931 for freebsd-ipfw@FreeBSD.org; Mon, 24 Nov 2008 11:07:14 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 24 Nov 2008 11:07:14 GMT Message-Id: <200811241107.mAOB7EeC019931@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-ipfw@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-ipfw@FreeBSD.org X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Nov 2008 11:07:15 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/129103 ipfw [ipfw] IPFW check state does not work =( o kern/129093 ipfw [ipfw] ipfw nat must not drop packets o kern/129036 ipfw [ipfw] 'ipfw fwd' does not change outgoing interface n o kern/128260 ipfw [ipfw] [patch] ipfw_divert damages IPv6 packets o kern/127230 ipfw [ipfw] [patch] Feature request to add UID and/or GID l o kern/127209 ipfw [ipfw] IPFW table become corrupted after many changes o bin/125370 ipfw [ipfw] [patch] increase a line buffer limit o conf/123119 ipfw [patch] rc script for ipfw does not handle IPv6 o kern/122963 ipfw [ipfw] tcpdump does not show packets redirected by 'ip s kern/121807 ipfw [request] TCP and UDP port_table in ipfw o kern/121382 ipfw [dummynet]: 6.3-RELEASE-p1 page fault in dummynet (cor o kern/121122 ipfw [ipfw] [patch] add support to ToS IP PRECEDENCE fields o kern/118993 ipfw [ipfw] page fault - probably it's a locking problem o kern/117234 ipfw [ipfw] [patch] ipfw send_pkt() and ipfw_tick() don't s o kern/116009 ipfw [ipfw] [patch] Ignore errors when loading ruleset from p kern/115755 ipfw [ipfw][patch] unify message and add a rule number wher o bin/115172 ipfw [patch] ipfw(8) list show some rules with a wrong form o docs/113803 ipfw [patch] ipfw(8) - don't get bitten by the fwd rule p kern/113388 ipfw [ipfw][patch] Addition actions with rules within speci o kern/112708 ipfw [ipfw] ipfw is seems to be broken to limit number of c o kern/112561 ipfw [ipfw] ipfw fwd does not work with some TCP packets o kern/107305 ipfw [ipfw] ipfw fwd doesn't seem to work o kern/105330 ipfw [ipfw] [patch] ipfw (dummynet) does not allow to set q o bin/104921 ipfw [patch] ipfw(8) sometimes treats ipv6 input as ipv4 (a o kern/104682 ipfw [ipfw] [patch] Some minor language consistency fixes a o kern/103454 ipfw [ipfw] [patch] [request] add a facility to modify DF b o kern/103328 ipfw [ipfw] [request] sugestions about ipfw table o kern/102471 ipfw [ipfw] [patch] add tos and dscp support o kern/98831 ipfw [ipfw] ipfw has UDP hickups o kern/97951 ipfw [ipfw] [patch] ipfw does not tie interface details to o kern/97504 ipfw [ipfw] IPFW Rules bug o kern/95084 ipfw [ipfw] [regression] [patch] IPFW2 ignores "recv/xmit/v o kern/93300 ipfw [ipfw] ipfw pipe lost packets o kern/91847 ipfw [ipfw] ipfw with vlanX as the device o kern/88659 ipfw [modules] ipfw and ip6fw do not work properly as modul o kern/87032 ipfw [ipfw] [patch] ipfw ioctl interface implementation o kern/86957 ipfw [ipfw] [patch] ipfw mac logging o kern/82724 ipfw [ipfw] [patch] [request] Add setnexthop and defaultrou s kern/80642 ipfw [ipfw] [patch] ipfw small patch - new RULE OPTION o bin/78785 ipfw [patch] ipfw(8) verbosity locks machine if /etc/rc.fir o kern/74104 ipfw [ipfw] ipfw2/1 conflict not detected or reported, manp o kern/73910 ipfw [ipfw] serious bug on forwarding of packets after NAT o kern/72987 ipfw [ipfw] ipfw/dummynet pipe/queue 'queue [BYTES]KBytes ( o kern/71366 ipfw [ipfw] "ipfw fwd" sometimes rewrites destination mac a o kern/69963 ipfw [ipfw] install_state warning about already existing en o kern/60719 ipfw [ipfw] Headerless fragments generate cryptic error mes o kern/55984 ipfw [ipfw] [patch] time based firewalling support for ipfw o kern/51274 ipfw [ipfw] [patch] ipfw2 create dynamic rules with parent o kern/48172 ipfw [ipfw] [patch] ipfw does not log size and flags o kern/46159 ipfw [ipfw] [patch] [request] ipfw dynamic rules lifetime f a kern/26534 ipfw [ipfw] Add an option to ipfw to log gid/uid of who cau 51 problems total. From owner-freebsd-ipfw@FreeBSD.ORG Mon Nov 24 13:21:26 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9BD701065670 for ; Mon, 24 Nov 2008 13:21:26 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [220.233.188.227]) by mx1.freebsd.org (Postfix) with ESMTP id AF74A8FC1E for ; Mon, 24 Nov 2008 13:21:25 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id mAOCs65U036898; Mon, 24 Nov 2008 23:54:07 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Mon, 24 Nov 2008 23:54:05 +1100 (EST) From: Ian Smith To: Eugen Konkov In-Reply-To: <200811232342.mANNgOnr069400@freefall.freebsd.org> Message-ID: <20081124203046.I43853@sola.nimnet.asn.au> References: <200811232342.mANNgOnr069400@freefall.freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-ipfw@freebsd.org Subject: Re: kern/129103: [ipfw] IPFW check state does not work =( X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Nov 2008 13:21:26 -0000 On Sun, 23 Nov 2008, linimon@freebsd.org wrote: > Old Synopsis: IPFW check state does not work =( > New Synopsis: [ipfw] IPFW check state does not work =( > > Responsible-Changed-From-To: freebsd-bugs->freebsd-ipfw > Responsible-Changed-By: linimon > Responsible-Changed-When: Sun Nov 23 23:42:11 UTC 2008 > Responsible-Changed-Why: > Over to maintainer(s). > > http://www.freebsd.org/cgi/query-pr.cgi?pr=129103 Eugen, I'll have a go; worst that can happen is I get it all wrong :) Last things first: a check-state rule NEVER counts any packets; the packet/byte counts are attributed to the triggering keep-state rule, which I think explains the counts you show. The prob 0.5 split makes it a bit harder to see what's happening here, but I'll quote the relevant: > 00001 0 0 check-state As expected. It would be helpful to put another like rule 2 *before* this check-state, showing the total number of icmp packets via ng0, or perhaps even two, one for 'in recv ngo' and one for 'out xmit ng0'. > 00002 6 360 count log icmp from any to any via ng0 This only counts icmp that has not yet been attributed to one of the check-state rules below, ie original, trigger packets for a flow; the response packets in a flow are counted by the keep-state rule (and are shown below in your expired dynamic rule counts) > 00003 5 300 prob 0.500000 skipto 6 log icmp from any to any via ng0 So 5 packets did the skipto 6. Unclear how many failed the prob test, but perhaps 5 or 6 fell through; we don't have the total count .. > 00004 8 480 skipto 5 log icmp from any to any via ng0 keep-state .. however, this 8 count includes some responses to the already established state - that you apparently expected to see in the check-state count. > 00005 3 180 skipto 10 log icmp from any to any via ng0 I'm not sure why this shows only 3 packets though, perhaps because some of the keep-state response packets by now are going out via ng1? > 00006 3 180 skipto 7 log icmp from any to any via ng0 keep-state > 00007 3 180 count log icmp from any to any via ng0 > 00010 6 360 count log icmp from any to any via ng0 That seems to tally, though dynamic rule 6 shows 2 (plus 1 trigger pkt?) > 00099 47 2924 nat 1 ip from any to any via ng0 [..] > ## Dynamic rules (2): > 00004 7 420 (0s) STATE icmp 192.168.9.4 0 <-> 213.180.204.8 0 > 00006 2 120 (0s) STATE icmp 213.180.204.8 0 <-> 91.124.239.145 0 This seems to indicate a total of 9 packets passed due to kept state, so perhaps there were 11 icmp packets in total, 6 out and 5 responses (^C)? > Why 5 packets for rule 3 and 8 packets for rule 4? I'm not sure, but there may be another issue here. I'll tail-quote the first of your security logs below, which seem, on a quick check, to add up to the packet counts for each rule, however it's worth noting the packets logged as being via ng1 (presumably your inside, NAT interface?) Use of 'via' can be a bit misleading. On inbound packets (ie from ip_input) only the receive interface can be tested before routing has occurred, so 'via ng0' shows just those packets. However on outbound packets, 'via iface' applies to packets either received on *or* being transmitted on that interface, which I think is what's happening with those that are logged showing ng1 .. remembering these are shown before NAT translation. Happy to accept any correction on any of this .. Regarding the other (kern/129093: [ipfw] ipfw nat must not drop packets) I can't say anything about what I haven't used, but what's your value of net.inet.ip.fw.one_pass ? cheers, Ian ===== quote from PR, not indented: cat security Nov 23 23:18:39 home kernel: ipfw: 2 Count ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:39 home kernel: ipfw: 4 SkipTo 5 ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:39 home kernel: ipfw: 5 SkipTo 10 ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:39 home kernel: ipfw: 10 Count ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:39 home kernel: ipfw: 2 Count ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:39 home kernel: ipfw: 3 SkipTo 6 ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:39 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:39 home kernel: ipfw: 7 Count ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:39 home kernel: ipfw: 10 Count ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:39 home kernel: ipfw: 4 SkipTo 5 ICMP:0.0 213.180.204.8 192.168.9.4 out via ng1 Nov 23 23:18:40 home kernel: ipfw: 4 SkipTo 5 ICMP:8.0 192.168.9.4 213.180.204.8 in via ng1 Nov 23 23:18:40 home kernel: ipfw: 2 Count ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:40 home kernel: ipfw: 3 SkipTo 6 ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:40 home kernel: ipfw: 4 SkipTo 5 ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:40 home kernel: ipfw: 5 SkipTo 10 ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:40 home kernel: ipfw: 10 Count ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:40 home kernel: ipfw: 2 Count ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:40 home kernel: ipfw: 3 SkipTo 6 ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:40 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:40 home kernel: ipfw: 7 Count ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:40 home kernel: ipfw: 10 Count ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:40 home kernel: ipfw: 4 SkipTo 5 ICMP:0.0 213.180.204.8 192.168.9.4 out via ng1 Nov 23 23:18:41 home kernel: ipfw: 4 SkipTo 5 ICMP:8.0 192.168.9.4 213.180.204.8 in via ng1 Nov 23 23:18:41 home kernel: ipfw: 2 Count ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:41 home kernel: ipfw: 3 SkipTo 6 ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:41 home kernel: ipfw: 4 SkipTo 5 ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:41 home kernel: ipfw: 5 SkipTo 10 ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:41 home kernel: ipfw: 10 Count ICMP:8.0 192.168.9.4 213.180.204.8 out via ng0 Nov 23 23:18:42 home kernel: ipfw: 2 Count ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:42 home kernel: ipfw: 3 SkipTo 6 ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:42 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:42 home kernel: ipfw: 7 Count ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:42 home kernel: ipfw: 10 Count ICMP:0.0 213.180.204.8 91.124.239.145 in via ng0 Nov 23 23:18:42 home kernel: ipfw: 4 SkipTo 5 ICMP:0.0 213.180.204.8 192.168.9.4 out via ng1 Why in log do I see trafic for ng1 interface while rule 1 does not invoked? From owner-freebsd-ipfw@FreeBSD.ORG Mon Nov 24 21:11:37 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D9F021065672 for ; Mon, 24 Nov 2008 21:11:37 +0000 (UTC) (envelope-from kes-kes@yandex.ru) Received: from forwards4.yandex.ru (forwards4.yandex.ru [77.88.32.20]) by mx1.freebsd.org (Postfix) with ESMTP id 913D88FC08 for ; Mon, 24 Nov 2008 21:11:37 +0000 (UTC) (envelope-from kes-kes@yandex.ru) Received: from smtp12.yandex.ru (smtp12.yandex.ru [77.88.32.82]) by forwards4.yandex.ru (Yandex) with ESMTP id B4AE84C50FC; Mon, 24 Nov 2008 23:56:10 +0300 (MSK) Received: from 102-80-113-92.pool.ukrtel.net ([92.113.80.102]:14862 "EHLO KES" smtp-auth: "kes-kes" TLS-CIPHER: TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S5325777AbYKXU4E (ORCPT + 1 other); Mon, 24 Nov 2008 23:56:04 +0300 X-Yandex-Spam: 1 X-Yandex-Front: smtp12 X-Yandex-TimeMark: 1227560164 X-BornDate: 1149541200 X-Yandex-Karma: 0 X-Yandex-KarmaStatus: 0 X-MsgDayCount: 4 X-Comment: RFC 2476 MSA function at smtp12.yandex.ru logged sender identity as: kes-kes Date: Mon, 24 Nov 2008 22:56:02 +0200 From: KES X-Mailer: The Bat! (v4.0.24) Professional Organization: SaftTen X-Priority: 3 (Normal) Message-ID: <1517824.20081124225602@yandex.ru> To: Ian Smith In-Reply-To: <20081124203046.I43853@sola.nimnet.asn.au> References: <200811232342.mANNgOnr069400@freefall.freebsd.org> <20081124203046.I43853@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: 8bit Cc: freebsd-ipfw@freebsd.org Subject: Re[2]: kern/129103: [ipfw] IPFW check state does not work =( X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: KES List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Nov 2008 21:11:37 -0000 sorry, I miss some explanation Before beginngin tests I ipfw zero : > /var/log/security then for user on ng1 I do: ping -n 3 I.N.E.T > 00002 6 360 count log icmp from any to any via ng0 here I count all packets going through ng0 3 in + 3 out, all is ok here > 00003 5 300 prob 0.500000 skipto 6 log icmp from any to any via ng0 I want to split traffic. Now here I just study how it is done. Actually I want to fwd packets through differeng ISP but send packet to same ISP if connection is established. So traffic will flow over 4,5 or 6,7, 00004 8 480 skipto 5 log icmp from any to any via ng0 keep-state 00005 3 180 skipto 10 log icmp from any to any via ng0 00006 3 180 skipto 7 log icmp from any to any via ng0 keep-state 00007 3 180 count log icmp from any to any via ng0 expected results for rule 4 is 3 packets. Why it is 8 I do not know > 00010 6 360 count log icmp from any to any via ng0 here I count all packets going through ng0 again. As you see it is 6. All is ok > 00099 47 2924 nat 1 ip from any to any via ng0 just natting, nat all traffic, so counter is so big > 00004 7 420 (0s) STATE icmp 192.168.9.4 0 <-> 213.180.204.8 0 > 00006 2 120 (0s) STATE icmp 213.180.204.8 0 <-> 91.124.239.145 0 This is very strange. Here I expect 3 for first and second rule but why here 7 and 2 packets?? that is mistery (( From owner-freebsd-ipfw@FreeBSD.ORG Tue Nov 25 14:22:09 2008 Return-Path: Delivered-To: ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 69ABE1065692 for ; Tue, 25 Nov 2008 14:22:08 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (www.svzserv.kemerovo.su [213.184.65.80]) by mx1.freebsd.org (Postfix) with ESMTP id 48B828FC08 for ; Tue, 25 Nov 2008 14:22:05 +0000 (UTC) (envelope-from eugen@kuzbass.ru) Received: from www.svzserv.kemerovo.su (eugen@localhost [127.0.0.1]) by www.svzserv.kemerovo.su (8.13.8/8.13.8) with ESMTP id mAPDgd2w018301; Tue, 25 Nov 2008 20:42:39 +0700 (KRAT) (envelope-from eugen@www.svzserv.kemerovo.su) Received: (from eugen@localhost) by www.svzserv.kemerovo.su (8.13.8/8.13.8/Submit) id mAPDgdY1018299; Tue, 25 Nov 2008 20:42:39 +0700 (KRAT) (envelope-from eugen) Date: Tue, 25 Nov 2008 20:42:39 +0700 From: Eugene Grosbein To: net@freebsd.org Message-ID: <20081125134239.GA17138@svzserv.kemerovo.su> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i Cc: ipfw@freebsd.org Subject: m_tag_find() overhead X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Nov 2008 14:22:20 -0000 Hi! I wonder if one more call to m_tag_find() for every outgoing packet would be expensive. For many years conventional way to implement PBR was: use 'ipfw fwd'. Currently 'ipfw add fwd ... in' marks a packet with PACKET_TAG_IPFORWARD. Such forwarding changes packet's outgoing interface generally but none in the kernel reflect this in packet's metadata and packets are passed by ip_output() to PFIL hooks with wrong outgoing interface name ( details and How-To-Repeat may be found here: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/129036 ) I've patched src/sys/netinet/ip_output.c to fix this problem by extra check for PACKET_TAG_IPFORWARD and update of route if necessary. This adds a call to m_tag_find() for every outgoing packet if packet filtering is enabled and kernel has options IPFIREWALL_FORWARD. Please review. It works for my test lab but I'm not sure if that's correct: should a call to rtalloc_ign() be accomplished with some resource freeing call? And what's the meaning of RTF_CLONING second argument of rtalloc_ign? --- ip_output.c.orig 2008-11-25 14:21:26.000000000 +0700 +++ ip_output.c 2008-11-25 18:29:32.000000000 +0700 @@ -195,6 +195,20 @@ ro->ro_rt = (struct rtentry *)0; } #ifdef IPFIREWALL_FORWARD + /* + * Check if packet has changed next-hop in ip_input() + * If so, update route so pfil hooks get it right + */ + if ((inet_pfil_hook.ph_busy_count != -1) && + (fwd_tag = m_tag_find(m, PACKET_TAG_IPFORWARD, NULL))) { + bzero(&iproute, sizeof(iproute)); + ro = &iproute; + bcopy((fwd_tag+1), &ro->ro_dst, sizeof(struct sockaddr_in)); + m_tag_delete(m, fwd_tag); + rtalloc_ign(ro, RTF_CLONING); + dst = (struct sockaddr_in *)&ro->ro_dst; + } + if (ro->ro_rt == NULL && fwd_tag == NULL) { #else if (ro->ro_rt == NULL) { Eugene Grosbein From owner-freebsd-ipfw@FreeBSD.ORG Wed Nov 26 09:14:24 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 520611065676 for ; Wed, 26 Nov 2008 09:14:24 +0000 (UTC) (envelope-from bogdan_inedit@yahoo.com) Received: from web50306.mail.re2.yahoo.com (web50306.mail.re2.yahoo.com [206.190.38.60]) by mx1.freebsd.org (Postfix) with SMTP id F0E098FC13 for ; Wed, 26 Nov 2008 09:14:23 +0000 (UTC) (envelope-from bogdan_inedit@yahoo.com) Received: (qmail 39658 invoked by uid 60001); 26 Nov 2008 08:47:41 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Date:From:Reply-To:Subject:To:MIME-Version:Content-Type:Message-ID; b=wgDQ0YV2F/9HqUqsKffGXpF31s8+eqcGVBKrEm2Lbw0MEDuy3zJvveb6J6ABu6Lh9Uj3sg1OOz88rY6sq2k5qPZ/Wgh3RATi9Q8AwFGZNbbJasidJEiG81NMtt/yWS9XzL8sqyfLuyjHbYAQqCgdMvUrnuOCoKqScZpxoiSORoc=; X-YMail-OSG: YMKvrNwVM1mFBE8UKGMjvNfA4Sl._ELBKU59aadQuddlNW7xzFJsALB0JD8cFITCAKZBTljXS7jjo7YxP6bjFQhR4HS05xQG7SqiTTE0Hd7.B_LUZyWgbagGB4rLNYK4c8aWVN3b7BeIYPdwvwdSM6zdYcMTqlrgFJp0Ta3Y Received: from [91.194.234.133] by web50306.mail.re2.yahoo.com via HTTP; Wed, 26 Nov 2008 00:47:41 PST X-Mailer: YahooMailWebService/0.7.260.1 Date: Wed, 26 Nov 2008 00:47:41 -0800 (PST) From: bogdan oprea To: freebsd-ipfw@freebsd.org MIME-Version: 1.0 Message-ID: <623098.39499.qm@web50306.mail.re2.yahoo.com> Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: dummynet + ack http X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: bogdan_inedit@yahoo.com List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Nov 2008 09:14:24 -0000 i was wondering if the following scenario is possible: i am running a small isp with upto 20Mbps download speed and 10Mbps upload speed i want to prioritize http traffic (and dns, ping, possibly voip and some games) over other traffic (p2p etc) i know that this is done with queues but i don't know the right weights for the queues 3 queues i think should be enough: queue1-ack http, queue2-ack-other, queue3-rest of the traffic so if anyone knows what are the weights of the queues or a ipfw.rules example please help me From owner-freebsd-ipfw@FreeBSD.ORG Fri Nov 28 06:58:06 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DC199106567A for ; Fri, 28 Nov 2008 06:58:05 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [220.233.188.227]) by mx1.freebsd.org (Postfix) with ESMTP id 0E0788FC14 for ; Fri, 28 Nov 2008 06:58:04 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id mAS6w2SC013631; Fri, 28 Nov 2008 17:58:03 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Fri, 28 Nov 2008 17:58:02 +1100 (EST) From: Ian Smith To: KES In-Reply-To: <1638065040.20081125011317@yandex.ru> Message-ID: <20081128161236.P92549@sola.nimnet.asn.au> References: <200811232342.mANNgOnr069400@freefall.freebsd.org> <20081124203046.I43853@sola.nimnet.asn.au> <1638065040.20081125011317@yandex.ru> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-ipfw@freebsd.org Subject: Re[2]: kern/129103: [ipfw] IPFW check state does not work =( X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Nov 2008 06:58:06 -0000 Kes, your message didn't make it to the list. Probably because of the 650KB attached diagram :) Big stuff is better put up as an URL to a web site. On Tue, 25 Nov 2008, KES wrote: > #ipfw -ed show > 00001 0 0 check-state log > 00002 2 120 count log icmp from any to any via ng0 > 00003 2 120 prob 0.500000 skipto 6 log icmp from any to any via ng0 > 00004 0 0 skipto 5 log icmp from any to any via ng0 keep-state > 00005 0 0 skipto 10 log icmp from any to any via ng0 > 00006 11 660 skipto 7 log icmp from any to any via ng0 keep-state > 00007 6 360 count log icmp from any to any via ng0 > 00010 6 360 count log icmp from any to any via ng0 > 00049 726 132682 nat 1 ip from any to any via ng0 > 00050 9088 2196349 allow ip from any to any > 00050 0 0 nat 1 ip from any to any via ng0 > 00100 0 0 allow ip from any to any via lo0 > 00200 0 0 deny ip from any to 127.0.0.0/8 > 00300 0 0 deny ip from 127.0.0.0/8 to any > 65535 0 0 deny ip from any to any > ## Dynamic rules (2): > 00006 7 420 (1s) STATE icmp 192.168.9.4 0 <-> 213.180.193.123 0 > 00006 2 120 (1s) STATE icmp 213.180.193.123 0 <-> 92.113.76.40 0 Ok, this one is easier because no packets took the 4/5 prob split. Note 2 separate state entries because of the different address pairs. > Nov 24 23:54:25 home kernel: ipfw: 2 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0 > Nov 24 23:54:25 home kernel: ipfw: 3 SkipTo 6 ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0 > Nov 24 23:54:25 home kernel: ipfw: 6 SkipTo 7 ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0 > Nov 24 23:54:25 home kernel: ipfw: 7 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0 > Nov 24 23:54:25 home kernel: ipfw: 10 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0 1st packet. At 6 it creates state 192.168.9.4 <--> 213.180.193.123 > Nov 24 23:54:25 home kernel: ipfw: 2 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0 > Nov 24 23:54:25 home kernel: ipfw: 3 SkipTo 6 ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0 > Nov 24 23:54:25 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0 > Nov 24 23:54:25 home kernel: ipfw: 7 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0 > Nov 24 23:54:25 home kernel: ipfw: 10 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0 1st response packet .. doesn't match the above state as it's a different address, so it creates state between 213.180.193.123 <--> 92.113.76.40 .. because you've put your nat rule in the wrong place, it should go before any of this, then all state would be between 213... and 92... Note both of the above count at rule 2, because neither match existing state. These two are also the last packets that do match rule 2. > Nov 24 23:54:25 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 192.168.9.4 out via ng1 So now, after NAT, this matches the first state above. It still matches rules 'via ng0' because that was its source interface on keep-state. It doesn't show in rule 2 because it jumps to rule 6 at the check-state. > Nov 24 23:54:26 home kernel: ipfw: 6 SkipTo 7 ICMP:8.0 192.168.9.4 213.180.193.123 in via ng1 Second ping. Despite coming in via ng1, it also matches the first state flow above, but doesn't match 7 or 10 (wrong interface) > Nov 24 23:54:26 home kernel: ipfw: 6 SkipTo 7 ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0 Another packet out, matches established state for first rule 6 .. > Nov 24 23:54:26 home kernel: ipfw: 7 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0 > Nov 24 23:54:26 home kernel: ipfw: 10 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0 .. and also matches 7 and 10, being via ng0. > Nov 24 23:54:26 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0 > Nov 24 23:54:26 home kernel: ipfw: 7 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0 > Nov 24 23:54:26 home kernel: ipfw: 10 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0 Return packet matches second state entry on rule 6, from check-state. > Nov 24 23:54:26 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 192.168.9.4 out via ng1 Same packet matching first state entry on rule 6, despite interface ng1, after NAT. > Nov 24 23:54:27 home kernel: ipfw: 6 SkipTo 7 ICMP:8.0 192.168.9.4 213.180.193.123 in via ng1 Third ping packet from ng1, also matches the first of rule 6 states, but not counted by 7 or 10 because of via different interface. > Nov 24 23:54:27 home kernel: ipfw: 6 SkipTo 7 ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0 Same packet going out, matches second of the rule 6 states .. > Nov 24 23:54:27 home kernel: ipfw: 7 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0 > Nov 24 23:54:27 home kernel: ipfw: 10 Count ICMP:8.0 192.168.9.4 213.180.193.123 out via ng0 .. which also gets counted by 7 and 10, being out via ng0. > Nov 24 23:54:27 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0 Return packet, matches second rule 6 state .. > Nov 24 23:54:27 home kernel: ipfw: 7 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0 > Nov 24 23:54:27 home kernel: ipfw: 10 Count ICMP:0.0 213.180.193.123 92.113.76.40 in via ng0 .. and 7 and 10. > Nov 24 23:54:27 home kernel: ipfw: 6 SkipTo 7 ICMP:0.0 213.180.193.123 192.168.9.4 out via ng1 Third reply being delivered, after NAT, to ng1 because it matches state. That's it. If you'd had separate rules to count icmp via ng1 you'd have seen them all (or if you'd specifed ng* rather than ng0) ======= > Seems all is ok, except > 00001 0 0 check-state log > Once dynamic rule is matched, check-state counter must be increased. No use saying 'must', when that's simply not how it works; check-state never shows any counts. You've filed a PR on the basis of how you wish it might work, and I believe I've shown above that it's working as advertised. I suggest you should now close this PR. ======= > Also it seems when keep-state create new dynamic rule, it must > increment it. Because of when rule body is matched, keep-state create > dynamic rule and then packet flow through this dynamic rule, counter > for dynamic rule is incremented, packet for parent rule is > incremented. and you do not loose counters. Again, that's just not how it works. Dynamic rule counts show traffic due to established state; they don't include the original packet that created the state. Sometimes you'll see dynamic rule counts of 0, where a packet went out (say) and setup state, but no response came back. In short, there's no use trying to use statistics from keep-state rules for accounting. Use separate stateless rules, BEFORE any check-state. > But now NOTICE: 11 packets for rule 6 and 9 (7+2) packets for dinamic rule > with rule 6 as parent. 11 != 9, so here couter lost 2 packets =( Plus the original 2 trigger packets, shown above at rule 2, making 11 packets shown at rule 6. I'm not entirely sure why the total of 12 packets isn't shown (6 packets, each both in and out as you've not specified direction on your rules), but suspect it's to do with the position of your NAT rule, and/or the fact that you're not showing counts for 'via ng1'. > Another usefull feature is that that check-state must have body. > You must allow user to check same conditions as for other rules. For > example: > check-state icmp via ng0 > It will prevent user from having unexpected maching for flows on other > interfaces (ng1 interface as in my example). > Also I think it will be handy to have many tables for dynamic rules. > It is like to have many routing tables. For example: > ipfw add xxx check-state 1 via ng0 > ipfw add xxx check-state 2 via ng1 > ipfw add xxx skipto yyy icmp via ng0 keep-state 1 > ipfw add xxx skipto yyy icmp via ng1 keep-state 2 I suggest you'll need to study the code of /usr/src/sbin/ipfw/ipfw2.c and /sys/netinet/ip_fw*.[ch] to find out just how it works now, before writing code to implement the above. I can't see anyone else taking an interest in implementing such a thing (but I've been wrong before :) > ipfw is flexible firewall so it must remain its flexibility > But now ipfw has not its flexibility for check-state/keep-state because of > it does not allow to control what I want to check-state/keep-state You just need to understand how ipfw stateful rules really work, and code your rules to match that; then I expect you will enjoy success. cheers, Ian (professing no more than a VERY minimal grasp of this code ..) From owner-freebsd-ipfw@FreeBSD.ORG Fri Nov 28 23:45:43 2008 Return-Path: Delivered-To: freebsd-ipfw@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F92F1065677 for ; Fri, 28 Nov 2008 23:45:43 +0000 (UTC) (envelope-from kes-kes@yandex.ru) Received: from forwards4.yandex.ru (forwards4.yandex.ru [77.88.32.20]) by mx1.freebsd.org (Postfix) with ESMTP id 3D4858FC16 for ; Fri, 28 Nov 2008 23:45:42 +0000 (UTC) (envelope-from kes-kes@yandex.ru) Received: from smtp15.yandex.ru (smtp15.yandex.ru [77.88.32.85]) by forwards4.yandex.ru (Yandex) with ESMTP id 8FF164C5560; Sat, 29 Nov 2008 02:45:38 +0300 (MSK) Received: from 152-65-113-92.pool.ukrtel.net ([92.113.65.152]:43785 "EHLO kes" smtp-auth: "kes-kes" TLS-CIPHER: TLS-PEER-CN1: ) by mail.yandex.ru with ESMTP id S4866148AbYK1Xpg (ORCPT + 1 other); Sat, 29 Nov 2008 02:45:36 +0300 X-Yandex-Spam: 1 X-Yandex-Front: smtp15 X-Yandex-TimeMark: 1227915936 X-BornDate: 1149541200 X-Yandex-Karma: 0 X-Yandex-KarmaStatus: 0 X-MsgDayCount: 3 X-Comment: RFC 2476 MSA function at smtp15.yandex.ru logged sender identity as: kes-kes Date: Sat, 29 Nov 2008 01:45:17 +0200 From: KES X-Mailer: The Bat! (v4.0.24) Professional Organization: SaftTen X-Priority: 3 (Normal) Message-ID: <747868075.20081129014517@yandex.ru> To: Ian Smith In-Reply-To: <20081128161236.P92549@sola.nimnet.asn.au> References: <200811232342.mANNgOnr069400@freefall.freebsd.org> <20081124203046.I43853@sola.nimnet.asn.au> <1638065040.20081125011317@yandex.ru> <20081128161236.P92549@sola.nimnet.asn.au> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----------FFFE12A185F5BA1" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-ipfw@freebsd.org Subject: Re[3]: kern/129103: [ipfw] IPFW check state does not work =( X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: KES List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Nov 2008 23:45:43 -0000 ------------FFFE12A185F5BA1 Content-Type: text/plain; charset=windows-1251 Content-Transfer-Encoding: 8bit Здравствуйте, Ian. >> But now NOTICE: 11 packets for rule 6 and 9 (7+2) packets for dinamic rule >> with rule 6 as parent. 11 != 9, so here couter lost 2 packets =( IS> Plus the original 2 trigger packets, shown above at rule 2, making 11 IS> packets shown at rule 6. I'm not entirely sure why the total of 12 IS> packets isn't shown (6 packets, each both in and out as you've not IS> specified direction on your rules), but suspect it's to do with the IS> position of your NAT rule, and/or the fact that you're not showing IS> counts for 'via ng1'. IS> There are 11 packets because of first packet received from ng1 do not counted. This ok here. But dynamic rules must count packets when they are created (see plus sign on attached picture). Actually packets are pass them but not counted ( PS. Attachment that do not pass size control in last message =( >>I suggest you'll need to study the code of /usr/src/sbin/ipfw/ipfw2.c >>and /sys/netinet/ip_fw*.[ch] to find out just how it works now, before >>writing code to implement the above. I can't see anyone else taking an >>interest in implementing such a thing (but I've been wrong before :) Any way check-state must have body, to control whick trafic we want to check for dynamic rules. If you afraid of mistakes in firewall for dynamic rules are created but never check, for this check-state log can be added, so for expired dynamic rules, but no one packet had pass through them log message will be added to /var/log/security WARNING: dynamic rule has expired and no traffic pass through it >> Another usefull feature is that that check-state must have body. >> You must allow user to check same conditions as for other rules. For >> example: >> check-state icmp via ng0 >> It will prevent user from having unexpected maching for flows on other >> interfaces (ng1 interface as in my example). >> Also I think it will be handy to have many tables for dynamic rules. >> It is like to have many routing tables. For example: >> ipfw add xxx check-state 1 via ng0 >> ipfw add xxx check-state 2 via ng1 >> ipfw add xxx skipto yyy icmp via ng0 keep-state 1 >> ipfw add xxx skipto yyy icmp via ng1 keep-state 2 For last two days I have testing dynamic rules I notice next for this scheme of load balancing with two connection to different ISP 03510 check-state 03511 prob 0.500000 skipto 3531 ip from any to any via ng* 03521 skipto 3522 ip from any to any keep-state 03523 setfib 0 ip from any to any 03525 skipto 3550 ip from any to any 03531 skipto 3532 ip from any to any keep-state 03533 setfib 1 ip from any to any 03535 skipto 3550 ip from any to any 03990 allow ip from any to any via ng* Such services as ICQ does not work properly. Because of some idle time other dynamic rule is created and so ICQ client has lost its connection because it is actually change its IP So here it will be use full to point how long dynamic rule will exist 03521 skipto 3522 ip from any to any 5190 keep-state 0 << stands to keep dynamic rule until TCP->FIN is received \ or some sysctl ....MAX_TIME variable 03521 skipto 3522 ip from any to any keep-state 60 << here we want to extend/prolong dynamic rule will exists 03521 skipto 3522 ip from any to any keep-state <