From owner-freebsd-ipfw@FreeBSD.ORG  Mon Oct  2 06:55:07 2006
Return-Path: <owner-freebsd-ipfw@FreeBSD.ORG>
X-Original-To: freebsd-ipfw@freebsd.org
Delivered-To: freebsd-ipfw@freebsd.org
Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125])
	by hub.freebsd.org (Postfix) with ESMTP id D22E616A407
	for <freebsd-ipfw@freebsd.org>; Mon,  2 Oct 2006 06:55:06 +0000 (UTC)
	(envelope-from subscriber@osk.com.ua)
Received: from gandalf.osk.com.ua (osk.com.ua [195.5.17.76])
	by mx1.FreeBSD.org (Postfix) with ESMTP id 91EB943D46
	for <freebsd-ipfw@freebsd.org>; Mon,  2 Oct 2006 06:55:05 +0000 (GMT)
	(envelope-from subscriber@osk.com.ua)
Received: from localhost (localhost [127.0.0.1])
	by gandalf.osk.com.ua (Postfix) with ESMTP id 9845878C22
	for <freebsd-ipfw@freebsd.org>; Mon,  2 Oct 2006 09:57:54 +0300 (EEST)
Received: from gandalf.osk.com.ua ([127.0.0.1])
	by localhost (gandalf.osk.com.ua [127.0.0.1]) (amavisd-new, port 10024)
	with ESMTP id 53026-10 for <freebsd-ipfw@freebsd.org>;
	Mon,  2 Oct 2006 09:57:53 +0300 (EEST)
Received: from oleg.piramida.com (unknown [192.168.82.111])
	by gandalf.osk.com.ua (Postfix) with ESMTP id C6C9578C1F
	for <freebsd-ipfw@freebsd.org>; Mon,  2 Oct 2006 09:57:53 +0300 (EEST)
Date: Mon, 2 Oct 2006 09:55:02 +0300
From: Oleg Tarasov <subscriber@osk.com.ua>
X-Mailer: The Bat! (v3.64.01 Christmas Edition) Professional
X-Priority: 3 (Normal)
Message-ID: <1667794444.20061002095502@osk.com.ua>
To: freebsd-ipfw@freebsd.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Virus-Scanned: amavisd-new at osk.com.ua
Subject: ipfw forward does not work
X-BeenThere: freebsd-ipfw@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
Reply-To: FreeBSD MailList <subscriber@osk.com.ua>
List-Id: IPFW Technical Discussions <freebsd-ipfw.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw>,
	<mailto:freebsd-ipfw-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-ipfw>
List-Post: <mailto:freebsd-ipfw@freebsd.org>
List-Help: <mailto:freebsd-ipfw-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-ipfw>,
	<mailto:freebsd-ipfw-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 02 Oct 2006 06:55:07 -0000

Hello,

I've got a machine running FreeBSD 6.0. This problem occured on 6.0-p0
and 6.0-p12.

Introduction
=============
I've got two internet connections from two different providers. One
is the main and second for failover. Both interfaces have attached
natd using divert function of ipfw. Here are interface parameters:

ng0: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> mtu 1492
        inet xxx.xxx.xxx.xxx --> XXX.XXX.XXX.XXX netmask 0xffffffff
ng8: flags=88d1<UP,POINTOPOINT,RUNNING,NOARP,SIMPLEX,MULTICAST> mtu 1492
        inet yyy.yyy.yyy.yyy --> YYY.YYY.YYY.YYY netmask 0xffffffff

Here yyy.yyy.yyy.yyy is an IP address of main connection.

routing table looks like this:
-------------------------
default            YYY.YYY.YYY.YYY    UGS         0    21878    ng8
yyy.yyy.yyy.yyy    lo0                UHS         0       51    lo0
xxx.xxx.xxx.xxx    lo0                UHS         0        0    lo0
127.0.0.1          127.0.0.1          UH          0     3810    lo0
192.168.82         link#1             UC          0        0    rl0
192.168.82.253     00:30:4f:27:ae:85  UHLW        1       74    lo0
YYY.YYY.YYY.YYY    yyy.yyy.yyy.yyy    UH          3        0    ng8
XXX.XXX.XXX.XXX    xxx.xxx.xxx.xxx    UH          3        0    ng0
-------------------------

My kernel is compiled using following options:
-------------------------
options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=300
options         IPFIREWALL_DEFAULT_TO_ACCEPT
options         IPFIREWALL_FORWARD
options         IPDIVERT
options         IPSTEALTH
options         DUMMYNET
options         HZ=1000
-------------------------

Both interfaces have real IPs and should simultaneously work supplying
DNS, mail and other services.

Usually this is implemented configuring ipfw fwd command for policy
routing so I've inserted two following lines into ipfw script:
-------------------------
fwd XXX.XXX.XXX.XXX ip from xxx.xxx.xxx.xxx to any out xmit ng8
fwd YYY.YYY.YYY.YYY ip from yyy.yyy.yyy.yyy to any out xmit ng0
-------------------------

This usually works and works on my second server. But for some reason
here I met strange behaviour. It just seems that fwd command does not
do anything at all.

When I ping xxx.xxx.xxx.xxx (which is failover one) icmp packets come
into ng0 but replies from xxx.xxx.xxx.xxx go through default route on
ng8. This should be normal if there were no fwd commands. But I see
counters on the rule increasing and logging these rules shows
following lines:
Oct  2 08:35:49 central kernel: ipfw: 20500 Forward to XXX.XXX.XXX.XXX
     ICMP:0.0 xxx.xxx.xxx.xxx some.outer.ip.address out via ng8

but packets still go out through ng8 using default route.
There can be two reasons as I see. First is that fwd command does not
work for some reason and the second is that system routing table
considered that default route is preferrable over direct route to
router. The second near impossible so I wonder...

Please tell me if possible how to locate the possible reason of this
problem!

-- 
Best regards,
 Oleg Tarasov                          mailto:subscriber@osk.com.ua