From owner-freebsd-stable@FreeBSD.ORG  Thu Feb  6 00:24:57 2014
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: freebsd-stable@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 90FCD77F;
 Thu,  6 Feb 2014 00:24:57 +0000 (UTC)
Received: from secure.freebsdsolutions.net (secure.freebsdsolutions.net
 [69.55.234.48])
 (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id 6DA0C1B7C;
 Thu,  6 Feb 2014 00:24:56 +0000 (UTC)
Received: from [10.10.1.198] (office.betterlinux.com [199.58.199.60])
 (authenticated bits=0)
 by secure.freebsdsolutions.net (8.14.4/8.14.4) with ESMTP id s1608ShM073873
 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT);
 Wed, 5 Feb 2014 19:08:29 -0500 (EST)
 (envelope-from lists@jnielsen.net)
From: John Nielsen <lists@jnielsen.net>
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: quoted-printable
Subject: IPFW fwd not working after upgrade from 9.2 to 10.0
Date: Wed, 5 Feb 2014 17:08:24 -0700
Message-Id: <8C9CDEF4-A44A-4207-BB87-DA3E7CF89917@jnielsen.net>
To: "freebsd-stable@freebsd.org Stable" <freebsd-stable@freebsd.org>
Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\))
X-Mailer: Apple Mail (2.1827)
X-DCC-x.dcc-servers-Metrics: ns1.jnielsen.net 104; Body=2 Fuz1=2 Fuz2=2
X-Virus-Scanned: clamav-milter 0.97.8 at ns1.jnielsen.net
X-Virus-Status: Clean
Cc: freebsd-ipfw@freebsd.org
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Thu, 06 Feb 2014 00:24:57 -0000

I have been using IPFW FWD to do per-interface routing on a VM instance. =
The default gateway is on interface vtnet0, but there is a second =
interface, vtnet1, on a different network with its own public IP =
address. The second network has its own gateway, which I'd like to use =
for responses to connections coming on on vtnet1. Under 9.2, the below =
worked fine:

fwd ${GW2} ip from ${PUBIP2} to not table(120) out via vtnet0

Table 120 contains all the local networks for which I don't want the =
rule to apply.

I updated the VM to 10.0-RELEASE, with no changes to the IPFW rules or =
network configuration. The forwarding to the secondary router no longer =
works. Traffic comes in on ${PUBIP2} fine, and the counter for the IPFW =
rule increments, but no packets are actually sent out vtnet1. Instead, =
it's trying to do a weird ARP query:


# tcpdump -n -p -i vtnet1
...
16:46:33.146324 IP ${OUTSIDE_IP}.55063 > ${PUBIP2}.22: Flags [S], seq =
2242981455, win 65535, options [mss 1460,nop,wscale 4,nop,nop,TS val =
1978614336 ecr 0,sackOK,eol], length 0
16:46:33.146372 ARP, Request who-has ${GW1} tell ${PUBIP2}, length 28

If I try to SSH from an outside IP to the public IP on vtnet1, a =
response never goes out either interface (vtnet0 or vtnet1). Instead, an =
ARP query is going out (on vtnet1) looking for the default gateway IP, =
which is only reachable on vtnet0.

On the off chance this is not a bug, is there a better way I should be =
doing per-interface routing under FreeBSD 10? If it is a bug, can anyone =
suggest what might be going on here and how to track it down further?

Thanks,

JN