From owner-freebsd-net@FreeBSD.ORG Sun Jul 22 15:24:53 2007 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 37E3016A417 for ; Sun, 22 Jul 2007 15:24:53 +0000 (UTC) (envelope-from m_wlist@weirdwire.ru) Received: from weirdwire.ru (weirdwire.ru [85.118.229.210]) by mx1.freebsd.org (Postfix) with ESMTP id 45D1613C442 for ; Sun, 22 Jul 2007 15:24:51 +0000 (UTC) (envelope-from m_wlist@weirdwire.ru) Received: from mail.weirdwire.ru (localhost [127.0.0.1]) by weirdwire.ru (Postfix) with ESMTP id AA9151D1C82 for ; Sun, 22 Jul 2007 22:07:24 +0700 (NOVST) Received: from 10.23.23.1 (SquirrelMail authenticated user m_wlist) by mail.weirdwire.ru with HTTP; Sun, 22 Jul 2007 22:07:24 +0700 (NOVST) Message-ID: <51976.10.23.23.1.1185116844.squirrel@mail.weirdwire.ru> Date: Sun, 22 Jul 2007 22:07:24 +0700 (NOVST) From: m_wlist@weirdwire.ru To: freebsd-net@freebsd.org User-Agent: SquirrelMail/1.4.10a MIME-Version: 1.0 Content-Type: text/plain;charset=utf-8 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal References: In-Reply-To: In-Reply-To: Subject: Policy-based routing for packets originating from local machine ('reinject' packets back into kernel?) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Jul 2007 15:24:53 -0000 Hello. I have a FreeBSD machine connected to internal network and several ISPs. I have set up (with pf) nat and balanced (policy-based) routing for machines from internal network, proper routing for remote connections to interfaces that connected to corresponding ISPs. One thing that I can't manage to get done is to make balanced routing to work with packets originating from the router itself. Simple pf route-to rules don't work as it seems local packets don't have any 'in' interface -- they just emerge somewhere in kernel just before routing table and when they are accessible to pf they are already in 'out' part where policy routing is not possible (the packet are already going out on interface that is determined from rather static routing table). When there are no default route in routing table packets just have nowhere to go and fail not even reaching any pf rules. I suddenly came up with an idea how to try to get that done -- somehow 'reinject' packets back into kernel to get them processed by 'in' rule with 'route-to'. The idea sounds quite simple -- make some virtual interface with non-existing address and subnet and make default route be somewhere in that subnet. Then make another virtual interface and bridge/connect (with netgraph for example) it to first one the way that any packets sent out to first one appear back in on second. The question number one: Is there any less retarded way to do PBR for local packets? The question number two: How to do that properly? At the moment I'm trying to get that working with netgraph's ngeth interfaces. But they seem to behave in some really weird way. Details: # ifconfig ngeth0 10.42.42.1 netmask 255.255.255.250 # ngctl connect ngeth0: ngeth1: lower upper # ngctl connect ngeth1: ngeth0: lower upper ('tcpdump -ni ngeth0' on other terminal for great justice) # ping 10.42.42.2 (here after some delay I get 'host is down' messages with no output from tcpdump). # ping 10.42.42.5 (broadcast address, gives nothing from ping, and 'blal blah 10.42.42.1 > 10.42.42.5: ICMP echo request, blah' from tcpdump) (here i change tcpdump from ngeth0 to ngeth1) # ping 10.42.42.2 and # ping 10.42.42.3 give 'host is down' from ping and nothing from tcpdump # ping 10.42.42.4 (LOL WUT!) still gives 'host is down' from ping, but tcpdump -ni ngeth1 gives 'arp who-has 10.42.42.4 tell 10.42.42.1'! That raises two questins: 1) Wtf is going on? 2) How to make ngeth just send ip packet, avoiding that arp stuff (or is there any other virtual interface devices available that do that)? Seems like I don't understand something completely.