From owner-freebsd-questions@freebsd.org Tue Feb 14 07:56:05 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C50DCDE49A for ; Tue, 14 Feb 2017 07:56:05 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E63101BF0 for ; Tue, 14 Feb 2017 07:56:04 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id v1E7tp3O081675; Tue, 14 Feb 2017 18:55:52 +1100 (EST) (envelope-from smithi@nimnet.asn.au) Date: Tue, 14 Feb 2017 18:55:51 +1100 (EST) From: Ian Smith To: Robert Eckardt cc: freebsd-questions@freebsd.org Subject: Re: A simple routing question - SOLVED In-Reply-To: Message-ID: <20170214182539.U48485@sola.nimnet.asn.au> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 14 Feb 2017 07:56:05 -0000 In freebsd-questions Digest, Vol 662, Issue 7, Message: 6 On Sat, 11 Feb 2017 20:27:37 +0100 Robert Eckardt wrote: > Hi again, > > thanks to all, who helped me by pointing me to some interesting > articles on this topic. > Eventually, I found a solution for my problem by employing > policy based routing with layer-2 ipfw rules and multiple routing > tables (FIBs). > What makes things a little more complicated is the fact that both > routers are addressed via the same interface. > > This summary is for the records for those who face the same problem. > > Using ECMP routing with kernel option RADIX_MPATH and two default > routes proved unsuitable. > a) since the two routers do NAT, packets must be returned through > the same router as the connection was established and not the > one based on destination address. (This is different from the > situation without NAT.) > b) After one router had failed for a short time (disconnected from > power supply) a telnet to a host in the internet failed with > "no route to host" while ping worked. > I did not dig deeper into it as I concluded from several > postings that the RADIX_MPATH feature is still unstable. > > Instead, (thanks to Matthew) I decided for policy based routing. > Getting ipfw to filter the source MAC wasn't difficult. > Understanding the behaviour of the stateful rules and the effect > of layer-2 or layer-3 filtering was hard. > > Finally the following configuration works (for me): > > My SOLUTION: > 1) since we need another forward information base, increase available > FIBs > in /boot/loader.conf > net.fibs="2" > > 2) (after reboot) set alternate default routing (this needs to get set > on every boot, so I put it in /etc/rc.local) > setfib 1 route delete default > setfib 1 route add default 10.0.0.2 > > 3) Don't forget to also add routes to provider-specific infrastucture > (e.g. name servers) via the correspondig router ('setfib 0' opt.) > (This can go as a static route in /etc/rc.conf or in /etc/rc.local) > setfib 0 route add 2.2.3.4 10.0.0.2 > > 4) make sure that the logic of layer-3 rules works as before > 2000 skipto 10000 ip from any to any not layer2 > > 5) add layer-2 rule to ipfw > 3000 allow tag 101 ip from any to any MAC any 08:9a:bc:de:ff:ff in recv re1 > > 6) avoid matching of the default deny rule for layer-2 by accepting > everything in ether_demux > 9000 allow ip from any to any layer2 > 10000 > > 7) switch on filtering of layer-2 packets > (either in /etc/rc.local or in /etc/sysctl.conf) > net.link.ether.ipfw="1" If in /etc/rc.local, that obviously? needs preceding with 'sysctl ' > 8) use stateful information on layer-3 > 20000 check-state > 20100 setfib 1 ip from any to any tagged 101 keep-state > The incoming packet, identified on layer-2 by the MAC addres of > Router2, is handled by the alternate FIB with the default route > pointing back to Router2. > > 9) make sure that in the end packets get logged if some rules don't > work as expected > 65500 deny log ip from any to any MAC any any <-- layer-2 > 65530 deny log ip from any to any <-- layer-3 > > > 10) when accessing Router2, which forwards to Server, via Router1 and > the internet, you will see working dynamic rules > ## Dynamic rules (1 144): > 20100 STATE tcp 1.1.1.123 51620 <-> 10.0.0.2 643 > > > I found (besides 'man ipfw' ;-) the following links most helpful > http://daemonforums.org/showthread.php?t=4610 > https://forums.freebsd.org/threads/2201/ > https://blog.plitc.eu/2014/freebsd-10-1-pbr-ecmp-fib-und-ipfw-layer2-filtering/ > > Comments welcome. > > Regards, > Robert Very elegant. I haven't done anything at layer2 for years, not have I used tagging, but it smells good; happy to take your word it's working. Would you care to post that - including your original question and diagram below - to freebsd-ipfw@freebsd.org, where I think it would be a) well appreciated and b) scrutinised by those who know ipfw best? cheers, Ian