From owner-freebsd-net@FreeBSD.ORG Sat Apr 5 02:56:50 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C55F8677 for ; Sat, 5 Apr 2014 02:56:50 +0000 (UTC) Received: from smtp.webfaction.com (mail6.webfaction.com [74.55.86.74]) by mx1.freebsd.org (Postfix) with ESMTP id AAB02164 for ; Sat, 5 Apr 2014 02:56:49 +0000 (UTC) Received: from [10.71.101.130] (unknown [203.86.207.104]) by smtp.webfaction.com (Postfix) with ESMTP id 60038227E9BF for ; Sat, 5 Apr 2014 02:22:40 +0000 (UTC) Message-ID: <533F68EF.8060607@nevermind.co.nz> Date: Sat, 05 Apr 2014 15:22:39 +1300 From: Chris Smith User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: freebsd-net@freebsd.org Subject: Multihomed system with jails routing issues Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Apr 2014 02:56:50 -0000 Hi All, I have a system with 1 network interface with 2 extra VLANs off it and I'm having some trouble getting the routing working correctly with it and jails. bge0 - management - 10.71.100.0/24 bge0.101 - LAN - 10.71.101.0/24 bge0.103 - DMZ - 10.71.101.0/24 Here's what I want to achieve... Host: I want the host system to only listen on one interface, bge0. I want NO ip addresses of the host on the vlan interfaces. The only service it will be exposing is its sshd. The management address for this system is 10.71.100.50. Jails: The system will also host a variety of jails, each with an IP either on the LAN or DMZ. I am using ezjail to manage the jails. Router: There is a router at the .254 address of every subnet that can route between each network. I set up jail1 on bge0.101 with the IP 10.71.101.51. Since the host does not have an address configured on bge0.101, I configured the jail address as /24 instead of the default /32. My issues: * If I do not configure the jail as a /24 (e.g. /32), the LAN cannot communicate with the jail. * When the jail is up and 10.71.101.51/24 is active, SSHing from the LAN to the mgmt interface via the router fails, as the host tries to send return traffic via the bge0.101 interface, even though traffic arrived via the bge0 interface. So I did a whole lot of research for people having these apparently problems, and decided to try the multiple routing table/fib approach. So I recompiled my kernel, configured fib 1 with the LAN interface route (setfib route add 10.71.101.0/24 -iface bge0.101), set the jail fib and set the tunable net.addr_all_fibs = 0. I still can't get this working correctly. ezjail still seems to add the interface route to fib 0 by default (but it won't if i run ezjail with the setfib 1 command). Using FIB 1 and trying to ping hosts on the LAN gives an error like: sendto failed: invalid argument. Does anybody have any best practices for doing this, or anything else I can try? I'm happy to share/pastebin any configuration and I've tried most things I've found on the internet. I'm using FreeBSD 10.0 with a custom kernel for multiple routing tables. Thanks in advance! Chris.