From owner-freebsd-net@FreeBSD.ORG Mon Apr 7 15:31:03 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 25981B99 for ; Mon, 7 Apr 2014 15:31:03 +0000 (UTC) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AD21DDC7 for ; Mon, 7 Apr 2014 15:31:02 +0000 (UTC) Received: by mail-wg0-f41.google.com with SMTP id n12so6905187wgh.0 for ; Mon, 07 Apr 2014 08:31:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=PVwI6kv45XHw1RMU6cl6/u4RwwY4wvwchgoPlb2f56s=; b=BL74D9W2oQLSS0sIIP1cMCa+aV73LoavR2LoQp5vHzgG+slnccQnbsXbUcnb2s6LZL 51KytUsissPpccDKEbod8LR0VMTlHR8EIf1gHXFZb7XBwigdbKvT/APwHTn40DlTPgSb tDTGLofE6RNe+FZwqvqlPoda1Ju53OT3UYkpcsJhhzb2P5KWdeM9uvvgpAalfAvXkfz7 Dh/HgUMizCzbLrG0taQBEL9EC7BSJaIObBI/XvFjss5xS34MBwgmFnyo97FsV33mHFWP yRDJ9rFMFrxI9FW8Sr2CZvhJWewUn7a6CzC/9c+fMzDETKrmmwN3aJvoobP04GBQ1dTa o6HA== MIME-Version: 1.0 X-Received: by 10.180.187.16 with SMTP id fo16mr26341355wic.26.1396884660997; Mon, 07 Apr 2014 08:31:00 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.168.130 with HTTP; Mon, 7 Apr 2014 08:31:00 -0700 (PDT) In-Reply-To: <533F68EF.8060607@nevermind.co.nz> References: <533F68EF.8060607@nevermind.co.nz> Date: Mon, 7 Apr 2014 09:31:00 -0600 X-Google-Sender-Auth: Fp__wGkccCq4qFPu-7-0-eijrh8 Message-ID: Subject: Re: Multihomed system with jails routing issues From: Alan Somers To: Chris Smith Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Apr 2014 15:31:03 -0000 On Fri, Apr 4, 2014 at 8:22 PM, Chris Smith wrote: > Hi All, > > I have a system with 1 network interface with 2 extra VLANs off it and I'm > having some trouble getting the routing working correctly with it and jails. > > bge0 - management - 10.71.100.0/24 > bge0.101 - LAN - 10.71.101.0/24 > bge0.103 - DMZ - 10.71.101.0/24 > > Here's what I want to achieve... > > Host: > I want the host system to only listen on one interface, bge0. I want NO ip > addresses of the host on the vlan interfaces. The only service it will be > exposing is its sshd. The management address for this system is > 10.71.100.50. > > Jails: > The system will also host a variety of jails, each with an IP either on the > LAN or DMZ. I am using ezjail to manage the jails. > > Router: > There is a router at the .254 address of every subnet that can route between > each network. > > I set up jail1 on bge0.101 with the IP 10.71.101.51. Since the host does not > have an address configured on bge0.101, I configured the jail address as /24 > instead of the default /32. > > My issues: > > * If I do not configure the jail as a /24 (e.g. /32), the LAN cannot > communicate with the jail. > > * When the jail is up and 10.71.101.51/24 is active, SSHing from the LAN to > the mgmt interface via the router fails, as the host tries to send return > traffic via the bge0.101 interface, even though traffic arrived via the bge0 > interface. > > So I did a whole lot of research for people having these apparently > problems, and decided to try the multiple routing table/fib approach. So I > recompiled my kernel, configured fib 1 with the LAN interface route (setfib > route add 10.71.101.0/24 -iface bge0.101), set the jail fib and set the > tunable net.addr_all_fibs = 0. I still can't get this working correctly. > ezjail still seems to add the interface route to fib 0 by default (but it > won't if i run ezjail with the setfib 1 command). Routing is very broken when you use multiple FIBs and set net.addr_all_fibs=0. See bugs http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/167947 http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/187549 http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/187550 http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/187551 http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/187552 http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/187553 http://www.freebsd.org/cgi/query-pr.cgi?pr=bin/187699 I'm actively working on fixing the above, but getting consensus on the solution is slow. If you can't solve your problem without using multiple FIBs, I will suggest a set of patches that may help you. I will need 1) Your ifconfig_bge0 etc lines from /etc/rc.conf 2) The output for "setfib X netstat -rn -f inet" for each fib in use. 3) The rc.conf lines for any manually create routes that you're using, for example gateways. Also you may want to look at this possibly relevant bug. I'm not involved with this one: http://www.freebsd.org/cgi/query-pr.cgi?pr=misc/181794 -Alan > > Using FIB 1 and trying to ping hosts on the LAN gives an error like: sendto > failed: invalid argument. > > Does anybody have any best practices for doing this, or anything else I can > try? I'm happy to share/pastebin any configuration and I've tried most > things I've found on the internet. I'm using FreeBSD 10.0 with a custom > kernel for multiple routing tables. > > Thanks in advance! > Chris. > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"