From owner-freebsd-net@freebsd.org Wed Dec 23 04:42:36 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A33CA4F81C; Wed, 23 Dec 2015 04:42:36 +0000 (UTC) (envelope-from fullermd@over-yonder.net) Received: from mail.infocus-llc.com (mail.infocus-llc.com [199.15.120.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 462AC108B; Wed, 23 Dec 2015 04:42:35 +0000 (UTC) (envelope-from fullermd@over-yonder.net) Received: from draco.over-yonder.net (c-75-65-60-66.hsd1.ms.comcast.net [75.65.60.66]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.tarragon.infocus-llc.com (Postfix) with ESMTPSA id 3pQMMt0f0LzTB; Tue, 22 Dec 2015 22:42:34 -0600 (CST) Received: by draco.over-yonder.net (Postfix, from userid 100) id 3pQMMs2JqkzqZ; Tue, 22 Dec 2015 22:42:33 -0600 (CST) Date: Tue, 22 Dec 2015 22:42:33 -0600 From: "Matthew D. Fuller" To: Garrett Wollman Cc: freebsd-net@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Have I got this VIMAGE setup correct? Message-ID: <20151223044233.GM33115@over-yonder.net> References: <22137.33475.645324.203196@hergotha.csail.mit.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <22137.33475.645324.203196@hergotha.csail.mit.edu> X-Editor: vi X-OS: FreeBSD X-Virus-Scanned: clamav-milter 0.99 at mail.tarragon.infocus-llc.com X-Virus-Status: Clean User-Agent: Mutt/1.5.24-fullermd.4 (2015-08-30) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Dec 2015 04:42:36 -0000 On Tue, Dec 22, 2015 at 12:05:07PM -0500 I heard the voice of Garrett Wollman, and lo! it spake thus: > > The consensus when I asked seemed to be that VIMAGE+jail was the > right combination to give every container its own private loopback > interface, so I tried to build that. I noticed a few things: I've got a server running a dozen or so VIMAGE jails, so I can at least chime in a little... > 1) The kernel prints out a warning message at boot time that VIMAGE > is "highly experimental". Should I be concerned about running this > in production? It hasn't blown up anything for me yet. > 2) Stopping jails with virtual network stacks generates warnings from > UMA about memory being leaked. I'm given to understand that's Known, and presumably Not Quite Trivial To Fix. Since I'm not starting/stopping jails repeatedly as a normal runtime thing, I'm ignoring it. If you were spinning jails up and down dynamically dozens of times a day, I'd want to look more closely at just what is leaking and why... > 3) It wasn't clear (or documented anywhere that I could see) how to > get the host network set up properly. Obviously I'm not going to > have a vlan for every single jail, so it seemed like what most > people were doing was "bridge" along with a bunch of "epair" > interfaces. I ended up with the following: Is what I'm doing, though I'm creating the epair's and adding them to the bridges in the setup script rather than rc.conf (exec.prestart in jail.conf), because that makes it a more manageable IME, and since I'm already doing a bunch of setup in the script anyway... > In each of the jails I have to manually configure a MAC address > using /etc/start_if.epairNb to ensure that it's globally unique, but > then everything seems to work. I hardcode (well, dynamically generated hardcoded) MAC addresses on the epair's in the setup script, since bit me hard when I was first setting it up. -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream.