From owner-freebsd-net@freebsd.org Tue Dec 22 17:05:14 2015 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 62F54A4E8D3; Tue, 22 Dec 2015 17:05:14 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (wollman-1-pt.tunnel.tserv4.nyc4.ipv6.he.net [IPv6:2001:470:1f06:ccb::2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 126A41770; Tue, 22 Dec 2015 17:05:13 +0000 (UTC) (envelope-from wollman@hergotha.csail.mit.edu) Received: from hergotha.csail.mit.edu (localhost [127.0.0.1]) by hergotha.csail.mit.edu (8.14.9/8.14.9) with ESMTP id tBMH57IP097108; Tue, 22 Dec 2015 12:05:07 -0500 (EST) (envelope-from wollman@hergotha.csail.mit.edu) Received: (from wollman@localhost) by hergotha.csail.mit.edu (8.14.9/8.14.4/Submit) id tBMH57h1097105; Tue, 22 Dec 2015 12:05:07 -0500 (EST) (envelope-from wollman) MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <22137.33475.645324.203196@hergotha.csail.mit.edu> Date: Tue, 22 Dec 2015 12:05:07 -0500 From: Garrett Wollman To: freebsd-net@freebsd.org, freebsd-stable@freebsd.org Subject: Have I got this VIMAGE setup correct? X-Mailer: VM 7.17 under 21.4 (patch 22) "Instant Classic" XEmacs Lucid X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (hergotha.csail.mit.edu [127.0.0.1]); Tue, 22 Dec 2015 12:05:07 -0500 (EST) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, HEADER_FROM_DIFFERENT_DOMAINS autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hergotha.csail.mit.edu X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Dec 2015 17:05:14 -0000 The consensus when I asked seemed to be that VIMAGE+jail was the right combination to give every container its own private loopback interface, so I tried to build that. I noticed a few things: 1) The kernel prints out a warning message at boot time that VIMAGE is "highly experimental". Should I be concerned about running this in production? 2) Stopping jails with virtual network stacks generates warnings from UMA about memory being leaked. 3) It wasn't clear (or documented anywhere that I could see) how to get the host network set up properly. Obviously I'm not going to have a vlan for every single jail, so it seemed like what most people were doing was "bridge" along with a bunch of "epair" interfaces. I ended up with the following: network_interfaces="lo0 bridge0 bce0" autobridge_interfaces="bridge0" autobridge_bridge0="bce0 epair0a epair1a" cloned_interfaces="bridge0 epair0 epair1" ifconfig_bridge0="inet [deleted] netmask 0xffffff00" ifconfig_bridge0_ipv6="inet6 [deleted] prefixlen 64 accept_rtadv" ifconfig_bce0="up" ifconfig_epair0a="up" ifconfig_epair1a="up" The net.link.bridge.inherit_mac sysctl, which is documented in bridge(4), doesn't appear to work; I haven't yet verified that I can create a /etc/start_if.bridge0 to set the MAC address manually without breaking something else. The IPv6 stack regularly prints "in6_if2idlen: unknown link type (209)" to the console, which is annoying, and IPv6 on the host doesn't entirely work -- it accepts router advertisements but then gives [ENETUNREACH] trying to actually send packets to the default gateway. (IPv6 to the jails *does* work!) In each of the jails I have to manually configure a MAC address using /etc/start_if.epairNb to ensure that it's globally unique, but then everything seems to work. Does this match up with what other people have been doing? Anything I've missed? Any patches I should pull up to make this setup more reliable before I roll it out in production? -GAWollman