From owner-freebsd-current@freebsd.org Fri Jan 25 18:52:32 2019 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5BE0814BE8EE for ; Fri, 25 Jan 2019 18:52:32 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from mail.monkeybrains.net (mail.monkeybrains.net [208.69.40.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 779CB6A8BD for ; Fri, 25 Jan 2019 18:52:21 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from [10.2.86.182] (public.monkeybrains.net [208.69.41.107] (may be forged)) (authenticated bits=0) by mail.monkeybrains.net (8.15.2/8.15.2) with ESMTPSA id x0PIXT2Z079686 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Fri, 25 Jan 2019 10:33:29 -0800 (PST) (envelope-from crapsh@monkeybrains.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=monkeybrains.net; s=dkim; t=1548441209; bh=8/ZUYB5EqRQFjmnbg+3vO/oW+MvHMZrqq5HCgoCezRg=; h=To:From:Subject:Date; b=H/B1Z/t5deLuMrdkDAYF6bBZSylCDDXz8ZXsgzJxjbpc8mokqG6o9TJj7aZHkM0FF fzEM4hYIjwFBHlXyvOaCydLidNpD2i652EiPLGun0VM5J2rLi9fae5fZQgeyVoiV1H Zn0TuD5J46PJf46UlTnEZ/tur+OemkTHeIfKdMTs= X-Authentication-Warning: mail.monkeybrains.net: Host public.monkeybrains.net [208.69.41.107] (may be forged) claimed to be [10.2.86.182] To: freebsd-current@freebsd.org From: BulkMailForRudy Subject: HOWTO - jails - FreeBSD 12 + VNET + ZFS Message-ID: <62e6f600-b1ba-8900-b7e8-3af0f17fd910@monkeybrains.net> Date: Fri, 25 Jan 2019 10:33:29 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.2.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Virus-Scanned: clamav-milter 0.100.2 at mail.monkeybrains.net X-Virus-Status: Clean X-Rspamd-Queue-Id: 779CB6A8BD X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=monkeybrains.net header.s=dkim header.b=H/B1Z/t5; dmarc=pass (policy=none) header.from=monkeybrains.net; spf=pass (mx1.freebsd.org: domain of crapsh@monkeybrains.net designates 208.69.40.19 as permitted sender) smtp.mailfrom=crapsh@monkeybrains.net X-Spamd-Result: default: False [-3.18 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; HAS_XAW(0.00)[]; TO_DN_NONE(0.00)[]; DKIM_TRACE(0.00)[monkeybrains.net:+]; DMARC_POLICY_ALLOW(-0.50)[monkeybrains.net,none]; MX_GOOD(-0.01)[mx1.harsh.monkeybrains.net]; NEURAL_HAM_SHORT(-0.18)[-0.176,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.02)[country: US(-0.08)]; ASN(0.00)[asn:32329, ipnet:208.69.40.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[monkeybrains.net.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.975,0]; R_DKIM_ALLOW(-0.20)[monkeybrains.net:s=dkim]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_IN_DNSWL_NONE(0.00)[19.40.69.208.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jan 2019 18:52:32 -0000 I love using jails.  For many years, I used a tool to help out: ezjail, now I am just raw-dogging it by using the config file in /etc/jail.conf Here is my config: # /etc/jail.conf # VNET is used to send an epair to each jail. # The epair is renamed jail0 with exec.created in each jail. # exec.prestrt Script creates bridge0 if needed. # Global settings applied to all jails. # haven't found a good reason to run a jail as NOT root exec.system_user  = "root"; exec.jail_user    = "root"; mount.devfs; allow.raw_sockets; devfs_ruleset     = "5"; # Networking and the exec cycle $uplinkdev        = "ix0"; vnet; vnet.interface    = "jail0";               # default vnet interface exec.prestart     = "ifconfig bridge0 > /dev/null 2> /dev/null || ( ifconfig bridge0 create up && ifconfig bridge0 addm $uplinkdev )"; exec.prestart    += "ifconfig $epair create up                 || echo 'Skipped creating epair (exists?)'"; exec.prestart    += "ifconfig bridge0 addm ${epair}a           || echo 'Skipped adding bridge member (already member?)''"; exec.created      = "ifconfig ${epair}b name jail0             || echo 'Skipped renaming ifdev to jail0'"; exec.clean; exec.start        = "/bin/sh /etc/rc"; exec.stop         = "/bin/sh /etc/rc.shutdown"; exec.poststop     = "ifconfig bridge0 deletem ${epair}a"; #exec.poststop    += "ifconfig ${epair}a destroy"; # Per-jail settings ns1 {     path          = "/data/ns1.monkeybrains.net/";     host.hostname = "ns1.monkeybrains.net";     $epair        = "epair0";  # must be unique in every jail } tac {     path          = "/data/tac.monkeybrains.net/";     host.hostname = "tac.monkeybrains.net";     $epair        = "epair1"; } ===================================== Here is a look at ifconfig before and after jail creation. ============  Before jails start up ============ ix0: flags=8843 metric 0 mtu 1500 options=e53fbb     ether ac:1f:6b:6a:14:78     inet 10.1.2.3 netmask 0xffffff00 broadcast 10.1.2.255     inet6 fe80::ae1f:aaaa:aaaa:1478%ix0 prefixlen 64 scopeid 0x1     inet6 2607:f598::a:a prefixlen 64     media: Ethernet autoselect (1000baseT )     status: active     nd6 options=21 lo0: flags=8049 metric 0 mtu 16384 options=680003     inet6 ::1 prefixlen 128     inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3     inet 127.0.0.1 netmask 0xff000000     groups: lo ix0: flags=8943 metric 0 mtu 1500 options=a538b9     ether ac:1f:6b:6a:14:78     inet 208.69.40.26 netmask 0xffffff00 broadcast 208.69.40.255     inet6 fe80::ae1f:6bff:fe6a:1478%ix0 prefixlen 64 scopeid 0x1     inet6 2607:f598::d045:281a prefixlen 64     media: Ethernet autoselect (1000baseT )     status: active     nd6 options=21 ix1: flags=8802 metric 0 mtu 1500 options=e53fbb     ether ac:1f:6b:6a:14:79     media: Ethernet autoselect     status: no carrier     nd6 options=29 lo0: flags=8049 metric 0 mtu 16384 options=680003     inet6 ::1 prefixlen 128     inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3     inet 127.0.0.1 netmask 0xff000000     groups: lo     nd6 options=21 bridge0: flags=8843 metric 0 mtu 1500     ether 02:16:09:1c:af:00     id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15     maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200     root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0     member: epair1a flags=143             ifmaxaddr 0 port 6 priority 128 path cost 2000     member: epair0a flags=143             ifmaxaddr 0 port 5 priority 128 path cost 2000     member: ix0 flags=143             ifmaxaddr 0 port 1 priority 128 path cost 2000     groups: bridge     nd6 options=1 epair0a: flags=8943 metric 0 mtu 1500     options=8     ether 02:8d:76:e8:34:0a     inet6 fe80::8d:76ff:fee8:340a%epair0a prefixlen 64 scopeid 0x5     groups: epair     media: Ethernet 10Gbase-T (10Gbase-T )     status: active     nd6 options=21 epair1a: flags=8943 metric 0 mtu 1500     options=8     ether 02:7a:d1:7c:f8:0a     inet6 fe80::7a:d1ff:fe7c:f80a%epair1a prefixlen 64 scopeid 0x6     groups: epair     media: Ethernet 10Gbase-T (10Gbase-T )     status: active     nd6 options=21 ============  Start up jails ============ # service jail start Starting jails: ns1 tac. # ifconfig ix0: flags=8943 metric 0 mtu 1500 options=a538b9     ether ac:1f:6b:6a:14:78     inet 10.1.2.3 netmask 0xffffff00 broadcast 10.1.2.255     inet6 fe80::ae1f:aaaa:aaaa:1478%ix0 prefixlen 64 scopeid 0x1     inet6 2607:f598::a:a prefixlen 64     media: Ethernet autoselect (1000baseT )     status: active     nd6 options=21 lo0: flags=8049 metric 0 mtu 16384 options=680003     inet6 ::1 prefixlen 128     inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3     inet 127.0.0.1 netmask 0xff000000     groups: lo     nd6 options=21 bridge0: flags=8843 metric 0 mtu 1500     ether 02:16:09:1c:af:00     id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15     maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200     root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0     member: epair1a flags=143             ifmaxaddr 0 port 6 priority 128 path cost 2000     member: epair0a flags=143             ifmaxaddr 0 port 5 priority 128 path cost 2000     member: ix0 flags=143             ifmaxaddr 0 port 1 priority 128 path cost 2000     groups: bridge     nd6 options=1 epair0a: flags=8943 metric 0 mtu 1500     options=8     ether 02:8d:76:e8:34:0a     inet6 fe80::8d:76ff:fee8:340a%epair0a prefixlen 64 scopeid 0x5     groups: epair     media: Ethernet 10Gbase-T (10Gbase-T )     status: active     nd6 options=21 epair1a: flags=8943 metric 0 mtu 1500     options=8     ether 02:7a:d1:7c:f8:0a     inet6 fe80::7a:d1ff:fe7c:f80a%epair1a prefixlen 64 scopeid 0x6     groups: epair     media: Ethernet 10Gbase-T (10Gbase-T )     status: active     nd6 options=21 # jls    JID  IP Address      Hostname                      Path     19                  ns1.monkeybrains.net /data/ns1.monkeybrains.net     20                  tac.monkeybrains.net /data/tac.monkeybrains.net # jexec ns1 ifconfig lo0: flags=8049 metric 0 mtu 16384 options=680003     inet6 ::1 prefixlen 128     inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1     inet 127.0.0.1 netmask 0xff000000     groups: lo     nd6 options=21 jail0: flags=8842 metric 0 mtu 1500     options=8     ether 02:8d:76:e8:34:0b     groups: epair     media: Ethernet 10Gbase-T (10Gbase-T )     status: active     nd6 options=29 # jexec tac ifconfig lo0: flags=8049 metric 0 mtu 16384 options=680003     inet6 ::1 prefixlen 128     inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1     inet 127.0.0.1 netmask 0xff000000     groups: lo     nd6 options=21 jail0: flags=8842 metric 0 mtu 1500     options=8     ether 02:7a:d1:7c:f8:0b     groups: epair     media: Ethernet 10Gbase-T (10Gbase-T )     status: active     nd6 options=29 vlan91: flags=8003 metric 0 mtu 1500     ether 00:00:00:00:00:00     groups: vlan     vlan: 0 vlanpcp: 0 parent interface:     nd6 options=29