From owner-freebsd-current@freebsd.org Fri Jan 25 21:30:57 2019 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8D46114C3A76 for ; Fri, 25 Jan 2019 21:30:57 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-io1-xd2d.google.com (mail-io1-xd2d.google.com [IPv6:2607:f8b0:4864:20::d2d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AFF4F73B84 for ; Fri, 25 Jan 2019 21:30:56 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-io1-xd2d.google.com with SMTP id s8so8856218iob.13 for ; Fri, 25 Jan 2019 13:30:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=8k/z0yeSs2ldg5RQaG9GblTyVMcV4GKWcl6WCZNbItk=; b=Svh04JUKVnNtMQAaGOBJ1QeoKTBTTpFki1Fsj6n6V7axPjQBfV9Ow5CTRQSITU3OvH pjZlwvyverppnTjvE48a5s9sGLZRIJfRyVxShL5OOuIcjA90xFbZYRfHy0pLMr4GAU/D dXrdBPR/gf+zFtmWeE6J0TNCm+Ijh9Tf4OpATeMcn1DZ0f6nY6vfxK0t3djvS2wk6f+N VU47OzPcU9sYhQInp6P/IBTm3TnEtjTeJ6GxOGBYWWTua50yuEFsCzCcsPMHXa8bod8C 05bCkp5zEkW5KDH1ZUVD/cMpTngEJS/oGubeLLRLLuc5zp056/bk1ZSxL7/uwrPbkzJy 8j2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=8k/z0yeSs2ldg5RQaG9GblTyVMcV4GKWcl6WCZNbItk=; b=LcAY9Ix4AkA5vW40nQSUdcIkqvvEpQGY6zhp7zIae1f57SR5V5v66mP2Q3YzMdmiam 80GcXtGCbRX32h//GlnseVpgxDLDwfUjeq8tu3TbSyH6dr2S4v3PCfW5JYXqBMmeU1Wt yEkmV1EH6hSUK6CTSIlIRn79dlnT8TzW8HUdYfcSALawlzterZzUGbF5fl5HKo/2EwMV fV7K7nsBHBjoJLj3lBzMy8zmMYV0S3vbxcA20+BXcWcAS2VoGk9Z17StMWoS/+RcaBmt J6N802Fi0URkWSNLqc2HgkLD3uBouGzcR3tShO4CaOVYDEKWePYyZECPIW6vFGye1fGc 71Mw== X-Gm-Message-State: AHQUAuZ/8Or+Ttm7Rli8BuESFVHlQ+G5AqTX0dMUKdAwTkIJBgvr0KPt qY6eh+RriFkErw+TeY+gieHU0sOF X-Google-Smtp-Source: AHgI3Ib4udF7TPCNP+EUQuiqash4zbySdef4Uh2WeKyjSlz6ffdfpPJEzqLSn36M/zlziZwvBuytIA== X-Received: by 2002:a6b:ab85:: with SMTP id u127mr6938711ioe.143.1548451855837; Fri, 25 Jan 2019 13:30:55 -0800 (PST) Received: from [10.0.10.8] (cpe-65-25-53-210.neo.res.rr.com. [65.25.53.210]) by smtp.googlemail.com with ESMTPSA id x21sm10645496iob.84.2019.01.25.13.30.54 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 25 Jan 2019 13:30:54 -0800 (PST) Message-ID: <5C4B800D.1070504@gmail.com> Date: Fri, 25 Jan 2019 16:30:53 -0500 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: BulkMailForRudy CC: freebsd-current@freebsd.org Subject: Re: HOWTO - jails - FreeBSD 12 + VNET + ZFS References: <62e6f600-b1ba-8900-b7e8-3af0f17fd910@monkeybrains.net> In-Reply-To: <62e6f600-b1ba-8900-b7e8-3af0f17fd910@monkeybrains.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: AFF4F73B84 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Svh04JUK; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of luzar722@gmail.com designates 2607:f8b0:4864:20::d2d as permitted sender) smtp.mailfrom=luzar722@gmail.com X-Spamd-Result: default: False [-6.70 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.95)[-0.954,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[d.2.d.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.74)[ip: (-9.20), ipnet: 2607:f8b0::/32(-2.50), asn: 15169(-1.93), country: US(-0.08)] X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jan 2019 21:30:57 -0000 BulkMailForRudy wrote: > I love using jails. For many years, I used a tool to help out: ezjail, > now I am just raw-dogging it by using the config file in /etc/jail.conf > > > Here is my config: > > # /etc/jail.conf > # VNET is used to send an epair to each jail. > # The epair is renamed jail0 with exec.created in each jail. > # exec.prestart Script creates bridge0 if needed. > > # Global settings applied to all jails. > > # haven't found a good reason to run a jail as NOT root > exec.system_user = "root"; > exec.jail_user   = "root"; > mount.devfs; > allow.raw_sockets; > devfs_ruleset    = "5"; > > # Networking and the exec cycle > $uplinkdev       = "ix0"; > vnet; > vnet.interface   = "jail0";              # default > vnet interface > exec.prestart    = "ifconfig bridge0 > /dev/null 2> /dev/null || ( > ifconfig bridge0 create up && ifconfig bridge0 addm $uplinkdev )"; > exec.prestart   += "ifconfig $epair create > up                || echo 'Skipped creating epair > (exists?)'"; > exec.prestart   += "ifconfig bridge0 addm > ${epair}a          || echo 'Skipped adding bridge member > (already member?)''"; > exec.created     = "ifconfig ${epair}b name > jail0            || echo 'Skipped renaming ifdev to jail0'"; > exec.clean; > exec.start       = "/bin/sh /etc/rc"; > exec.stop        = "/bin/sh /etc/rc.shutdown"; > exec.poststop    = "ifconfig bridge0 deletem ${epair}a"; > #exec.poststop   += "ifconfig ${epair}a destroy"; > > # Per-jail settings > ns1 { >    path         = "/data/ns1.monkeybrains.net/"; >    host.hostname = "ns1.monkeybrains.net"; >    $epair       = "epair0"; # must be unique in every jail > } > > tac { >    path         = "/data/tac.monkeybrains.net/"; >    host.hostname = "tac.monkeybrains.net"; >    $epair       = "epair1"; > } > > > ===================================== > > Here is a look at ifconfig before and after jail creation. > > > ============ Before jails start up ============ > > ix0: flags=8843 metric 0 mtu 1500 > options=e53fbb >    ether ac:1f:6b:6a:14:78 >    inet 10.1.2.3 netmask 0xffffff00 broadcast 10.1.2.255 >    inet6 fe80::ae1f:aaaa:aaaa:1478%ix0 prefixlen 64 scopeid 0x1 >    inet6 2607:f598::a:a prefixlen 64 >    media: Ethernet autoselect (1000baseT ) >    status: active >    nd6 options=21 > lo0: flags=8049 metric 0 mtu 16384 > options=680003 >    inet6 ::1 prefixlen 128 >    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 >    inet 127.0.0.1 netmask 0xff000000 >    groups: lo > > ix0: flags=8943 metric 0 > mtu 1500 > options=a538b9 >    ether ac:1f:6b:6a:14:78 >    inet 208.69.40.26 netmask 0xffffff00 broadcast 208.69.40.255 >    inet6 fe80::ae1f:6bff:fe6a:1478%ix0 prefixlen 64 scopeid 0x1 >    inet6 2607:f598::d045:281a prefixlen 64 >    media: Ethernet autoselect (1000baseT ) >    status: active >    nd6 options=21 > ix1: flags=8802 metric 0 mtu 1500 > options=e53fbb >    ether ac:1f:6b:6a:14:79 >    media: Ethernet autoselect >    status: no carrier >    nd6 options=29 > lo0: flags=8049 metric 0 mtu 16384 > options=680003 >    inet6 ::1 prefixlen 128 >    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 >    inet 127.0.0.1 netmask 0xff000000 >    groups: lo >    nd6 options=21 > bridge0: flags=8843 metric 0 mtu > 1500 >    ether 02:16:09:1c:af:00 >    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 >    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 >    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >    member: epair1a flags=143 >            ifmaxaddr 0 port 6 priority 128 path cost 2000 >    member: epair0a flags=143 >            ifmaxaddr 0 port 5 priority 128 path cost 2000 >    member: ix0 flags=143 >            ifmaxaddr 0 port 1 priority 128 path cost 2000 >    groups: bridge >    nd6 options=1 > epair0a: flags=8943 > metric 0 mtu 1500 >    options=8 >    ether 02:8d:76:e8:34:0a >    inet6 fe80::8d:76ff:fee8:340a%epair0a prefixlen 64 scopeid 0x5 >    groups: epair >    media: Ethernet 10Gbase-T (10Gbase-T ) >    status: active >    nd6 options=21 > epair1a: flags=8943 > metric 0 mtu 1500 >    options=8 >    ether 02:7a:d1:7c:f8:0a >    inet6 fe80::7a:d1ff:fe7c:f80a%epair1a prefixlen 64 scopeid 0x6 >    groups: epair >    media: Ethernet 10Gbase-T (10Gbase-T ) >    status: active >    nd6 options=21 > > > > ============ Start up jails ============ > > # service jail start > Starting jails: ns1 tac. > > # ifconfig > > ix0: flags=8943 metric 0 > mtu 1500 > options=a538b9 >    ether ac:1f:6b:6a:14:78 >    inet 10.1.2.3 netmask 0xffffff00 broadcast 10.1.2.255 >    inet6 fe80::ae1f:aaaa:aaaa:1478%ix0 prefixlen 64 scopeid 0x1 >    inet6 2607:f598::a:a prefixlen 64 >    media: Ethernet autoselect (1000baseT ) >    status: active >    nd6 options=21 > lo0: flags=8049 metric 0 mtu 16384 > options=680003 >    inet6 ::1 prefixlen 128 >    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 >    inet 127.0.0.1 netmask 0xff000000 >    groups: lo >    nd6 options=21 > bridge0: flags=8843 metric 0 mtu > 1500 >    ether 02:16:09:1c:af:00 >    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 >    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 >    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >    member: epair1a flags=143 >            ifmaxaddr 0 port 6 priority 128 path cost 2000 >    member: epair0a flags=143 >            ifmaxaddr 0 port 5 priority 128 path cost 2000 >    member: ix0 flags=143 >            ifmaxaddr 0 port 1 priority 128 path cost 2000 >    groups: bridge >    nd6 options=1 > epair0a: flags=8943 > metric 0 mtu 1500 >    options=8 >    ether 02:8d:76:e8:34:0a >    inet6 fe80::8d:76ff:fee8:340a%epair0a prefixlen 64 scopeid 0x5 >    groups: epair >    media: Ethernet 10Gbase-T (10Gbase-T ) >    status: active >    nd6 options=21 > epair1a: flags=8943 > metric 0 mtu 1500 >    options=8 >    ether 02:7a:d1:7c:f8:0a >    inet6 fe80::7a:d1ff:fe7c:f80a%epair1a prefixlen 64 scopeid 0x6 >    groups: epair >    media: Ethernet 10Gbase-T (10Gbase-T ) >    status: active >    nd6 options=21 > > > # jls >   JID IP Address     > Hostname                     Path >    19                 ns1.monkeybrains.net > /data/ns1.monkeybrains.net > >    20                 tac.monkeybrains.net > /data/tac.monkeybrains.net > > > # jexec ns1 ifconfig > jail0: flags=8842 metric 0 mtu 1500 >    options=8 >    ether 02:8d:76:e8:34:0b >    groups: epair >    media: Ethernet 10Gbase-T (10Gbase-T ) >    status: active >    nd6 options=29 > # jexec tac ifconfig > jail0: flags=8842 metric 0 mtu 1500 >    options=8 >    ether 02:7a:d1:7c:f8:0b >    groups: epair >    media: Ethernet 10Gbase-T (10Gbase-T ) >    status: active >    nd6 options=29 > vlan91: flags=8003 metric 0 mtu 1500 >    ether 00:00:00:00:00:00 >    groups: vlan >    vlan: 0 vlanpcp: 0 parent interface: >    nd6 options=29 > > You have to learn to crawl before you can run. Start with a single vnet jail in jail.conf until you get something that works. Fix your post by getting rid of those   characters. Your post subject says + ZFS and you have no ZFS options in your jail.conf. Edit out lo0 on ifconfig displays, they add no info to this post. In the ifconfig before jail start shows ix0 2 time with different ip address. Why? jexec tac ifconfig shows vlan91, but nowhere do you show this being created or assigned to this jail. What is going on here? exec.system_user = "root"; un-necessary, remove exec.jail_user   = "root"; un-necessary, remove allow.raw_sockets; only valid in non-vnet jails devfs_ruleset    = "5"; What is custom contents of this rule #5 The vnet.interface statement needs to be per jail. Each vnet jail must have it's own epair mumber. This is not something you can do in global section. Must be per jail. Do you have any entries in the vnet jails rc.conf file? If so, show. What is your overall goal? Be able to access the public internet? Is this host you are trying to create working vnet jails on, on a LAN or is it the gateway host? How do you test your vnet jail? Keep in mind that just because your vnet jail starts does not mean that its working. Just means nothing fatal happened to cause it to dump. Bye