From owner-freebsd-jail@freebsd.org Mon Feb 12 11:06:54 2018 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E0E7F0F06B; Mon, 12 Feb 2018 11:06:54 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6AEE46E1D4; Mon, 12 Feb 2018 11:06:53 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from freyja.zeit4.iv.bundesimmobilien.de ([87.138.105.249]) by mail.gmx.com (mrgmx001 [212.227.17.190]) with ESMTPSA (Nemesis) id 0LgptO-1eQvZ52KQg-00oG2L; Mon, 12 Feb 2018 12:06:45 +0100 Date: Mon, 12 Feb 2018 12:06:39 +0100 From: "O. Hartmann" To: freebsd-jail@freebsd.org, freebsd-current Cc: "O. Hartmann" , Olivier =?ISO-8859-1?Q?Cochard?= =?ISO-8859-1?Q?-Labb=E9?= Subject: Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing Message-ID: <20180212120639.7931f609@freyja.zeit4.iv.bundesimmobilien.de> In-Reply-To: <20180212093747.20024e5f@freyja.zeit4.iv.bundesimmobilien.de> References: <20180208093052.7f5d7a98@freyja.zeit4.iv.bundesimmobilien.de> <20180209172259.1ec9b9f4@thor.intern.walstatt.dynvpn.de> <2D57FE3A-744A-4A44-B572-5338AB9E187D@lists.zabbadoz.net> <20180210085248.7b9af104@thor.intern.walstatt.dynvpn.de> <20180212093747.20024e5f@freyja.zeit4.iv.bundesimmobilien.de> Organization: Walstatt MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K0:22WH3ZncUiXy9S5kWV1v51sv7V+50pBCZke/JciCicLz9S9gyAO uLexVLwD0Gq7+GOX0jXs/Z1y/FHAkrbtlkx5Kc0PsL0MgCiXkzgFyNhNbP1/aX2vkKZSxh2 yrEUbk3xb6/bkJeHSoTrJ/iFftAjKZOHsIY0xD2fiAj9qzEQnKji2Mjt35rdcz5ZL51tiWO QCfeADEBExffj3NYYgrSg== X-UI-Out-Filterresults: notjunk:1;V01:K0:Kwjy5f30NRA=:8FZUJATBQYjiq0/CblSDU8 0MsgUT+Yh5XaPdTjaYsyAOvIsBO05w2+eMpuMgdKJZLACZ1nWzY7Fzy+W0TyffJehOKkgKhCh Ebd2witOnGXRWhzeSn3YIN7uPgZcBuz42TmefxuwVkT445gX+fICAiNjSCnRBOJxNKnOZiavP kBwRZ7BhZ0xl95Em4HBNQ8NWoU1jd0L0CxHmif/VcDUnDLMHohfvzx7lmBIbQz1b91wQusmPI XN8BlwJL9dlKS6lZfmrGTLvfYaQb6FhUsfhtbAvsbnFEBekMgGhLyD18PjtsbOPUWussgjXHT lDs6SgfE0mv9+MhGN3KwIaaHykpZZWgWyNJGij/71bJzY+TFRA3cEnxHtJl4qbnTRogFpmkzx cwxsFdnnG+cHbAhdoadb41uKYCsjSugGijj7TOkdlWd8BqcSF5ZVJ8bO4tEcNsPvrGcg+fpnO GE9zdO8iCnr5yKoZMlLkG9JXmy16kTASeEHW+hi6MVPFNTeoubR0VhAdqcqmDKNKUu+wgDsEG /Iu15JVtw1jJdB+gYFd7QHX3Cgu6ZU4oOPv9Vk2jdNM+wj+yGtKJz+LGhN78bEvK7jjNmXWoH dXFM2hQ8uXXNCmP8ZT8UnCkMzBGlk77GV9bzVh/ilH0mxrT2X1KMY52LSGAC4frJTv1w1SJ7T SLchy8b7nICtYK0LFe89sqQQlXBdzz3+RvXARrU+QcEAPip8EtHGJJWOkMC+3QxoI/aG+aJuu Y4zdGAEm/Guu5u6bpIQAbsEQ6Qz4rnc2dXc2gwGQkz0Lr39+nVQuNkOu6yaLXfgFeOjglzEgt wk5sqjpRblTyeKgQeO/+I6sd3Q/yN3tsVJymhODflO9+mR0Osg= X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2018 11:06:54 -0000 On Mon, 12 Feb 2018 09:37:47 +0100 "O. Hartmann" wrote: > On Sat, 10 Feb 2018 11:52:18 +0100 > Olivier Cochard-Labb=C3=A9 wrote: >=20 > > On Sat, Feb 10, 2018 at 8:52 AM, O. Hartmann w= rote: > > =20 > > > > > > The moment any of the bridges gets an additional member epair interfa= ce > > > (so the bridge > > > has at least three members including the on reaching into the virtual > > > router jail) the > > > vbridge seems to operate unpredictable (to me). Pinging jails memeber= of > > > that vbridge > > > are unreachable. > > > > > > =20 > > =E2=80=8BFirst idea: > > Did you try with a more simple setup, like with just 3 jails members of= one > > bridge ? =20 > > =3D> I've tried it on a -head, and all 4 members (3 jails and the host= ) =20 > > reach to communicate. > >=20 > > Second idea: > > Can you check that all epairs have different MAC address each ?=E2=80=8B > > I hit this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D176= 671 =20 >=20 > Wow, that PR is from 2013(!) and it is still not solved? >=20 > >=20 > > Regards, > >=20 > > Olivier > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.o= rg" =20 >=20 > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" After rebooting recent CURRENT, the view with "ifconfig -a ether" looked go= od so far, each epair/bridge has its unique MAC. But then, login on the jails and checking the epair's counterpart owned by = the VIMAGE jail, I found almost EVERY jail has the same MAC, even those jails members of the same bridge: [...] jail 11 on bridge2 epair20129a: flags=3D8843 metric 0 = mtu 1500 options=3D8 ether 02:68:d0:00:07:0a jail 10 on bridge2 epair20128a: flags=3D8843 metric 0 = mtu 1500 options=3D8 ether 02:68:d0:00:07:0a jail 9 on bridge 1 epair10250a: flags=3D8843 metric 0 = mtu 1500 options=3D8 ether 02:68:d0:00:07:0a jail 8 on bridge1 epair10238a: flags=3D8843 metric 0 = mtu 1500 options=3D8 ether 02:68:d0:00:07:0a jail 7 on bridge0 epair238a: flags=3D8843 metric 0 mt= u 1500 options=3D8 ether 02:68:d0:00:07:0a epair251a: flags=3D8843 metric 0 mt= u 1500 options=3D8 ether 02:68:d0:00:07:0a The way I create epairs and put them into a jail's context/domain is straig= ht forward. In jail.conf, I have more generic setup with variables like: # DNS Master ns1 { $if =3D "2"; $ip4_addr =3D "10.10.0.${if}"; $ip4_cidr =3D "24"; $ip4_my_default_route =3D "10.10.0.1"; $vnet_if =3D "epair${if}"; $home_bridge =3D "${if_bridge_core}"; depend=3D "vrouter"; allow.raw_sockets=3D "1"; } and in the common portion of jail.conf definitions, I use this: [...] vnet =3D "new"; vnet.interface =3D "${vnet_if}a"; persist; exec.clean; exec.start=3D ""; exec.start+=3D "/sbin/ifconfig ${vnet_if}a inet ${ip4_addr}/${ip4_cidr} up"; exec.start+=3D "/bin/sh /etc/rc"; exec.start+=3D "/sbin/route add default ${ip4_my_default_route}"; exec.start+=3D "/sbin/sysctl net.link.bridge.pfil_member=3D0"; exec.start+=3D "/sbin/sysctl net.link.bridge.pfil_bridge=3D0"; exec.start+=3D "/sbin/sysctl net.link.bridge.pfil_onlyip=3D0"; exec.stop=3D "/bin/sh /etc/rc.shutdown"; exec.prestart=3D ""; exec.prestart+=3D "ifconfig ${vnet_if} create"; exec.prestart+=3D "ifconfig ${vnet_if}b up"; exec.prestart+=3D "ifconfig ${home_bridge} addm ${vnet_if}b up"; exec.poststop=3D "ifconfig ${home_bridge} deletem ${vnet_if}b"; exec.poststop+=3D "ifconfig ${vnet_if}b destroy"; exec.consolelog=3D "/var/log/jail_${name}_console.log"; The big question here is: when a jail with VIMAGE kernel "swallows" a epair-pseudo device, it leaves the ciontext or visibility of the host. How = can the FreeBSD VIMAGE kernel then know about what former epair's MAC was? Is t= his mechanism maybe the culprit? It is just a thought, so I do not want to be beheaded - I'm not much into system development. Kind regards, O. Hartmann