From owner-freebsd-jail@freebsd.org Mon Feb 12 08:29:11 2018 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15F1BF02004; Mon, 12 Feb 2018 08:29:11 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7B49F8663E; Mon, 12 Feb 2018 08:29:09 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from freyja.zeit4.iv.bundesimmobilien.de ([87.138.105.249]) by mail.gmx.com (mrgmx103 [212.227.17.168]) with ESMTPSA (Nemesis) id 0MCcE2-1euXQ602Gw-009RRe; Mon, 12 Feb 2018 09:29:08 +0100 Date: Mon, 12 Feb 2018 09:29:01 +0100 From: "O. Hartmann" To: freebsd-jail@freebsd.org, freebsd-current Cc: Olivier =?ISO-8859-1?Q?Cochard-Labb=E9?= , "O. Hartmann" Subject: Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing Message-ID: <20180212092901.671488e6@freyja.zeit4.iv.bundesimmobilien.de> In-Reply-To: References: <20180208093052.7f5d7a98@freyja.zeit4.iv.bundesimmobilien.de> <20180209172259.1ec9b9f4@thor.intern.walstatt.dynvpn.de> <2D57FE3A-744A-4A44-B572-5338AB9E187D@lists.zabbadoz.net> <20180210085248.7b9af104@thor.intern.walstatt.dynvpn.de> Organization: Walstatt MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K0:xeiWk3e6P1ysv2nemTrqbfFuU5y+BVkJ7RPywnMfoMo4e7LOTsi /UmEZ4PP8MKzY/+Mm+I/v3HM7mMEK6f3KSn1F1goLu2qnEeag/h1LBpwBMu2+BbVFPM4KiE 6F+eVajLraBMTGaAAgBGkPBmZ6h8ui0sYmcbgbBlFrg8xIAXPS/lmaujQFWQIxxb8EI2Chp VXXPvYaN1OZfgOL1WNycw== X-UI-Out-Filterresults: notjunk:1;V01:K0:ZiupmWK6Xbo=:lVrXy8dwYRblhadOqr4+Dv +9p94GKozoxUPshvC0HXz2ZMxcN6gqECvRDbwNaYa4mQ8ohJpGf+cj6SgzZfb2tbZw13/9wZQ CBzcGuM633hfX90crPMTTMvkI3DD1cIUXvD+LLpXN+PsimbGgWKveioUDIB4tkhqlRQLRXeto wFfVI6Qv57q7IEB6dn1BnAXxQiFazkwrkS0P73K5mHPNppf3+ZbfM402MuSdUv2nr7moHU+BJ hUaUAixRmlXnPncg9VC2BsX+lYNwdiAd19XLPlnL8y09eGJDRb/TQKzqAqpCymcNuhEA0/qSI HUdpIz/n6c5uDUG3yMBMSirF5rzxUIG36NH3hRq7O3x5YLVTu/pLk8VrC/UEzE7odyxbBaawn UFN5V0w6TTxPOnWUyQ0f2xhVz0n/BcwjFsZ7AQTRRjv9RzvMVEGk0C8SmK5uOkjTflimIswfN WzJlG+GV/HXNkTOwBssY9R/8mgSeRMfBGTAh21wnldn7dQhiB3dW2DLI2GvXoxvrl1GibsoWd M0tAUXbNeDHty+iMaOAKIabBE6KxaFodj+j0Mo8pdAlPCt56ySciK21jGdD5f3drW1ymAwrBr 40zaX5T4dAB2uLZ2kTRtOmvJSqF7w2uW26eOMTQkygCSsM10KIKmNzuI9YcRGeMnU6NDBvrF7 LI64nWEehIEqDJ51sPGmLqxdD3cZdMp7OzzGws4sKW+WfcmXAgAYnb3diUr/buA+Hi8zW/372 D2+FIr79ar1M2rqnIgvvwtQLFtOGoHmjk2KoWTQFcv3EFKA9EcNBmkuUJFFqnn1GS+pb2S6FU /IyX9WUXO1VC47yCKxjKiGG3+uHjLrHMyfuaJO+TxssGSDDlpU= X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2018 08:29:11 -0000 On Sat, 10 Feb 2018 11:52:18 +0100 Olivier Cochard-Labb=C3=A9 wrote: > On Sat, Feb 10, 2018 at 8:52 AM, O. Hartmann wro= te: >=20 > > > > The moment any of the bridges gets an additional member epair interface > > (so the bridge > > has at least three members including the on reaching into the virtual > > router jail) the > > vbridge seems to operate unpredictable (to me). Pinging jails memeber of > > that vbridge > > are unreachable. > > > > =20 > =E2=80=8BFirst idea: > Did you try with a more simple setup, like with just 3 jails members of o= ne > bridge ? > =3D> I've tried it on a -head, and all 4 members (3 jails and the host) = =20 > reach to communicate. Yes, I did. I used to setup one bridge (bridge0) and three jails. Each jail owns its epairXXa device with IP assigned. epairXXb of each jail is member of the bridge. Bridge is up, epairXXb is up. >=20 > Second idea: > Can you check that all epairs have different MAC address each ?=E2=80=8B > I hit this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D176671 Very good shot! Thanks. No, they do not have all of them unique MAC adrress= es and in some occassions members of the very same bridge have the very same M= AC addresses, mostly the a-part of the epair: =46rom console: [...] ng_ether_ifnet_arrival_event: can't re-name node epair10250a =3D=3D>> epair20128a: Ethernet address: 02:ce:d0:00:07:0a epair20128b: Ethernet address: 02:ce:d0:00:13:0b epair20128a: link state changed to UP epair20128b: link state changed to UP epair20128b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair20128a =3D=3D>> epair20129a: Ethernet address: 02:ce:d0:00:07:0a epair20129b: Ethernet address: 02:ce:d0:00:14:0b epair20129a: link state changed to UP epair20129b: link state changed to UP epair20129b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair20129a epair20XXX are member of bridge2 and dedicated epairs of jails. The following is the desastrous picture of bridge1: [...] =3D=3D>> epair235a: Ethernet address: 02:ce:d0:00:07:0a epair235b: Ethernet address: 02:ce:d0:00:0d:0b epair235a: link state changed to UP epair235b: link state changed to UP epair235b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair235a =3D=3D>> epair237a: Ethernet address: 02:ce:d0:00:07:0a epair237b: Ethernet address: 02:ce:d0:00:0e:0b epair237a: link state changed to UP epair237b: link state changed to UP epair237b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair237a =3D=3D>> epair251a: Ethernet address: 02:ce:d0:00:07:0a epair251b: Ethernet address: 02:ce:d0:00:0f:0b epair251a: link state changed to UP epair251b: link state changed to UP epair251b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair251a =3D=3D>> epair238a: Ethernet address: 02:ce:d0:00:07:0a epair238b: Ethernet address: 02:ce:d0:00:10:0b epair238a: link state changed to UP epair238b: link state changed to UP epair238b: promiscuous mode enabled ng_ether_ifnet_arrival_event: can't re-name node epair238a [...] This is on CURRENT (FreeBSD 12.0-CURRENT #36 r329150: Mon Feb 12 06:30:47 C= ET 2018 amd64). I did check on Friday in the bureau and didn't catch it since I was checkin= g on each jail, but the console log accessed via dmesg revealed the problem very easily. I did a check on the 11.1-RELENG-p6 box and I got the same picture there, different, but very similar setup.=20 So I didn't see it in the masses of epairs our setup requires :-( I'll go now for setting MAC addresses manually and check functionality agai= n. >=20 > Regards, >=20 > Olivier Kind regards, Oliver From owner-freebsd-jail@freebsd.org Mon Feb 12 08:43:01 2018 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20216F03275; Mon, 12 Feb 2018 08:43:01 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.17.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7AC4D872E3; Mon, 12 Feb 2018 08:43:00 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from freyja.zeit4.iv.bundesimmobilien.de ([87.138.105.249]) by mail.gmx.com (mrgmx102 [212.227.17.168]) with ESMTPSA (Nemesis) id 0MV30j-1fIwHx0WGT-00YRAT; Mon, 12 Feb 2018 09:37:48 +0100 Date: Mon, 12 Feb 2018 09:37:47 +0100 From: "O. Hartmann" To: Olivier =?ISO-8859-1?Q?Cochard-Labb=E9?= Cc: "O. Hartmann" , freebsd-jail@freebsd.org, freebsd-current Subject: Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing Message-ID: <20180212093747.20024e5f@freyja.zeit4.iv.bundesimmobilien.de> In-Reply-To: References: <20180208093052.7f5d7a98@freyja.zeit4.iv.bundesimmobilien.de> <20180209172259.1ec9b9f4@thor.intern.walstatt.dynvpn.de> <2D57FE3A-744A-4A44-B572-5338AB9E187D@lists.zabbadoz.net> <20180210085248.7b9af104@thor.intern.walstatt.dynvpn.de> Organization: Walstatt MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K0:J4ZmmFcj0ZX5QXQJkpi7TzdFxvRbabQZ1FysksWkVhUYmlRgp6O zrx4I368uHPPzsha0adbaxqMvtzgD9uulwqZ91sbypYSTabJxkW9nuIeXjsxuz1d7/3Tkad Jei4HvJ2lN+WfeNKAjyf0pLissyNncVkUsEmhcMMQ61/mYM8la8oQh6OR2WA6u3kMZy/c5z 1ERXfSnxqRbNRzQY+Co4Q== X-UI-Out-Filterresults: notjunk:1;V01:K0:LtV6hFVb1Ek=:KGhazDgW8ZHrCpQtsvw7Gi V46jdJSTKS8FkGfeEPmfuyovX6koCCth7XagGf/a4AR8FvARvISmxIGDOSSsPRnOMj5SkwJhi cbs316mVFT9YIy4dx99ApcG/RN2L79aYetTPO4fJXKWXRytdd0S3p6eoUVIGfZjI/5tscAz+P b3nfGJcI3KFHK9pQMXEpgwTbv5U4ReWAMYQSUay53aY8d+/h5QaPzljcdcd+5dJ33EJWsQtF2 oODJQeTO1WvzzOUXm1X10tkeMsISUEM/sqzC8rvPDVQXSb3ZXZwURSOD5dP3eactUaC4V50bF wVhDcII5DLfpiAIWRDu6uGBWkiWQqmvLNaPoIv1xpFYv1P4cQNApZl3PyBgEloOXr1/yxbZrf 1uGjEZHvhPnWAuyC1jZcpa1xv6lzFzRjH4ma+wQkUypKTHexEJSLOBlEFv5LOA/Ll3uzRDLN3 lXqAvYKcmpSTXWv+UHK3WGLD0uvq8fW+vzi97Fn76chv1ZdgMDiUGdZM1x0zI4BYip+///+3l 9PR3Xl660QSMt11af0zFJK57vGIxHbO6JK5VyMSmIFr55KplNVVJ1xjjf9hE5CnTEIqNCjEgw 8z1DTvZuCFh4T1LyiL3Lk9oAIIKg9MkdLeE72xNoxhoz7hNNfo2G+8gGihCPsxrEQOcj/Ct5s w4w+8Pa6a9Gg3HwypfbEXumfVLgXOArxtwMSYXe3tDS0f9IeZK8aXEaMyIW8afxzm9l+FqYFg OJGfMRrbuQsXejUnMJlcfUvP4TJ8EJKwuKXniCbqAT9czoun9YWzC5dFIjHTLy/TAodX5H1bL mgfMMaiYnKDcM3424W+IeYDZbnDCTIpINcaZRiOhfq2u82QYgg= X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2018 08:43:01 -0000 On Sat, 10 Feb 2018 11:52:18 +0100 Olivier Cochard-Labb=C3=A9 wrote: > On Sat, Feb 10, 2018 at 8:52 AM, O. Hartmann wro= te: >=20 > > > > The moment any of the bridges gets an additional member epair interface > > (so the bridge > > has at least three members including the on reaching into the virtual > > router jail) the > > vbridge seems to operate unpredictable (to me). Pinging jails memeber of > > that vbridge > > are unreachable. > > > > =20 > =E2=80=8BFirst idea: > Did you try with a more simple setup, like with just 3 jails members of o= ne > bridge ? > =3D> I've tried it on a -head, and all 4 members (3 jails and the host) = =20 > reach to communicate. >=20 > Second idea: > Can you check that all epairs have different MAC address each ?=E2=80=8B > I hit this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D176671 Wow, that PR is from 2013(!) and it is still not solved? >=20 > Regards, >=20 > Olivier > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-jail@freebsd.org Mon Feb 12 11:06:54 2018 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E0E7F0F06B; Mon, 12 Feb 2018 11:06:54 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6AEE46E1D4; Mon, 12 Feb 2018 11:06:53 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from freyja.zeit4.iv.bundesimmobilien.de ([87.138.105.249]) by mail.gmx.com (mrgmx001 [212.227.17.190]) with ESMTPSA (Nemesis) id 0LgptO-1eQvZ52KQg-00oG2L; Mon, 12 Feb 2018 12:06:45 +0100 Date: Mon, 12 Feb 2018 12:06:39 +0100 From: "O. Hartmann" To: freebsd-jail@freebsd.org, freebsd-current Cc: "O. Hartmann" , Olivier =?ISO-8859-1?Q?Cochard?= =?ISO-8859-1?Q?-Labb=E9?= Subject: Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing Message-ID: <20180212120639.7931f609@freyja.zeit4.iv.bundesimmobilien.de> In-Reply-To: <20180212093747.20024e5f@freyja.zeit4.iv.bundesimmobilien.de> References: <20180208093052.7f5d7a98@freyja.zeit4.iv.bundesimmobilien.de> <20180209172259.1ec9b9f4@thor.intern.walstatt.dynvpn.de> <2D57FE3A-744A-4A44-B572-5338AB9E187D@lists.zabbadoz.net> <20180210085248.7b9af104@thor.intern.walstatt.dynvpn.de> <20180212093747.20024e5f@freyja.zeit4.iv.bundesimmobilien.de> Organization: Walstatt MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Provags-ID: V03:K0:22WH3ZncUiXy9S5kWV1v51sv7V+50pBCZke/JciCicLz9S9gyAO uLexVLwD0Gq7+GOX0jXs/Z1y/FHAkrbtlkx5Kc0PsL0MgCiXkzgFyNhNbP1/aX2vkKZSxh2 yrEUbk3xb6/bkJeHSoTrJ/iFftAjKZOHsIY0xD2fiAj9qzEQnKji2Mjt35rdcz5ZL51tiWO QCfeADEBExffj3NYYgrSg== X-UI-Out-Filterresults: notjunk:1;V01:K0:Kwjy5f30NRA=:8FZUJATBQYjiq0/CblSDU8 0MsgUT+Yh5XaPdTjaYsyAOvIsBO05w2+eMpuMgdKJZLACZ1nWzY7Fzy+W0TyffJehOKkgKhCh Ebd2witOnGXRWhzeSn3YIN7uPgZcBuz42TmefxuwVkT445gX+fICAiNjSCnRBOJxNKnOZiavP kBwRZ7BhZ0xl95Em4HBNQ8NWoU1jd0L0CxHmif/VcDUnDLMHohfvzx7lmBIbQz1b91wQusmPI XN8BlwJL9dlKS6lZfmrGTLvfYaQb6FhUsfhtbAvsbnFEBekMgGhLyD18PjtsbOPUWussgjXHT lDs6SgfE0mv9+MhGN3KwIaaHykpZZWgWyNJGij/71bJzY+TFRA3cEnxHtJl4qbnTRogFpmkzx cwxsFdnnG+cHbAhdoadb41uKYCsjSugGijj7TOkdlWd8BqcSF5ZVJ8bO4tEcNsPvrGcg+fpnO GE9zdO8iCnr5yKoZMlLkG9JXmy16kTASeEHW+hi6MVPFNTeoubR0VhAdqcqmDKNKUu+wgDsEG /Iu15JVtw1jJdB+gYFd7QHX3Cgu6ZU4oOPv9Vk2jdNM+wj+yGtKJz+LGhN78bEvK7jjNmXWoH dXFM2hQ8uXXNCmP8ZT8UnCkMzBGlk77GV9bzVh/ilH0mxrT2X1KMY52LSGAC4frJTv1w1SJ7T SLchy8b7nICtYK0LFe89sqQQlXBdzz3+RvXARrU+QcEAPip8EtHGJJWOkMC+3QxoI/aG+aJuu Y4zdGAEm/Guu5u6bpIQAbsEQ6Qz4rnc2dXc2gwGQkz0Lr39+nVQuNkOu6yaLXfgFeOjglzEgt wk5sqjpRblTyeKgQeO/+I6sd3Q/yN3tsVJymhODflO9+mR0Osg= X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Feb 2018 11:06:54 -0000 On Mon, 12 Feb 2018 09:37:47 +0100 "O. Hartmann" wrote: > On Sat, 10 Feb 2018 11:52:18 +0100 > Olivier Cochard-Labb=C3=A9 wrote: >=20 > > On Sat, Feb 10, 2018 at 8:52 AM, O. Hartmann w= rote: > > =20 > > > > > > The moment any of the bridges gets an additional member epair interfa= ce > > > (so the bridge > > > has at least three members including the on reaching into the virtual > > > router jail) the > > > vbridge seems to operate unpredictable (to me). Pinging jails memeber= of > > > that vbridge > > > are unreachable. > > > > > > =20 > > =E2=80=8BFirst idea: > > Did you try with a more simple setup, like with just 3 jails members of= one > > bridge ? =20 > > =3D> I've tried it on a -head, and all 4 members (3 jails and the host= ) =20 > > reach to communicate. > >=20 > > Second idea: > > Can you check that all epairs have different MAC address each ?=E2=80=8B > > I hit this bug: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D176= 671 =20 >=20 > Wow, that PR is from 2013(!) and it is still not solved? >=20 > >=20 > > Regards, > >=20 > > Olivier > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.o= rg" =20 >=20 > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" After rebooting recent CURRENT, the view with "ifconfig -a ether" looked go= od so far, each epair/bridge has its unique MAC. But then, login on the jails and checking the epair's counterpart owned by = the VIMAGE jail, I found almost EVERY jail has the same MAC, even those jails members of the same bridge: [...] jail 11 on bridge2 epair20129a: flags=3D8843 metric 0 = mtu 1500 options=3D8 ether 02:68:d0:00:07:0a jail 10 on bridge2 epair20128a: flags=3D8843 metric 0 = mtu 1500 options=3D8 ether 02:68:d0:00:07:0a jail 9 on bridge 1 epair10250a: flags=3D8843 metric 0 = mtu 1500 options=3D8 ether 02:68:d0:00:07:0a jail 8 on bridge1 epair10238a: flags=3D8843 metric 0 = mtu 1500 options=3D8 ether 02:68:d0:00:07:0a jail 7 on bridge0 epair238a: flags=3D8843 metric 0 mt= u 1500 options=3D8 ether 02:68:d0:00:07:0a epair251a: flags=3D8843 metric 0 mt= u 1500 options=3D8 ether 02:68:d0:00:07:0a The way I create epairs and put them into a jail's context/domain is straig= ht forward. In jail.conf, I have more generic setup with variables like: # DNS Master ns1 { $if =3D "2"; $ip4_addr =3D "10.10.0.${if}"; $ip4_cidr =3D "24"; $ip4_my_default_route =3D "10.10.0.1"; $vnet_if =3D "epair${if}"; $home_bridge =3D "${if_bridge_core}"; depend=3D "vrouter"; allow.raw_sockets=3D "1"; } and in the common portion of jail.conf definitions, I use this: [...] vnet =3D "new"; vnet.interface =3D "${vnet_if}a"; persist; exec.clean; exec.start=3D ""; exec.start+=3D "/sbin/ifconfig ${vnet_if}a inet ${ip4_addr}/${ip4_cidr} up"; exec.start+=3D "/bin/sh /etc/rc"; exec.start+=3D "/sbin/route add default ${ip4_my_default_route}"; exec.start+=3D "/sbin/sysctl net.link.bridge.pfil_member=3D0"; exec.start+=3D "/sbin/sysctl net.link.bridge.pfil_bridge=3D0"; exec.start+=3D "/sbin/sysctl net.link.bridge.pfil_onlyip=3D0"; exec.stop=3D "/bin/sh /etc/rc.shutdown"; exec.prestart=3D ""; exec.prestart+=3D "ifconfig ${vnet_if} create"; exec.prestart+=3D "ifconfig ${vnet_if}b up"; exec.prestart+=3D "ifconfig ${home_bridge} addm ${vnet_if}b up"; exec.poststop=3D "ifconfig ${home_bridge} deletem ${vnet_if}b"; exec.poststop+=3D "ifconfig ${vnet_if}b destroy"; exec.consolelog=3D "/var/log/jail_${name}_console.log"; The big question here is: when a jail with VIMAGE kernel "swallows" a epair-pseudo device, it leaves the ciontext or visibility of the host. How = can the FreeBSD VIMAGE kernel then know about what former epair's MAC was? Is t= his mechanism maybe the culprit? It is just a thought, so I do not want to be beheaded - I'm not much into system development. Kind regards, O. Hartmann From owner-freebsd-jail@freebsd.org Tue Feb 13 08:04:11 2018 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A93B4F078B6; Tue, 13 Feb 2018 08:04:11 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 15F3B69FA7; Tue, 13 Feb 2018 08:04:10 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from freyja.zeit4.iv.bundesimmobilien.de ([87.138.105.249]) by mail.gmx.com (mrgmx102 [212.227.17.168]) with ESMTPSA (Nemesis) id 0MdEsh-1f2thG0D8V-00IQLk; Tue, 13 Feb 2018 09:04:05 +0100 Date: Tue, 13 Feb 2018 09:03:58 +0100 From: "O. Hartmann" To: Marko Zec Cc: "O. Hartmann" , "Bjoern A. Zeeb" , , freebsd-current Subject: Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing Message-ID: <20180213090358.3729045e@freyja.zeit4.iv.bundesimmobilien.de> In-Reply-To: <20180210095449.3117e6d9@x23> References: <20180208093052.7f5d7a98@freyja.zeit4.iv.bundesimmobilien.de> <20180209172259.1ec9b9f4@thor.intern.walstatt.dynvpn.de> <2D57FE3A-744A-4A44-B572-5338AB9E187D@lists.zabbadoz.net> <20180210085248.7b9af104@thor.intern.walstatt.dynvpn.de> <20180210095449.3117e6d9@x23> Organization: Walstatt MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:IsPhPT5Ge3OatU0uG11pn72RAKohf+PH+WTfKT42c/RQcw+iKkF GtkBa+cTfyzaOcCYn1wFrx81So9noKlwXmlDNTRqRGwFAUoWeqQwq5MS3Dz/B8yiriLpWiH QZ/vPcHhlsUMt1z2MSH9VyrmSLMSLdrJ/v7llmpjUgebAg9pt+pxHxaWpGwSFUNWGTiCOqv Z8EzzS7GV5P7Y5DbnvcrQ== X-UI-Out-Filterresults: notjunk:1;V01:K0:UAuk+mpzsuk=:+3YKZZIw9EsdYfRqlU74ex 1iUZUifTxHQFfsd2okdgFssuWlGcd18rNP9fksc1WmS3j3pr4dEEQE1rPtUG6ealfMZAZMb/g 5uQggT3d32AHX3pGaGMNG6mbiaHm18wppyTGg72lpMsYc5FlpBeZNlwhqWJQ3Ur0sYOrh1waO JM7VMRaW/6fR5ry44lPEypNArABvzORPg/U2RelYStouQjggwHxcGx214kfLUtBCbqzWDNaTf I6PO3cP6ZF72x6voqZLxRCFq2Ba03lCCCHv3lTaS3bMHTb1dxRfiuuJFda6jXIJOyV7I7x0rB oaKXwakCvctAFZ2AA6YdYibnlWuG2Y0lF1+tT25Hfxa0275KFgI/1vB6ONkcCH7h2oNfbZdbJ 0bqAzEHcZ5IBrw5FfzoQLEPaHhH3hI+fvcI0IlVHBWgfag4FNumRhH9Jp1xeTqFA8U61B6LZL KKXLPH3+Cbo/DdQ9pBMqqY0NOAaCDmkn14TMxFAEnY+qLLnpg7AOpY+iv5ASuPOv7PLrtQimK S3yhfjOXyUgjH6cyhxSCne8FE+7FrEAS6DijhWsxAhMEB2GXP+HUXunPitgD9irLMOzO8CtcS A3gx0LKmB+u5JoXshYGivCEoxFG5sUwpcTnssogmn7jytZMN6MoFJANweXhTBPkhVsz9Qrnm4 vPeFdz13QUTJhNKWD4KVqUFjmIjlHYjJ4wxotnBztI3QmYb+R/VLKRXkQ7NFEt754PMU9oHdr 4xpPmbXR7CtXlTZLm7/KL9jK3X00DnFymJPzxce3SnNKuhTq9aEajvDuKlqmAV6QPU+QO7nnK oRhqjeaOtQRZ2H1aTOjFDh9/xFCxnaIpK34DrPO43Aol8ZJ0GE= X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Feb 2018 08:04:12 -0000 On Sat, 10 Feb 2018 09:54:49 +0100 Marko Zec wrote: > On Sat, 10 Feb 2018 08:52:21 +0100 > "O. Hartmann" wrote: > > > Am Fri, 09 Feb 2018 16:43:17 +0000 > > "Bjoern A. Zeeb" schrieb: > > > > > On 9 Feb 2018, at 16:22, O. Hartmann wrote: > > > > > > > Am Thu, 8 Feb 2018 09:31:15 +0100 > > > > "O. Hartmann" schrieb: > > > > > > > > Is this problem to trivial? > > > > > > I read through it yesterday and found myself in the position that I > > > need a whiteboard or paper and pencil or an ASCII art of your > > > situation. But by the time I made it to the question I was > > > basically lost. Could you massively simplify this and maybe > > > produce the ASCII art? > > > > > > /bz > > > _______________________________________________ > > > freebsd-current@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > > To unsubscribe, send any mail to > > > "freebsd-current-unsubscribe@freebsd.org" > > > > All right. > > > > I'm not much of an artist and at this very moment, I haven't much > > experience with neat ASCII art tools. But I'll provide a sketch > > later, but I also will simplify the situation. > > > > Consider three "vswitches", basically based on the creation of > > bridges, bridge0, bridge1, bridge2. Create at least three individual > > vnet-jails attached to each vbridge. Those jails have epair pseudo > > devices. The jail itself owns the "a-part" of the epair and the > > b-part is "member of the bridge". Each jail's epairXXXa has an IP > > assigned of the network the vswitch is part of. I mention a- and > > b-part of the epair here, because I thought it could matter, but I > > think for symmetry reasons it doesn't. > > > > Now consider a further, special jail. This jail is supposed to have > > three epair devices, each one is reaching into one of the vbridges. > > This jail is the router/routing jail. Later, this jail should filter > > via IPFW the traffic between the three vbridges according to rules, > > but this doesn't matter here, beacuase the basics are not working as > > expected. > > > > Now the problems. It doesn't matter on which jail of the three > > vswitches I login, the moment a vbridge has more than two member > > epairs (one is alway member of the routing jail, now consider a > > database jail and a webserver jail), pinging each jail or the routing > > jail fails. It works sometimes for a couple of ICMP packets and then > > stops. > > > > If each vbridge has only one member jail, I have NO PROBLEMS > > traversing accordingly to the static routing rules from one vbridge > > to any other, say from vbridge1 to vbridge0 or vbridge2 and any > > permutation of that. > > > > The moment any of the bridges gets an additional member epair > > interface (so the bridge has at least three members including the on > > reaching into the virtual router jail) the vbridge seems to operate > > unpredictable (to me). Pinging jails memeber of that vbridge are > > unreachable. > > > > Technical information: > > > > The kernel has options IPFIREWALL, VIMAGE. The host's ipfw (kernel) > > declines packets by default. Each jail is configured to have ipfw > > "open". > > > > Thanks for the patience. > > If you could provide a script which sets up the topology you described > in two lengthy posts then others could reproduce this, and your chances > of getting useful feedback would certainly increase. > > We also have a graphical tool (https://github.com/imunes/imunes) which > can set up a topology like you described in a few clicks of a mouse, > albeit using netgraph and ng_eiface instead of epairs, but I assume this > is irellevant as long as you are not aiming for maximum packet > throughputs. If you attempt to use this tool, note that selecting > "stpswitch" will create if_bridge instances, whereas "lanswitch" > creates ng_bridge instances. > > Good luck, > > Marko > > > > > > Kind regards, > > > > O. Hartmann > > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" Hello Marko, thanks for your response. First of all: I looked at "imunes". From the first glimpse it looks great! Something really usefull; I regret not having a port for this tool or the chance to package it via poudriere. The problems I faced seem to be related to a bug Olivier Cochard-Labbe pointed me at: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=176671 Checking the MAC of the epairs created revealed, that either doubles or even more occur on the host side of the epair (in my case, all the b-parts of an epair), or, if there is nothing irregular, then the a-parts (owned by the VIMAGE jail) have same MAC. The more jails I create, the more ambiguous MACs are present. It is the first time I ran into a more complex network topology and thanks for the hint using netgraph. Kind regards, Oliver