From owner-freebsd-current@freebsd.org Fri Feb 9 16:23:16 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6877F10215 for ; Fri, 9 Feb 2018 16:23:15 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6F1E26B7C4 for ; Fri, 9 Feb 2018 16:23:15 +0000 (UTC) (envelope-from ohartmann@walstatt.org) Received: from thor.intern.walstatt.dynvpn.de ([85.181.254.98]) by mail.gmx.com (mrgmx001 [212.227.17.190]) with ESMTPSA (Nemesis) id 0LwJRe-1elfhS0bpG-017zay; Fri, 09 Feb 2018 17:23:08 +0100 Date: Fri, 9 Feb 2018 17:22:32 +0100 From: "O. Hartmann" To: "O. Hartmann" Cc: freebsd-jails@freebsd.org, freebsd-current Subject: Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing Message-ID: <20180209172259.1ec9b9f4@thor.intern.walstatt.dynvpn.de> In-Reply-To: <20180208093052.7f5d7a98@freyja.zeit4.iv.bundesimmobilien.de> References: <20180208093052.7f5d7a98@freyja.zeit4.iv.bundesimmobilien.de> Organization: WALSTATT User-Agent: OutScare 3.1415926 X-Operating-System: ImNotAnOperatingSystem 3.141592527 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; boundary="Sig_/jHX=/o7OQav.vD032FciI5b"; protocol="application/pgp-signature" X-Provags-ID: V03:K0:4G0hdEl1rGhUtwDT5+umWVXmmI2+HtNMylT2HwrjoEU4AF1HWAg 5xlQtQVyJzgZnPT6dOtqB7EKu+fl+M8nhGO+sduKsu0PJQ1d0NIG91IFQ9q9FdrAjjBg5yl 5s2fQi3HbudUWli/kKOj32/cEg7W1SKVfbEe/KSXaNlB4R7nRbxStC0/Uuqs7q7MqHk6K2A t1V68KFw7YWR0nhN+pVUw== X-UI-Out-Filterresults: notjunk:1;V01:K0:RIaHtrpECP4=:8oV7SefvNM0kso+u/ZxlXw 90tsZPUGTqwPvsYLgLSKGqWU6B6hPqPsivgyoj3HtN1kcoJGDoRMYhL9DiaTCc/UXnfKfz3dw 7Kaj5CwrQJKTNtT3BB7+6F545WnIdh8mhBT1P5FJR0CQjU8kc7ZAPO3AC76dXJMl0Vh01XrPW QG0mXkjqBNJf+RLzksnL6J+c+xgO1g+FBED3HwKhamL09Hk3YpwurS+iOErjUHSSvm/nmE0CE RSQyEEOPi5ZC7XPWMsPrkvK4EivIlRQeSTHdLUOvkJtGUIEdZ34ERjr1q4o+pgpFFla6KaJYi 5gz+iQelPOYVnk0gdcFz2s1xWudP/fhsqDxOQs+PQXQOnXCimtP3Je3E5573tuSrVg346lIM0 Suu1+BsvNEBK7JNGjmh0/1aTGPwLSzZ90u+jIq5m6IeNKzxlEfeHSgcQM+txXurj7FqqyRX6s Bml62VVakl5IfKedsHdSPLvXwuIMi+ulfj7N5c4WPGTJReX08JpsUq9GrXGprvhrqIbOb+oPJ PwIVJl1RrRUE5MS0jR8WfsOSs7m924AZ8gm2IQ45TrbyUb6LnpvCkFRRJG/Qwji1a+kAUdqJI lD1Sjkb12ISN8RGEUhjAzsNu+6ZfRcgXgk2z2PcAak1+MwyYbcOgqA16CeQQAJ2QMxQBSca91 xIALTpVgveXVgKhs6o/KoiKyDqJKYfKlVGOffhaEyhSNZjxq6t+yK59vhNT3I1oJ4m9BidmOu iTGScrNmq6gZM+DGu6cpG3S+QKb5s4z+cvy8ISFyIuQnwqS5XHz0o26legmllVkVhCzPc25jE QEdhIDv2LsFOBvgyfgFaIr3yzXqlA== X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Feb 2018 16:23:16 -0000 --Sig_/jHX=/o7OQav.vD032FciI5b Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Am Thu, 8 Feb 2018 09:31:15 +0100 "O. Hartmann" schrieb: > Hello, >=20 > I fight with the following problem without any kind of success and I need= some > help and/or advice. >=20 > We are running several CURRENT and 11.1-RELENG-p6 boxes. CURRENT is at th= e most > recent version as of today. >=20 > VIMAGE is compiled in into all kernels. > IPFW is compiled into all kernels and is the one and only firewall used. > On CURRENT, the host's ipfw is set to "OPEN" (using the rc.-scripts so fa= r). By > convention, I address the host running the kernel by "host". >=20 > Every jail is created/configured with its own "vnet" cloned network stack > (vnet=3Dnew). >=20 > All hosts do have at least three physical NICs. The host itself is suppos= ed to > be member of the "friendly" network via a dedicated NIC. The two remainin= g NICs > are split into fractions belonging to an "hostile" network on which I'd l= ike to > place exposed jails (for now), and to the "friendly" network, on which al= so > jails will be hosted, but via a dedicated NIC. >=20 > Inbetween those two networks, the host will have a third, intermediate, > network, call it the "service" network. >=20 > The following will be true for ALL jails created, including the host itse= lf: >=20 > net.link.bridge.pfil_member=3D0 > net.link.bridge.pfil_bridge=3D0 > net.link.bridge.pfil_onlyip=3D0 >=20 > First, I clone/create three bridge(4) devices, bridge0 (considered to be = the > "glue" between the "service" jails), bridge1 (considered to be the glue b= etween > the jails on the friednly network side) and bridge2, which is the glue be= tween > the jails on the hostile side. bridge1 has eth1 as a member, which provid= es the > physical access to the friendly network, eth2 is member of bridge2, which > provides access to the hostile network. >=20 > By convention, when creating epair(4), the a-portion belongs to the jail = itself > and is assigned with an IPv6 address. The b-portion of the epair(4) is me= mber > of its bridge according to its realm (friendly, service or hostile networ= k).=20 >=20 > Additionally, there is a special jail, the router, which has three epair(= 4) > devices, the b-portion of the epair is member of the appropriate bridge(4= ) and > this router jail has static routes assigned, pointing to the appropriate > epairXXXa that is suppoesd to be the link into the correct bridge/network= . IPFW > is set to open on this jail (for now). On this special > jail it is set: net.inet.ip.forwarding=3D1. >=20 > I hope, the topology is clear so far. All epairs or epair endpoints as we= ll as > the bridges are UP! Double checked this. >=20 > Jails on bridge0 (service net) have IPs in the range 10.10.0.0/24, the > b-portion of the routing jail's epair is member of bridge0, as described = above, > and the a-portion of the epair has IP 10.10.0.1. Default route on each je= ail > on bridge0 is set to 10.10.0.1 accordingly. >=20 > Consider a similar setup on the other jails on the friendly and hostile > network, except the fact that their bridges do have a physical NIC to whi= ch > they may have access to a real network. >=20 > The setup might not be ideal and/or applicable for the purpose of separar= tion > of networks virtually, but that shouldn't be the subject here. More impor= tant > is that I assume that I haven't understood some essentials, because the s= etup > doens't work as expected. Furthermore, it behaves on FreeBSD 11.1-RELENG-= p6 > sometimes completely unpredictable - but in that special case, I think I = ran > IPFW on the host as "WORKSTATION" and dynamic rules may play an important= role > here. But focussing on the CURRENT box, the host's IPFW is set to OPEN. >=20 > With jexec -l hostA I gain access to host A on the "service" bridge0 and I > want to ping its neighbour, hostB, on the same bridge and in the same net= . It > doesn't work! From the routing jail, I CAN NOT ping any host on bridge0. = The > routing jail has these network settings: >=20 > [... routing jail ...] > lo0: flags=3D8049 metric 0 mtu 16384 > options=3D600003 > inet 127.0.0.1 netmask 0xff000000=20 > groups: lo > [epair to bridge0 - service net]=20 > epair4000a: flags=3D8843 metric 0= mtu 1500 > options=3D8 > ether 02:57:d0:00:07:0a > inet 10.10.0.1 netmask 0xffffff00 broadcast 10.10.0.255=20 > media: Ethernet 10Gbase-T (10Gbase-T ) > status: active > groups: epair > [epair to bridge1, friendly net]=20 > epair4001a: flags=3D8843 metric 0= mtu 1500 > options=3D8 > ether 02:57:d0:00:09:0a > inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255=20 > media: Ethernet 10Gbase-T (10Gbase-T ) > status: active > groups: epair > [epair to bridge2, hostile net]=20 > epair4002a: flags=3D8843 metric 0= mtu 1500 > options=3D8 > ether 02:57:d0:00:0b:0a > inet 10.10.10.1 netmask 0xfffffc00 broadcast 10.10.10.255=20 > media: Ethernet 10Gbase-T (10Gbase-T ) > status: active > groups: epair=20 >=20 > routing: > netstat -Warn > Routing tables >=20 > Internet: > Destination Gateway Flags Use Mtu Netif E= xpire > 10.10.0.0/24 link#2 U 11 1500 epair4000a > 10.10.0.1 link#2 UHS 4 16384 lo0 > 10.10.10.0/24 link#4 U 210 1500 epair4002a > 10.10.10.1 link#4 UHS 44 16384 lo0 > 127.0.0.1 link#1 UH 0 16384 lo0 > 192.168.11.0/24 link#3 U 9 1500 epair4001a > 192.168.11.1 link#3 UHS 0 16384 lo0 >=20 > Consider a jail hostCC on bridge2 in the hostile network, IP 10.10.10.128= .=20 > I can ping that jail, although it has conceptionally the very same setup = as the > unreachable jails on bridge0! >=20 > It is weird. On bridge0, no jail can be pinged, it looks like the etherne= t is > somehwo down on that bridge. I would expect to ping each host member of t= he > very same bridge! On 11.1-RELENG-p6, there are other weird issues, I was = able > to ping those jails, even ssh to them, but that vanished after several > restarts of the jails system (each bridge, epair is created by jail.conf = and > destroyed after the jails has been deactivated and doing so a considerable > amount brings down the FreeBSD 11.1-RELENG-p6 host verys successfully - it > crashes!). >=20 > So, since VIMAGE is now default in CURRENT's GENERIC, I consider its > functionality at least "predictable", but I fail somehow here. >=20 > Does someone have a deeper insight or realise the mistake I'm celebrating= here? >=20 > Thanks in adavnce, >=20 > Oliver=20 Is this problem to trivial? oh --Sig_/jHX=/o7OQav.vD032FciI5b Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iLUEARMKAB0WIQQZVZMzAtwC2T/86TrS528fyFhYlAUCWn3K4wAKCRDS528fyFhY lPY3Af48aP52clF4lWkMfRUKUX46rG/ZhmxdQ3wllMhOEUi8ASG2VHtrt/wSuIRq Xes2dCF8IIVUdsOPRjmde8Z/bCD1Af4mQT+H5pdcnSYeSIWFjWBs1yw9GEwyxurx sqSzggNjdSLWIJTSNyZI6WCdEBO53jfdC0OokRM9i027/T6VZ8s2 =+Vwl -----END PGP SIGNATURE----- --Sig_/jHX=/o7OQav.vD032FciI5b--