Date: Fri, 9 Feb 2018 17:22:32 +0100 From: "O. Hartmann" <ohartmann@walstatt.org> To: "O. Hartmann" <ohartmann@walstatt.org> Cc: freebsd-jails@freebsd.org, freebsd-current <freebsd-current@freebsd.org> Subject: Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing Message-ID: <20180209172259.1ec9b9f4@thor.intern.walstatt.dynvpn.de> In-Reply-To: <20180208093052.7f5d7a98@freyja.zeit4.iv.bundesimmobilien.de> References: <20180208093052.7f5d7a98@freyja.zeit4.iv.bundesimmobilien.de>
next in thread | previous in thread | raw e-mail | index | archive | help
--Sig_/jHX=/o7OQav.vD032FciI5b Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Am Thu, 8 Feb 2018 09:31:15 +0100 "O. Hartmann" <ohartmann@walstatt.org> schrieb: > Hello, >=20 > I fight with the following problem without any kind of success and I need= some > help and/or advice. >=20 > We are running several CURRENT and 11.1-RELENG-p6 boxes. CURRENT is at th= e most > recent version as of today. >=20 > VIMAGE is compiled in into all kernels. > IPFW is compiled into all kernels and is the one and only firewall used. > On CURRENT, the host's ipfw is set to "OPEN" (using the rc.-scripts so fa= r). By > convention, I address the host running the kernel by "host". >=20 > Every jail is created/configured with its own "vnet" cloned network stack > (vnet=3Dnew). >=20 > All hosts do have at least three physical NICs. The host itself is suppos= ed to > be member of the "friendly" network via a dedicated NIC. The two remainin= g NICs > are split into fractions belonging to an "hostile" network on which I'd l= ike to > place exposed jails (for now), and to the "friendly" network, on which al= so > jails will be hosted, but via a dedicated NIC. >=20 > Inbetween those two networks, the host will have a third, intermediate, > network, call it the "service" network. >=20 > The following will be true for ALL jails created, including the host itse= lf: >=20 > net.link.bridge.pfil_member=3D0 > net.link.bridge.pfil_bridge=3D0 > net.link.bridge.pfil_onlyip=3D0 >=20 > First, I clone/create three bridge(4) devices, bridge0 (considered to be = the > "glue" between the "service" jails), bridge1 (considered to be the glue b= etween > the jails on the friednly network side) and bridge2, which is the glue be= tween > the jails on the hostile side. bridge1 has eth1 as a member, which provid= es the > physical access to the friendly network, eth2 is member of bridge2, which > provides access to the hostile network. >=20 > By convention, when creating epair(4), the a-portion belongs to the jail = itself > and is assigned with an IPv6 address. The b-portion of the epair(4) is me= mber > of its bridge according to its realm (friendly, service or hostile networ= k).=20 >=20 > Additionally, there is a special jail, the router, which has three epair(= 4) > devices, the b-portion of the epair is member of the appropriate bridge(4= ) and > this router jail has static routes assigned, pointing to the appropriate > epairXXXa that is suppoesd to be the link into the correct bridge/network= . IPFW > is set to open on this jail (for now). On this special > jail it is set: net.inet.ip.forwarding=3D1. >=20 > I hope, the topology is clear so far. All epairs or epair endpoints as we= ll as > the bridges are UP! Double checked this. >=20 > Jails on bridge0 (service net) have IPs in the range 10.10.0.0/24, the > b-portion of the routing jail's epair is member of bridge0, as described = above, > and the a-portion of the epair has IP 10.10.0.1. Default route on each je= ail > on bridge0 is set to 10.10.0.1 accordingly. >=20 > Consider a similar setup on the other jails on the friendly and hostile > network, except the fact that their bridges do have a physical NIC to whi= ch > they may have access to a real network. >=20 > The setup might not be ideal and/or applicable for the purpose of separar= tion > of networks virtually, but that shouldn't be the subject here. More impor= tant > is that I assume that I haven't understood some essentials, because the s= etup > doens't work as expected. Furthermore, it behaves on FreeBSD 11.1-RELENG-= p6 > sometimes completely unpredictable - but in that special case, I think I = ran > IPFW on the host as "WORKSTATION" and dynamic rules may play an important= role > here. But focussing on the CURRENT box, the host's IPFW is set to OPEN. >=20 > With jexec -l hostA I gain access to host A on the "service" bridge0 and I > want to ping its neighbour, hostB, on the same bridge and in the same net= . It > doesn't work! From the routing jail, I CAN NOT ping any host on bridge0. = The > routing jail has these network settings: >=20 > [... routing jail ...] > lo0: flags=3D8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 > options=3D600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6> > inet 127.0.0.1 netmask 0xff000000=20 > groups: lo > [epair to bridge0 - service net]=20 > epair4000a: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0= mtu 1500 > options=3D8<VLAN_MTU> > ether 02:57:d0:00:07:0a > inet 10.10.0.1 netmask 0xffffff00 broadcast 10.10.0.255=20 > media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) > status: active > groups: epair > [epair to bridge1, friendly net]=20 > epair4001a: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0= mtu 1500 > options=3D8<VLAN_MTU> > ether 02:57:d0:00:09:0a > inet 192.168.11.1 netmask 0xffffff00 broadcast 192.168.11.255=20 > media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) > status: active > groups: epair > [epair to bridge2, hostile net]=20 > epair4002a: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0= mtu 1500 > options=3D8<VLAN_MTU> > ether 02:57:d0:00:0b:0a > inet 10.10.10.1 netmask 0xfffffc00 broadcast 10.10.10.255=20 > media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) > status: active > groups: epair=20 >=20 > routing: > netstat -Warn > Routing tables >=20 > Internet: > Destination Gateway Flags Use Mtu Netif E= xpire > 10.10.0.0/24 link#2 U 11 1500 epair4000a > 10.10.0.1 link#2 UHS 4 16384 lo0 > 10.10.10.0/24 link#4 U 210 1500 epair4002a > 10.10.10.1 link#4 UHS 44 16384 lo0 > 127.0.0.1 link#1 UH 0 16384 lo0 > 192.168.11.0/24 link#3 U 9 1500 epair4001a > 192.168.11.1 link#3 UHS 0 16384 lo0 >=20 > Consider a jail hostCC on bridge2 in the hostile network, IP 10.10.10.128= .=20 > I can ping that jail, although it has conceptionally the very same setup = as the > unreachable jails on bridge0! >=20 > It is weird. On bridge0, no jail can be pinged, it looks like the etherne= t is > somehwo down on that bridge. I would expect to ping each host member of t= he > very same bridge! On 11.1-RELENG-p6, there are other weird issues, I was = able > to ping those jails, even ssh to them, but that vanished after several > restarts of the jails system (each bridge, epair is created by jail.conf = and > destroyed after the jails has been deactivated and doing so a considerable > amount brings down the FreeBSD 11.1-RELENG-p6 host verys successfully - it > crashes!). >=20 > So, since VIMAGE is now default in CURRENT's GENERIC, I consider its > functionality at least "predictable", but I fail somehow here. >=20 > Does someone have a deeper insight or realise the mistake I'm celebrating= here? >=20 > Thanks in adavnce, >=20 > Oliver=20 Is this problem to trivial? oh --Sig_/jHX=/o7OQav.vD032FciI5b Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- iLUEARMKAB0WIQQZVZMzAtwC2T/86TrS528fyFhYlAUCWn3K4wAKCRDS528fyFhY lPY3Af48aP52clF4lWkMfRUKUX46rG/ZhmxdQ3wllMhOEUi8ASG2VHtrt/wSuIRq Xes2dCF8IIVUdsOPRjmde8Z/bCD1Af4mQT+H5pdcnSYeSIWFjWBs1yw9GEwyxurx sqSzggNjdSLWIJTSNyZI6WCdEBO53jfdC0OokRM9i027/T6VZ8s2 =+Vwl -----END PGP SIGNATURE----- --Sig_/jHX=/o7OQav.vD032FciI5b--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20180209172259.1ec9b9f4>