Date: Fri, 20 Dec 2019 14:03:07 -0800 From: Julian Elischer <julian@freebsd.org> To: Nick Wolff <darkfiberiru@gmail.com>, Marko Zec <zec@fer.hr> Cc: "Patrick M. Hausen" <hausen@punkt.de>, Kristof Provost <kp@eurobsdcon.org>, "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Continuing problems in a bridged VNET setup Message-ID: <e9da965d-4a6f-c750-28e7-af1de318f13f@freebsd.org> In-Reply-To: <CACxAneBLDCpOCc4YVE5jSCmfd-9kvf81hMp7EoXci1ZGEVPuNw@mail.gmail.com> References: <BD4018F8-0BB7-4EA9-A726-F6383E9AC892@punkt.de> <20191220122256.76942c07@x23> <CACxAneBLDCpOCc4YVE5jSCmfd-9kvf81hMp7EoXci1ZGEVPuNw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 12/20/19 10:09 AM, Nick Wolff wrote: > Marko, > > Are you aware of any write ups for using ng_eiface and ng_bridge instead of > if_bridge? look in /usr/share/examples/netgraph here are a couple of examples of exactly what you ask for. > Thanks, > > Nick Wolff > > On Fri, Dec 20, 2019 at 6:22 AM Marko Zec <zec@fer.hr> wrote: > >> Perhaps you could ditch if_bridge(4) and epair(4), and try ng_eiface(4) >> with ng_bridge(4) instead? Works rock-solid 24/7 here on 11.2 / 11.3. >> >> Marko >> >> On Fri, 20 Dec 2019 11:19:24 +0100 >> "Patrick M. Hausen" <hausen@punkt.de> wrote: >> >>> Hi all, >>> >>> we still experience occasional network outages in production, >>> yet have not been able to find the root cause. >>> >>> We run around 50 servers with VNET jails. some of them with >>> a handful, the busiest ones with 50 or more jails each. >>> >>> Every now and then the jails are not reachable over the net, >>> anymore. The server itself is up and running, all jails are >>> up and running, one can ssh to the server but none of the >>> jails can communicate over the network. >>> >>> There seems to be no pattern to the time of occurrance except >>> that more jails on one system make it "more likely". >>> Also having more than one bridge, e.g. for private networks >>> between jails seems to increase the probability. >>> When a server shows the problem it tends to get into the state >>> rather frequently, a couple of hours inbetween. Then again >>> most servers run for weeks without exhibiting the problem. >>> That's what makes it so hard to reproduce. The last couple of >>> days one system was failing regularly until we reduced the number >>> of jails from around 80 to around 50. Now it seems stable again. >>> >>> I have a test system with lots of jails that I work with gatling >>> that did not show a single failure so far :-( >>> >>> >>> Setup: >>> >>> All jails are iocage jails with VNET interfaces. They are >>> connected to at least one bridge that starts with the >>> physical external interface as a member and gets jails' >>> epair interfaces added as they start up. All jails are managed >>> by iocage. >>> >>> ifconfig_igb0="-rxcsum -rxcsum6 -txcsum -txcsum6 -vlanhwtag >>> -vlanhwtso up" cloned_interfaces="bridge0" >>> ifconfig_bridge0_name="inet0" >>> ifconfig_inet0="addm igb0 up" >>> ifconfig_inet0_ipv6="inet6 <host-address>/64 auto_linklocal" >>> >>> $ iocage get interfaces vpro0087 >>> vnet0:inet0 >>> >>> $ ifconfig inet0 >>> inet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 >>> mtu 1500 ether 90:1b:0e:63:ef:51 >>> inet6 fe80::921b:eff:fe63:ef51%inet0 prefixlen 64 scopeid 0x4 >>> inet6 <host-address> prefixlen 64 >>> nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> >>> groups: bridge >>> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 >>> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 >>> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >>> member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> >>> ifmaxaddr 0 port 7 priority 128 path cost 2000 >>> member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> >>> ifmaxaddr 0 port 6 priority 128 path cost 2000 >>> member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> >>> ifmaxaddr 0 port 1 priority 128 path cost 2000000 >>> >>> >>> What we tried: >>> >>> At first we suspected the bridge to become "wedged" somehow. This was >>> corroborated by talking to various people at devsummits and EuroBSDCon >>> with Kristof Provost specifically suggesting that if_bridge was >>> still under giant lock and there might be a problem here that the >>> lock is not released under some race condition and then the entire >>> bridge subsystem would be stalled. That sounds plausible given the >>> random occurrance. >>> >>> But I think we can rule out that one, because: >>> >>> - ifconfig up/down does not help >>> - the host is still communicating fine over the same bridge interface >>> - tearing down the bridge, kldunload (!) of if_bridge.ko followed by >>> a new kldload and reconstructing the members with `ifconfig addm` >>> does not help, either >>> - only a host reboot restores function >>> >>> Finally I created a not iocage managed jail on the problem host. >>> Please ignore the `iocage` in the path, I used it to populate the >>> root directory. But it is not started by iocage at boot time and >>> the manual config is this: >>> >>> testjail { >>> host.hostname = "testjail"; # hostname >>> path = "/iocage/jails/testjail/root"; # root directory >>> exec.clean; >>> exec.system_user = "root"; >>> exec.jail_user = "root"; >>> vnet; >>> vnet.interface = "epair999b"; >>> exec.prestart += "ifconfig epair999 create; ifconfig >>> epair999a inet6 2A00:B580:8000:8000::1/64 auto_linklocal"; >>> exec.poststop += "sleep 2; ifconfig epair999a destroy; sleep 2"; >>> # Standard stuff >>> exec.start += "/bin/sh /etc/rc"; >>> exec.stop = "/bin/sh /etc/rc.shutdown"; >>> exec.consolelog = "/var/log/jail_testjail_console.log"; >>> mount.devfs; #mount devfs >>> allow.raw_sockets; #allow ping-pong >>> devfs_ruleset="4"; #devfs ruleset for this jail >>> } >>> >>> $ cat /iocage/jails/testjail/root/etc/rc.conf >>> hostname="testjail" >>> >>> ifconfig_epair999b_ipv6="inet6 2A00:B580:8000:8000::2/64 >>> auto_linklocal" >>> >>> When I do `service jail onestart testjail` I can then ping6 the jail >>> from the host and the host from the jail. As you can see the >>> if_bridge is not involved in this traffic. >>> >>> When the host is in the wedged state and I start this testjail the >>> same way, no communication across the epair interface is possible. >>> >>> To me this seems to indicate that not the bridge but all epair >>> interfaces stop working at the very same time. >>> >>> >>> OS is RELENG_11_3, hardware and specifically network adapters vary, >>> we have igb, ix, ixl, bnxt ... >>> >>> >>> Does anyone have a suggestion what diagnostic measures could help to >>> pinpoint the culprit? The random occurrance and the fact that the >>> problem seems to prefer the production environment only makes this a >>> real pain ... >>> >>> >>> Thanks and kind regards, >>> Patrick >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?e9da965d-4a6f-c750-28e7-af1de318f13f>