From owner-freebsd-net@freebsd.org Fri Dec 20 22:03:24 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BBBE91E4FCD for ; Fri, 20 Dec 2019 22:03:24 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47fjRw4Ysvz4VB0 for ; Fri, 20 Dec 2019 22:03:24 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (c-73-225-95-104.hsd1.wa.comcast.net [73.225.95.104]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id xBKM3C41093518 (version=TLSv1.2 cipher=AES128-SHA bits=128 verify=NO); Fri, 20 Dec 2019 14:03:13 -0800 (PST) (envelope-from julian@freebsd.org) Subject: Re: Continuing problems in a bridged VNET setup To: Nick Wolff , Marko Zec Cc: "Patrick M. Hausen" , Kristof Provost , "freebsd-net@freebsd.org" References: <20191220122256.76942c07@x23> From: Julian Elischer Message-ID: Date: Fri, 20 Dec 2019 14:03:07 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 47fjRw4Ysvz4VB0 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.97 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.99)[-0.992,0]; NEURAL_HAM_LONG(-0.98)[-0.977,0]; ASN(0.00)[asn:36236, ipnet:204.109.60.0/22, country:US] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Dec 2019 22:03:24 -0000 On 12/20/19 10:09 AM, Nick Wolff wrote: > Marko, > > Are you aware of any write ups for using ng_eiface and ng_bridge instead of > if_bridge? look in /usr/share/examples/netgraph here are a couple of examples of exactly what you ask for. > Thanks, > > Nick Wolff > > On Fri, Dec 20, 2019 at 6:22 AM Marko Zec wrote: > >> Perhaps you could ditch if_bridge(4) and epair(4), and try ng_eiface(4) >> with ng_bridge(4) instead? Works rock-solid 24/7 here on 11.2 / 11.3. >> >> Marko >> >> On Fri, 20 Dec 2019 11:19:24 +0100 >> "Patrick M. Hausen" wrote: >> >>> Hi all, >>> >>> we still experience occasional network outages in production, >>> yet have not been able to find the root cause. >>> >>> We run around 50 servers with VNET jails. some of them with >>> a handful, the busiest ones with 50 or more jails each. >>> >>> Every now and then the jails are not reachable over the net, >>> anymore. The server itself is up and running, all jails are >>> up and running, one can ssh to the server but none of the >>> jails can communicate over the network. >>> >>> There seems to be no pattern to the time of occurrance except >>> that more jails on one system make it "more likely". >>> Also having more than one bridge, e.g. for private networks >>> between jails seems to increase the probability. >>> When a server shows the problem it tends to get into the state >>> rather frequently, a couple of hours inbetween. Then again >>> most servers run for weeks without exhibiting the problem. >>> That's what makes it so hard to reproduce. The last couple of >>> days one system was failing regularly until we reduced the number >>> of jails from around 80 to around 50. Now it seems stable again. >>> >>> I have a test system with lots of jails that I work with gatling >>> that did not show a single failure so far :-( >>> >>> >>> Setup: >>> >>> All jails are iocage jails with VNET interfaces. They are >>> connected to at least one bridge that starts with the >>> physical external interface as a member and gets jails' >>> epair interfaces added as they start up. All jails are managed >>> by iocage. >>> >>> ifconfig_igb0="-rxcsum -rxcsum6 -txcsum -txcsum6 -vlanhwtag >>> -vlanhwtso up" cloned_interfaces="bridge0" >>> ifconfig_bridge0_name="inet0" >>> ifconfig_inet0="addm igb0 up" >>> ifconfig_inet0_ipv6="inet6 /64 auto_linklocal" >>> >>> $ iocage get interfaces vpro0087 >>> vnet0:inet0 >>> >>> $ ifconfig inet0 >>> inet0: flags=8843 metric 0 >>> mtu 1500 ether 90:1b:0e:63:ef:51 >>> inet6 fe80::921b:eff:fe63:ef51%inet0 prefixlen 64 scopeid 0x4 >>> inet6 prefixlen 64 >>> nd6 options=21 >>> groups: bridge >>> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 >>> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 >>> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 >>> member: vnet0.4 flags=143 >>> ifmaxaddr 0 port 7 priority 128 path cost 2000 >>> member: vnet0.1 flags=143 >>> ifmaxaddr 0 port 6 priority 128 path cost 2000 >>> member: igb0 flags=143 >>> ifmaxaddr 0 port 1 priority 128 path cost 2000000 >>> >>> >>> What we tried: >>> >>> At first we suspected the bridge to become "wedged" somehow. This was >>> corroborated by talking to various people at devsummits and EuroBSDCon >>> with Kristof Provost specifically suggesting that if_bridge was >>> still under giant lock and there might be a problem here that the >>> lock is not released under some race condition and then the entire >>> bridge subsystem would be stalled. That sounds plausible given the >>> random occurrance. >>> >>> But I think we can rule out that one, because: >>> >>> - ifconfig up/down does not help >>> - the host is still communicating fine over the same bridge interface >>> - tearing down the bridge, kldunload (!) of if_bridge.ko followed by >>> a new kldload and reconstructing the members with `ifconfig addm` >>> does not help, either >>> - only a host reboot restores function >>> >>> Finally I created a not iocage managed jail on the problem host. >>> Please ignore the `iocage` in the path, I used it to populate the >>> root directory. But it is not started by iocage at boot time and >>> the manual config is this: >>> >>> testjail { >>> host.hostname = "testjail"; # hostname >>> path = "/iocage/jails/testjail/root"; # root directory >>> exec.clean; >>> exec.system_user = "root"; >>> exec.jail_user = "root"; >>> vnet; >>> vnet.interface = "epair999b"; >>> exec.prestart += "ifconfig epair999 create; ifconfig >>> epair999a inet6 2A00:B580:8000:8000::1/64 auto_linklocal"; >>> exec.poststop += "sleep 2; ifconfig epair999a destroy; sleep 2"; >>> # Standard stuff >>> exec.start += "/bin/sh /etc/rc"; >>> exec.stop = "/bin/sh /etc/rc.shutdown"; >>> exec.consolelog = "/var/log/jail_testjail_console.log"; >>> mount.devfs; #mount devfs >>> allow.raw_sockets; #allow ping-pong >>> devfs_ruleset="4"; #devfs ruleset for this jail >>> } >>> >>> $ cat /iocage/jails/testjail/root/etc/rc.conf >>> hostname="testjail" >>> >>> ifconfig_epair999b_ipv6="inet6 2A00:B580:8000:8000::2/64 >>> auto_linklocal" >>> >>> When I do `service jail onestart testjail` I can then ping6 the jail >>> from the host and the host from the jail. As you can see the >>> if_bridge is not involved in this traffic. >>> >>> When the host is in the wedged state and I start this testjail the >>> same way, no communication across the epair interface is possible. >>> >>> To me this seems to indicate that not the bridge but all epair >>> interfaces stop working at the very same time. >>> >>> >>> OS is RELENG_11_3, hardware and specifically network adapters vary, >>> we have igb, ix, ixl, bnxt ... >>> >>> >>> Does anyone have a suggestion what diagnostic measures could help to >>> pinpoint the culprit? The random occurrance and the fact that the >>> problem seems to prefer the production environment only makes this a >>> real pain ... >>> >>> >>> Thanks and kind regards, >>> Patrick >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >