From owner-freebsd-net@freebsd.org Fri Dec 20 20:36:28 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BD14B1E2A6C for ; Fri, 20 Dec 2019 20:36:28 +0000 (UTC) (envelope-from ryan@ixsystems.com) Received: from mail-ua1-x92d.google.com (mail-ua1-x92d.google.com [IPv6:2607:f8b0:4864:20::92d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47fgWb6grrz4Mkp for ; Fri, 20 Dec 2019 20:36:27 +0000 (UTC) (envelope-from ryan@ixsystems.com) Received: by mail-ua1-x92d.google.com with SMTP id u17so2814112uap.9 for ; Fri, 20 Dec 2019 12:36:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ixsystems.com; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=2XaVLye+4CHa3PHkju87kup2HNb3drWmHfm3o+AKbPc=; b=ka5NhQ25nYXUNfSn0NAn2A1Xl1HRnljtWtEsmZ5hkMSm2xq0GT601JtzOe/3ZKgoDK /P6o3TZR8kkl1uq9hoZ5QgqB96zqdU6acEyNIWTr6BPj3p/skLQQVgPXbFqWGlY/Nf1p imRl5xYrjHhQervWCtZS2F7uVPsJmtzta0GsVOaxdszdrPB7uJ25KZbV2LXHs/0YEdTA A2Sk9SDlY4kDHdrEBVdOtUwURhzVIjhVfF93Nypbdi0X/o2hWfufeqcwHU2eq12J9lpv nrN6Z1J8CxFZPWvZyXPGLL1FTNw9QXyK0Irj+oEv1L4KG0wn4rYpLIiY/lg4IyUiBxSG r3vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=2XaVLye+4CHa3PHkju87kup2HNb3drWmHfm3o+AKbPc=; b=QNPwFM9oa4UNcAQK5m3vFEUp2iP2QnPAYr7DaDXLKl+igKbi5APKEqBePYW3nUCaOn 8rKwy7RQQLZvaebKhaWUtogqutNqiccklyvAurmApP28Ml5QsF5zL10OfE2HMyDHIimf wuvLsha7URj1jGPLrRjJqQVWvNhMsKlE7d6eRvsLfwtQM1YRwOsFBw46H6HP4LY8ql6N ssanlZiZv85SMuBNB2dQOIt2vNNNNGBxxGFw0nery8PUJ7fuD+1WZuzWFgxXTR5ADaOP ZdodeU44GC5zR+7WtxTq0dzkHKLSGCittiSnFKGztYKq7tDLaRI9fuitmB0b8a9Yn/RJ hHNw== X-Gm-Message-State: APjAAAVL4tqBq+SmOcZMhxdpIgBdk8lrdRyev+bHJJn1J4U7euROK7ms MHDAmkvjhBSNRKoptu6LPfW4o0MWuF0+nI/63xcC8Q== X-Google-Smtp-Source: APXvYqx4LlPjG8mw6R3YzTyE57FCkBGILnrs91kW7vKUBJ1orOIP4PsEemiqaLu1FznspTGheU8hhOXzBr6rV+zT6N8= X-Received: by 2002:ab0:21cb:: with SMTP id u11mr10449316uan.16.1576874186364; Fri, 20 Dec 2019 12:36:26 -0800 (PST) MIME-Version: 1.0 References: <20191220122256.76942c07@x23> In-Reply-To: From: Ryan Moeller Date: Fri, 20 Dec 2019 15:36:15 -0500 Message-ID: Subject: Re: Continuing problems in a bridged VNET setup To: Nick Wolff Cc: Marko Zec , "Patrick M. Hausen" , Kristof Provost , "freebsd-net@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 47fgWb6grrz4Mkp X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ixsystems.com header.s=google header.b=ka5NhQ25; dmarc=pass (policy=none) header.from=ixsystems.com; spf=pass (mx1.freebsd.org: domain of ryan@ixsystems.com designates 2607:f8b0:4864:20::92d as permitted sender) smtp.mailfrom=ryan@ixsystems.com X-Spamd-Result: default: False [-5.64 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[ixsystems.com:s=google]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[ixsystems.com:+]; DMARC_POLICY_ALLOW(-0.50)[ixsystems.com,none]; RCVD_IN_DNSWL_NONE(0.00)[d.2.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-2.64)[ip: (-9.08), ipnet: 2607:f8b0::/32(-2.18), asn: 15169(-1.89), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Dec 2019 20:36:28 -0000 See /usr/share/examples/jails/README On Fri, Dec 20, 2019 at 1:09 PM Nick Wolff wrote: > > Marko, > > Are you aware of any write ups for using ng_eiface and ng_bridge instead of > if_bridge? > > Thanks, > > Nick Wolff > > On Fri, Dec 20, 2019 at 6:22 AM Marko Zec wrote: > > > Perhaps you could ditch if_bridge(4) and epair(4), and try ng_eiface(4) > > with ng_bridge(4) instead? Works rock-solid 24/7 here on 11.2 / 11.3. > > > > Marko > > > > On Fri, 20 Dec 2019 11:19:24 +0100 > > "Patrick M. Hausen" wrote: > > > > > Hi all, > > > > > > we still experience occasional network outages in production, > > > yet have not been able to find the root cause. > > > > > > We run around 50 servers with VNET jails. some of them with > > > a handful, the busiest ones with 50 or more jails each. > > > > > > Every now and then the jails are not reachable over the net, > > > anymore. The server itself is up and running, all jails are > > > up and running, one can ssh to the server but none of the > > > jails can communicate over the network. > > > > > > There seems to be no pattern to the time of occurrance except > > > that more jails on one system make it "more likely". > > > Also having more than one bridge, e.g. for private networks > > > between jails seems to increase the probability. > > > When a server shows the problem it tends to get into the state > > > rather frequently, a couple of hours inbetween. Then again > > > most servers run for weeks without exhibiting the problem. > > > That's what makes it so hard to reproduce. The last couple of > > > days one system was failing regularly until we reduced the number > > > of jails from around 80 to around 50. Now it seems stable again. > > > > > > I have a test system with lots of jails that I work with gatling > > > that did not show a single failure so far :-( > > > > > > > > > Setup: > > > > > > All jails are iocage jails with VNET interfaces. They are > > > connected to at least one bridge that starts with the > > > physical external interface as a member and gets jails' > > > epair interfaces added as they start up. All jails are managed > > > by iocage. > > > > > > ifconfig_igb0="-rxcsum -rxcsum6 -txcsum -txcsum6 -vlanhwtag > > > -vlanhwtso up" cloned_interfaces="bridge0" > > > ifconfig_bridge0_name="inet0" > > > ifconfig_inet0="addm igb0 up" > > > ifconfig_inet0_ipv6="inet6 /64 auto_linklocal" > > > > > > $ iocage get interfaces vpro0087 > > > vnet0:inet0 > > > > > > $ ifconfig inet0 > > > inet0: flags=8843 metric 0 > > > mtu 1500 ether 90:1b:0e:63:ef:51 > > > inet6 fe80::921b:eff:fe63:ef51%inet0 prefixlen 64 scopeid 0x4 > > > inet6 prefixlen 64 > > > nd6 options=21 > > > groups: bridge > > > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > > > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > > > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > > > member: vnet0.4 flags=143 > > > ifmaxaddr 0 port 7 priority 128 path cost 2000 > > > member: vnet0.1 flags=143 > > > ifmaxaddr 0 port 6 priority 128 path cost 2000 > > > member: igb0 flags=143 > > > ifmaxaddr 0 port 1 priority 128 path cost 2000000 > > > > > > > > > What we tried: > > > > > > At first we suspected the bridge to become "wedged" somehow. This was > > > corroborated by talking to various people at devsummits and EuroBSDCon > > > with Kristof Provost specifically suggesting that if_bridge was > > > still under giant lock and there might be a problem here that the > > > lock is not released under some race condition and then the entire > > > bridge subsystem would be stalled. That sounds plausible given the > > > random occurrance. > > > > > > But I think we can rule out that one, because: > > > > > > - ifconfig up/down does not help > > > - the host is still communicating fine over the same bridge interface > > > - tearing down the bridge, kldunload (!) of if_bridge.ko followed by > > > a new kldload and reconstructing the members with `ifconfig addm` > > > does not help, either > > > - only a host reboot restores function > > > > > > Finally I created a not iocage managed jail on the problem host. > > > Please ignore the `iocage` in the path, I used it to populate the > > > root directory. But it is not started by iocage at boot time and > > > the manual config is this: > > > > > > testjail { > > > host.hostname = "testjail"; # hostname > > > path = "/iocage/jails/testjail/root"; # root directory > > > exec.clean; > > > exec.system_user = "root"; > > > exec.jail_user = "root"; > > > vnet; > > > vnet.interface = "epair999b"; > > > exec.prestart += "ifconfig epair999 create; ifconfig > > > epair999a inet6 2A00:B580:8000:8000::1/64 auto_linklocal"; > > > exec.poststop += "sleep 2; ifconfig epair999a destroy; sleep 2"; > > > # Standard stuff > > > exec.start += "/bin/sh /etc/rc"; > > > exec.stop = "/bin/sh /etc/rc.shutdown"; > > > exec.consolelog = "/var/log/jail_testjail_console.log"; > > > mount.devfs; #mount devfs > > > allow.raw_sockets; #allow ping-pong > > > devfs_ruleset="4"; #devfs ruleset for this jail > > > } > > > > > > $ cat /iocage/jails/testjail/root/etc/rc.conf > > > hostname="testjail" > > > > > > ifconfig_epair999b_ipv6="inet6 2A00:B580:8000:8000::2/64 > > > auto_linklocal" > > > > > > When I do `service jail onestart testjail` I can then ping6 the jail > > > from the host and the host from the jail. As you can see the > > > if_bridge is not involved in this traffic. > > > > > > When the host is in the wedged state and I start this testjail the > > > same way, no communication across the epair interface is possible. > > > > > > To me this seems to indicate that not the bridge but all epair > > > interfaces stop working at the very same time. > > > > > > > > > OS is RELENG_11_3, hardware and specifically network adapters vary, > > > we have igb, ix, ixl, bnxt ... > > > > > > > > > Does anyone have a suggestion what diagnostic measures could help to > > > pinpoint the culprit? The random occurrance and the fact that the > > > problem seems to prefer the production environment only makes this a > > > real pain ... > > > > > > > > > Thanks and kind regards, > > > Patrick > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Ryan Moeller iXsystems, Inc. OS Developer Email: ryan@iXsystems.com