From owner-freebsd-net@freebsd.org Fri Dec 20 20:57:06 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0D1291E311B for ; Fri, 20 Dec 2019 20:57:06 +0000 (UTC) (envelope-from zec@fer.hr) Received: from EUR04-DB3-obe.outbound.protection.outlook.com (mail-eopbgr60088.outbound.protection.outlook.com [40.107.6.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47fgzN5cryz4P5F for ; Fri, 20 Dec 2019 20:57:04 +0000 (UTC) (envelope-from zec@fer.hr) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=h/ROe7TDjD9K0uCccfPToftv+VQKc9dOd+ge3O31dS/0In2u8Pmqjacc/S+XElpALx7K62zEFE/P096JWJAnGnMMnY7yv26g+UqKUJR6nky+ImGlmnk3URtkIsCHY2aV3KQxpXcJt3Yl2HHgbGFUYjeiSzPNkmytncV3QHdPQXjgP+dQjuD1Xje3iqID6M7cGhyY15+3lMAJtMzl1E4Ms5wxx1gAfkCJ4Fr+i72y40v6hN1glz9neOiwKO5u5C7o1A1X0E5h4LWNpv5ZHMKcwjy8LYgiz2e+wgmTQRis3CI9FZYahKwbv7+TVyqpAlF0hi7lmVb/QOabtRblmGRMhQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o11rmkpCHGCC+3n2NkyZgUR1nxPvTMIpSSBoA1hZ8ks=; b=lZmpe7mUZ82JAUa3mZUtgX0hlkz3iZefHHpaQtI/YaApD5XVPRZlp0migxrzg+uAFzMQYqyja5h7a/bdrWOIZEPfj6ocyPhB5P2VhxAypAhaurnbn4nFo4+6bZ7JgW6LhJvoMHtpfSmmjkhbU0jfCb7ibZIdh0qMG2ULEj1brSq4wCvhPiX/4uY4S5XQWgCoguQz/Q1l+3lG80m8OLwCrEQB3OaX3wU6YkNLzSpA4S0xkEyTuWVpdixPMpo/5gHQ1paXrsqPaXRSjeb666+BqYcQFlwOcMr+cWBCE0GHlctk60Ka5GmSL1sUCG44Fe5v/DbapAsU4MFsArb6sPTO2Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fer.hr; dmarc=pass action=none header.from=fer.hr; dkim=pass header.d=fer.hr; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ferhr.onmicrosoft.com; s=selector2-ferhr-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=o11rmkpCHGCC+3n2NkyZgUR1nxPvTMIpSSBoA1hZ8ks=; b=gjVjNV7CBHlHUFmFW7X5FdluRhqvokedz4g3+kPrB48N02UDMtDIdVUIXIK8AwYsJaPtZaDsXSMtPvPoGl4OW+3Ep+WvVMrGlBRaAKBlpTz24lEqV2mQZNYms3g7FIta3zYawbGfGGdLo1G3EIZzVrnbhMI9efdjmsB7rXW34XU= Received: from AM6PR08MB3078.eurprd08.prod.outlook.com (52.135.164.16) by AM6PR08MB4151.eurprd08.prod.outlook.com (20.179.0.90) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.16; Fri, 20 Dec 2019 20:57:02 +0000 Received: from AM6PR08MB3078.eurprd08.prod.outlook.com ([fe80::a8d0:1e6:a51:66aa]) by AM6PR08MB3078.eurprd08.prod.outlook.com ([fe80::a8d0:1e6:a51:66aa%3]) with mapi id 15.20.2559.016; Fri, 20 Dec 2019 20:57:02 +0000 From: Marko Zec To: Nick Wolff CC: "Patrick M. Hausen" , Kristof Provost , "freebsd-net@freebsd.org" Subject: Re: Continuing problems in a bridged VNET setup Thread-Topic: Continuing problems in a bridged VNET setup Thread-Index: AQHVtx7/PdlOyA2FWE2l/uUQiHRcnKfC4WkAgABxsgCAAC71AA== Date: Fri, 20 Dec 2019 20:57:01 +0000 Message-ID: <20191220215756.18814c22@x23> References: <20191220122256.76942c07@x23> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: VI1PR08CA0230.eurprd08.prod.outlook.com (2603:10a6:802:15::39) To AM6PR08MB3078.eurprd08.prod.outlook.com (2603:10a6:209:46::16) x-ms-exchange-messagesentrepresentingtype: 1 x-mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; amd64-portbld-freebsd11.3) x-originating-ip: [31.147.104.126] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 0d367708-025f-4203-c1e9-08d7858f295c x-ms-traffictypediagnostic: AM6PR08MB4151: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:8882; x-forefront-prvs: 025796F161 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(7916004)(346002)(396003)(376002)(136003)(366004)(39850400004)(53754006)(189003)(199004)(64756008)(2906002)(66446008)(6512007)(5660300002)(66946007)(66476007)(966005)(1076003)(9686003)(6486002)(52116002)(4326008)(71200400001)(33716001)(66556008)(6916009)(316002)(81156014)(186003)(86362001)(8676002)(8936002)(81166006)(26005)(786003)(6506007)(478600001)(54906003)(53546011)(39210200001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR08MB4151; H:AM6PR08MB3078.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: fer.hr does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: PLr9//QdLPiafrBov+1HnocbMMu70woIfV85Sm6EdKKwmxlSoNkuwL6Fgj7Ifelw+6+hA8vdWlpGuiB5VQlVIhJzs1ux+PO069a0/2Qv3cnRjfrunS5Wz2RmjLZRbw/WEWua2I/VPp+3de5uGTlZ5HARZOGYjVe2LpfkNDLC93bRg53WWJsJdQGvuLBCc7V11alle7yBHppKilH6OqTIStPxjxABu51A99QAX2v6evzz1Mi+4sYqGK5a+ZYnMwMDE4G0TrTI2N0MfgQHBI6/janyLk/Z0M+D0RQAVN9gbk8HTLHM0a1Fyjyn5mPoOg+Z0D+FIo76fvARBZ4V0cG8vGYOo3uxYsshQ6X0VgJ5gWF/UIVhv2BKkfxgdGAyXxvLja6x6tNWzjNBfXjGyY1fz/EXfEC9wqLpez2x4WajKPCdqq15TEF6G0uiAK4vR7vQ0lVEP6OTgc5GkNfgE/WCJLhFkksYluiQoV6d+N3lV1JNdSszzQkjNg6aqIrl+JyIElHQaoXF82xK7qDNVAzltRM14qQ7lZx8KpbZlKjUnPQ5741C/3LfehGeq5YisKzY x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-ID: <589089AB77AC204BBDBBA7AA1C6456C0@eurprd08.prod.outlook.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: fer.hr X-MS-Exchange-CrossTenant-Network-Message-Id: 0d367708-025f-4203-c1e9-08d7858f295c X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Dec 2019 20:57:01.9545 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: ca71eddc-cc7b-4e5b-95bd-55b658e696be X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: GgA4ehvKFms8tuqsnI7ehCtWwL40nnCo8ASgQAN6bfaiuTiey35vtXBf6thOwKD/ X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB4151 X-Rspamd-Queue-Id: 47fgzN5cryz4P5F X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ferhr.onmicrosoft.com header.s=selector2-ferhr-onmicrosoft-com header.b=gjVjNV7C; dmarc=none; spf=pass (mx1.freebsd.org: domain of zec@fer.hr designates 40.107.6.88 as permitted sender) smtp.mailfrom=zec@fer.hr X-Spamd-Result: default: False [-4.36 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[ferhr.onmicrosoft.com:s=selector2-ferhr-onmicrosoft-com]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[fer.hr]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[ferhr.onmicrosoft.com:+]; RCVD_IN_DNSWL_NONE(0.00)[88.6.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.36)[ipnet: 40.64.0.0/10(-3.84), asn: 8075(-2.92), country: US(-0.05)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Dec 2019 20:57:06 -0000 On Fri, 20 Dec 2019 13:09:52 -0500 Nick Wolff wrote: > Marko, >=20 > Are you aware of any write ups for using ng_eiface and ng_bridge > instead of if_bridge? It is not that complex at all: # kldload ng_ether # ifconfig em0 promisc # ngctl mkpeer em0: bridge lower link0 # ngctl name em0:lower b0 # ngctl connect em0: b0: upper link1 # ngctl mkpeer b0: eiface link2 ether # ngctl mkpeer b0: eiface link3 ether # ngctl mkpeer b0: eiface link4 ether Done - this should create interfaces ngeth0, ngeth1 and ngeth2, which one can assign to vnet jails. Note that unlike epair, ngeth instances do not automatically get a MAC address assigned, at least not on 11.x / 12.x, so this is an extra step one has to perform on his own. In our setup, we actually use https://github.com/imunes/imunes to set up the (netgraph-based) virtual network and nodes (vnet jails, aka vimages). Works reasonably well, having in mind that the thing was devised as a network emulation tool, not a virtual host provisioning framework. Marko >=20 > Thanks, >=20 > Nick Wolff >=20 > On Fri, Dec 20, 2019 at 6:22 AM Marko Zec wrote: >=20 > > Perhaps you could ditch if_bridge(4) and epair(4), and try > > ng_eiface(4) with ng_bridge(4) instead? Works rock-solid 24/7 here > > on 11.2 / 11.3. > > > > Marko > > > > On Fri, 20 Dec 2019 11:19:24 +0100 > > "Patrick M. Hausen" wrote: > > =20 > > > Hi all, > > > > > > we still experience occasional network outages in production, > > > yet have not been able to find the root cause. > > > > > > We run around 50 servers with VNET jails. some of them with > > > a handful, the busiest ones with 50 or more jails each. > > > > > > Every now and then the jails are not reachable over the net, > > > anymore. The server itself is up and running, all jails are > > > up and running, one can ssh to the server but none of the > > > jails can communicate over the network. > > > > > > There seems to be no pattern to the time of occurrance except > > > that more jails on one system make it "more likely". > > > Also having more than one bridge, e.g. for private networks > > > between jails seems to increase the probability. > > > When a server shows the problem it tends to get into the state > > > rather frequently, a couple of hours inbetween. Then again > > > most servers run for weeks without exhibiting the problem. > > > That's what makes it so hard to reproduce. The last couple of > > > days one system was failing regularly until we reduced the number > > > of jails from around 80 to around 50. Now it seems stable again. > > > > > > I have a test system with lots of jails that I work with gatling > > > that did not show a single failure so far :-( > > > > > > > > > Setup: > > > > > > All jails are iocage jails with VNET interfaces. They are > > > connected to at least one bridge that starts with the > > > physical external interface as a member and gets jails' > > > epair interfaces added as they start up. All jails are managed > > > by iocage. > > > > > > ifconfig_igb0=3D"-rxcsum -rxcsum6 -txcsum -txcsum6 -vlanhwtag > > > -vlanhwtso up" cloned_interfaces=3D"bridge0" > > > ifconfig_bridge0_name=3D"inet0" > > > ifconfig_inet0=3D"addm igb0 up" > > > ifconfig_inet0_ipv6=3D"inet6 /64 auto_linklocal" > > > > > > $ iocage get interfaces vpro0087 > > > vnet0:inet0 > > > > > > $ ifconfig inet0 > > > inet0: flags=3D8843 metric 0 > > > mtu 1500 ether 90:1b:0e:63:ef:51 > > > inet6 fe80::921b:eff:fe63:ef51%inet0 prefixlen 64 scopeid > > > 0x4 inet6 prefixlen 64 > > > nd6 options=3D21 > > > groups: bridge > > > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > > > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > > > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > > > member: vnet0.4 > > > flags=3D143 ifmaxaddr 0 port 7 > > > priority 128 path cost 2000 member: vnet0.1 > > > flags=3D143 ifmaxaddr 0 port 6 > > > priority 128 path cost 2000 member: igb0 > > > flags=3D143 ifmaxaddr 0 port 1 > > > priority 128 path cost 2000000 > > > > > > > > > What we tried: > > > > > > At first we suspected the bridge to become "wedged" somehow. This > > > was corroborated by talking to various people at devsummits and > > > EuroBSDCon with Kristof Provost specifically suggesting that > > > if_bridge was still under giant lock and there might be a problem > > > here that the lock is not released under some race condition and > > > then the entire bridge subsystem would be stalled. That sounds > > > plausible given the random occurrance. > > > > > > But I think we can rule out that one, because: > > > > > > - ifconfig up/down does not help > > > - the host is still communicating fine over the same bridge > > > interface > > > - tearing down the bridge, kldunload (!) of if_bridge.ko followed > > > by a new kldload and reconstructing the members with `ifconfig > > > addm` does not help, either > > > - only a host reboot restores function > > > > > > Finally I created a not iocage managed jail on the problem host. > > > Please ignore the `iocage` in the path, I used it to populate the > > > root directory. But it is not started by iocage at boot time and > > > the manual config is this: > > > > > > testjail { > > > host.hostname =3D "testjail"; # hostname > > > path =3D "/iocage/jails/testjail/root"; # root directory > > > exec.clean; > > > exec.system_user =3D "root"; > > > exec.jail_user =3D "root"; > > > vnet; > > > vnet.interface =3D "epair999b"; > > > exec.prestart +=3D "ifconfig epair999 create; ifconfig > > > epair999a inet6 2A00:B580:8000:8000::1/64 auto_linklocal"; > > > exec.poststop +=3D "sleep 2; ifconfig epair999a destroy; sleep 2"; > > > # Standard stuff > > > exec.start +=3D "/bin/sh /etc/rc"; > > > exec.stop =3D "/bin/sh /etc/rc.shutdown"; > > > exec.consolelog =3D "/var/log/jail_testjail_console.log"; > > > mount.devfs; #mount devfs > > > allow.raw_sockets; #allow ping-pong > > > devfs_ruleset=3D"4"; #devfs ruleset for this jail > > > } > > > > > > $ cat /iocage/jails/testjail/root/etc/rc.conf > > > hostname=3D"testjail" > > > > > > ifconfig_epair999b_ipv6=3D"inet6 2A00:B580:8000:8000::2/64 > > > auto_linklocal" > > > > > > When I do `service jail onestart testjail` I can then ping6 the > > > jail from the host and the host from the jail. As you can see the > > > if_bridge is not involved in this traffic. > > > > > > When the host is in the wedged state and I start this testjail the > > > same way, no communication across the epair interface is possible. > > > > > > To me this seems to indicate that not the bridge but all epair > > > interfaces stop working at the very same time. > > > > > > > > > OS is RELENG_11_3, hardware and specifically network adapters > > > vary, we have igb, ix, ixl, bnxt ... > > > > > > > > > Does anyone have a suggestion what diagnostic measures could help > > > to pinpoint the culprit? The random occurrance and the fact that > > > the problem seems to prefer the production environment only makes > > > this a real pain ... > > > > > > > > > Thanks and kind regards, > > > Patrick =20 > > > > _______________________________________________ > > freebsd-net@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-net > > To unsubscribe, send any mail to > > "freebsd-net-unsubscribe@freebsd.org"=20