From owner-freebsd-net@freebsd.org Fri Dec 20 11:22:06 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 98E431D6A89 for ; Fri, 20 Dec 2019 11:22:06 +0000 (UTC) (envelope-from zec@fer.hr) Received: from EUR02-AM5-obe.outbound.protection.outlook.com (mail-eopbgr00064.outbound.protection.outlook.com [40.107.0.64]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47fRCw4ZB0z3MT6 for ; Fri, 20 Dec 2019 11:22:04 +0000 (UTC) (envelope-from zec@fer.hr) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H9qaOMFOTqfu9pthsdemiNqUg86FUPJnCI9RWFGZRqcURiudVM1Prbn8yuNmlC6wNnRBXaiMlFebJ+c5IR8R+OzVytHLgVM6hvPVKAuSuwp6q9iAY1P1VX3CWAKrYpsa93mEZGW7KGyKmNEeTtEZ2Ox88YaHGtSOj8iiqdJnyVLRkSgdTBNu0eIY9DKDwmK4xCDTtq56b0ZtzpIBCtm0Ku9O/aVT9XWTsH7g5ZTVoUCqODaUrSderWZ/dVm4JHwGFcaLFmfGaYuMRAT9B+PtFOXz+neHpFoWvWIlWIyRwiHz4ThRnaXnTV/EaRLEYXxrQHSh2kSKCYehEA9Z/5rbEw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=K2H2xWCn4Sn31fYk8GloZC5yaHdWu8ekK8nkbPCaWVA=; b=dFwdiux0NTGih7L0dcS9Ff0Po7kaMpDIlxkucsRJ5Rqe9iyPt609J7RxDVMSYedl/Puyiftrats2+sllBXdAwRk+AplrK+aI3Tp+svFgYe7uQDAlqorcxYQXPCD9SUuRs5TTVtUSDC/7q3Tsyfi+bNSMX3bG5rgfeLqbue8LlClvTKdhAhkwuerFFkUN0mnhVxUHJqi8Q1ccVFGKUhyGx/Q73+PL8gDVukLWfmNKlin8r1+6abnp3rsylRrCJKt8aloIzRp8+xIp4lTQYu+UQ/RibYGLBu2yTYPG+ueJFGoAyU8p0u7V5sqojqV75uMXK2tnpDCVqd9XPII8SCLkTw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fer.hr; dmarc=pass action=none header.from=fer.hr; dkim=pass header.d=fer.hr; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ferhr.onmicrosoft.com; s=selector2-ferhr-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=K2H2xWCn4Sn31fYk8GloZC5yaHdWu8ekK8nkbPCaWVA=; b=hF36dFR/Wkb3ag3wuq6IgruSyuLTFcMf3fMO1j/GgFbuSYAeuDPjr+yK/piriBKwAw8hO/7iMiqrTFFS6c1Zk96n9eKno1c83Kt1//RvZvdF9AVYnnA6Q+f/Vc/PMJD0gW8q6oCVV6obSjvHIlZsr5ZIJEiPZnj3M36hnM2MRWE= Received: from AM6PR08MB3078.eurprd08.prod.outlook.com (52.135.164.16) by AM6PR08MB3541.eurprd08.prod.outlook.com (20.177.116.95) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2538.14; Fri, 20 Dec 2019 11:22:01 +0000 Received: from AM6PR08MB3078.eurprd08.prod.outlook.com ([fe80::a8d0:1e6:a51:66aa]) by AM6PR08MB3078.eurprd08.prod.outlook.com ([fe80::a8d0:1e6:a51:66aa%3]) with mapi id 15.20.2559.016; Fri, 20 Dec 2019 11:22:01 +0000 From: Marko Zec To: "Patrick M. Hausen" CC: "freebsd-net@freebsd.org" , Kristof Provost Subject: Re: Continuing problems in a bridged VNET setup Thread-Topic: Continuing problems in a bridged VNET setup Thread-Index: AQHVtx7/PdlOyA2FWE2l/uUQiHRcnKfC4WkA Date: Fri, 20 Dec 2019 11:22:01 +0000 Message-ID: <20191220122256.76942c07@x23> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-clientproxiedby: AM3PR07CA0069.eurprd07.prod.outlook.com (2603:10a6:207:4::27) To AM6PR08MB3078.eurprd08.prod.outlook.com (2603:10a6:209:46::16) x-ms-exchange-messagesentrepresentingtype: 1 x-mailer: Claws Mail 3.17.4 (GTK+ 2.24.32; amd64-portbld-freebsd11.3) x-originating-ip: [161.53.19.9] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: d8d5fb94-a385-4b3d-0757-08d7853ed5a6 x-ms-traffictypediagnostic: AM6PR08MB3541: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:9508; x-forefront-prvs: 025796F161 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(7916004)(396003)(39850400004)(136003)(366004)(376002)(346002)(199004)(189003)(53754006)(9686003)(6512007)(478600001)(186003)(6916009)(6486002)(33716001)(6506007)(5660300002)(52116002)(71200400001)(316002)(786003)(26005)(8936002)(81166006)(81156014)(8676002)(2906002)(66556008)(66946007)(66476007)(4326008)(64756008)(54906003)(1076003)(86362001)(66446008)(39210200001); DIR:OUT; SFP:1101; SCL:1; SRVR:AM6PR08MB3541; H:AM6PR08MB3078.eurprd08.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: fer.hr does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: E0DA4tZanMhIHKKdc5DcS0bJlgsLPsaqHjZi2rOEG34IMegWRmR+moTFFRz9JaGUe5HFlOTGFfEXStsv3JYyyGuA8CS2d9hCJShtoMQW1GSpDVl1tVb1AGLjwXI3Hm7sBIwOam2ZAZMWabpu45/qH70TqOF25tPV8vtoYooOqyPY10nRRmJCK4rez8deH0+Gq3eaxgbWDeg/yzTz9wEOKDgmQv1Z6Jzif4CjY38ob5wDTa5kavlhUOHdQ9WWCwjUOhLnEjKvKZPsDBvs3iswWyRknBQH3KhxUpDAN6KAFsSt+bnV/Rjq1XrxPtTNTO+2nKtSt7nMRg3J01XunVSpUrOFPRFWvrh2t4FAVFLWKOMbyjDQqW48L9XiGboWxyKKLSiwgucbxCpMwk1ezJp7hXNWMPds5f9tTd5MzEAdwv78wR3JvIuFvV8EncdQOOv8xjag4fnNwC8lOtB3XvixcJV/c98PuNhGnh4hyDo4ZOV5W2uKBZbFSfA7SHnH+4ykSW6HBXXP3qGjVPhh24q30g== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: fer.hr X-MS-Exchange-CrossTenant-Network-Message-Id: d8d5fb94-a385-4b3d-0757-08d7853ed5a6 X-MS-Exchange-CrossTenant-originalarrivaltime: 20 Dec 2019 11:22:01.6877 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: ca71eddc-cc7b-4e5b-95bd-55b658e696be X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: LKL+dg/L6MAONcvSxSHyx20+yK9GjINQaZw3Pb8L3Iied3Dl0kMfusNUruEdepuQ X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM6PR08MB3541 X-Rspamd-Queue-Id: 47fRCw4ZB0z3MT6 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=ferhr.onmicrosoft.com header.s=selector2-ferhr-onmicrosoft-com header.b=hF36dFR/; dmarc=none; spf=pass (mx1.freebsd.org: domain of zec@fer.hr designates 40.107.0.64 as permitted sender) smtp.mailfrom=zec@fer.hr X-Spamd-Result: default: False [-4.36 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[ferhr.onmicrosoft.com:s=selector2-ferhr-onmicrosoft-com]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[fer.hr]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[ferhr.onmicrosoft.com:+]; RCVD_IN_DNSWL_NONE(0.00)[64.0.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.36)[ipnet: 40.64.0.0/10(-3.84), asn: 8075(-2.92), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_NOT_FQDN(0.50)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Dec 2019 11:22:06 -0000 Perhaps you could ditch if_bridge(4) and epair(4), and try ng_eiface(4) with ng_bridge(4) instead? Works rock-solid 24/7 here on 11.2 / 11.3. Marko On Fri, 20 Dec 2019 11:19:24 +0100 "Patrick M. Hausen" wrote: > Hi all, >=20 > we still experience occasional network outages in production, > yet have not been able to find the root cause. >=20 > We run around 50 servers with VNET jails. some of them with > a handful, the busiest ones with 50 or more jails each. >=20 > Every now and then the jails are not reachable over the net, > anymore. The server itself is up and running, all jails are > up and running, one can ssh to the server but none of the > jails can communicate over the network. >=20 > There seems to be no pattern to the time of occurrance except > that more jails on one system make it "more likely". > Also having more than one bridge, e.g. for private networks > between jails seems to increase the probability. > When a server shows the problem it tends to get into the state > rather frequently, a couple of hours inbetween. Then again > most servers run for weeks without exhibiting the problem. > That's what makes it so hard to reproduce. The last couple of > days one system was failing regularly until we reduced the number > of jails from around 80 to around 50. Now it seems stable again. >=20 > I have a test system with lots of jails that I work with gatling > that did not show a single failure so far :-( >=20 >=20 > Setup: >=20 > All jails are iocage jails with VNET interfaces. They are > connected to at least one bridge that starts with the > physical external interface as a member and gets jails' > epair interfaces added as they start up. All jails are managed > by iocage. >=20 > ifconfig_igb0=3D"-rxcsum -rxcsum6 -txcsum -txcsum6 -vlanhwtag > -vlanhwtso up" cloned_interfaces=3D"bridge0" > ifconfig_bridge0_name=3D"inet0" > ifconfig_inet0=3D"addm igb0 up" > ifconfig_inet0_ipv6=3D"inet6 /64 auto_linklocal" >=20 > $ iocage get interfaces vpro0087 > vnet0:inet0 >=20 > $ ifconfig inet0 > inet0: flags=3D8843 metric 0 > mtu 1500 ether 90:1b:0e:63:ef:51 > inet6 fe80::921b:eff:fe63:ef51%inet0 prefixlen 64 scopeid 0x4 > inet6 prefixlen 64 > nd6 options=3D21 > groups: bridge > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: vnet0.4 flags=3D143 > ifmaxaddr 0 port 7 priority 128 path cost 2000 > member: vnet0.1 flags=3D143 > ifmaxaddr 0 port 6 priority 128 path cost 2000 > member: igb0 flags=3D143 > ifmaxaddr 0 port 1 priority 128 path cost 2000000 >=20 >=20 > What we tried: >=20 > At first we suspected the bridge to become "wedged" somehow. This was > corroborated by talking to various people at devsummits and EuroBSDCon > with Kristof Provost specifically suggesting that if_bridge was > still under giant lock and there might be a problem here that the > lock is not released under some race condition and then the entire > bridge subsystem would be stalled. That sounds plausible given the > random occurrance. >=20 > But I think we can rule out that one, because: >=20 > - ifconfig up/down does not help > - the host is still communicating fine over the same bridge interface > - tearing down the bridge, kldunload (!) of if_bridge.ko followed by > a new kldload and reconstructing the members with `ifconfig addm` > does not help, either > - only a host reboot restores function >=20 > Finally I created a not iocage managed jail on the problem host. > Please ignore the `iocage` in the path, I used it to populate the > root directory. But it is not started by iocage at boot time and > the manual config is this: >=20 > testjail { > host.hostname =3D "testjail"; # hostname > path =3D "/iocage/jails/testjail/root"; # root directory > exec.clean; > exec.system_user =3D "root"; > exec.jail_user =3D "root"; > vnet;=20 > vnet.interface =3D "epair999b"; > exec.prestart +=3D "ifconfig epair999 create; ifconfig > epair999a inet6 2A00:B580:8000:8000::1/64 auto_linklocal"; > exec.poststop +=3D "sleep 2; ifconfig epair999a destroy; sleep 2";=20 > # Standard stuff > exec.start +=3D "/bin/sh /etc/rc"; > exec.stop =3D "/bin/sh /etc/rc.shutdown"; > exec.consolelog =3D "/var/log/jail_testjail_console.log"; > mount.devfs; #mount devfs > allow.raw_sockets; #allow ping-pong > devfs_ruleset=3D"4"; #devfs ruleset for this jail > } >=20 > $ cat /iocage/jails/testjail/root/etc/rc.conf > hostname=3D"testjail" >=20 > ifconfig_epair999b_ipv6=3D"inet6 2A00:B580:8000:8000::2/64 > auto_linklocal" >=20 > When I do `service jail onestart testjail` I can then ping6 the jail > from the host and the host from the jail. As you can see the > if_bridge is not involved in this traffic. >=20 > When the host is in the wedged state and I start this testjail the > same way, no communication across the epair interface is possible. >=20 > To me this seems to indicate that not the bridge but all epair > interfaces stop working at the very same time. >=20 >=20 > OS is RELENG_11_3, hardware and specifically network adapters vary, > we have igb, ix, ixl, bnxt ... >=20 >=20 > Does anyone have a suggestion what diagnostic measures could help to > pinpoint the culprit? The random occurrance and the fact that the > problem seems to prefer the production environment only makes this a > real pain ... >=20 >=20 > Thanks and kind regards, > Patrick