From nobody Sat May 14 09:39:23 2022 X-Original-To: freebsd-current@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id A75E91AD1487 for ; Sat, 14 May 2022 09:39:34 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from smtp6.goneo.de (smtp6.goneo.de [85.220.129.31]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4L0gTK4gLQz3L6w for ; Sat, 14 May 2022 09:39:33 +0000 (UTC) (envelope-from freebsd@walstatt-de.de) Received: from hub1.goneo.de (hub1.goneo.de [IPv6:2001:1640:5::8:52]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by smtp6.goneo.de (Postfix) with ESMTPS id 7F7FD10A1E85 for ; Sat, 14 May 2022 11:39:26 +0200 (CEST) Received: from hub1.goneo.de (localhost [127.0.0.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPS id DBCE610A32F4 for ; Sat, 14 May 2022 11:39:24 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=walstatt-de.de; s=DKIM001; t=1652521164; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=lUgwIb88NBIwa5jgHyeUyJAX4mVmlmqgNMQtVNYsyJI=; b=J1+0TpypQzPX/g2q2EEbV7lE49DZUKORnFw52Z7rkHVJyvd0jH6vNkpqcZIbkQTzDl7ydL sITrO7rVmiiXIDAMe/m7bCrn8Geqvyu23cn4bdeJJ4L5i4H+YUZBmcXKG0guXmrnz0nD4h 7c6Oxsivq4beAHFzD3Fexvqc2EIFFVvinJZDrunP68PeGOm0BNzaDAAI0KZbySPBFVPwgx j745Vt7qqGbnnxHHFpEW/oHKH6w2ZE5/4lNPpd42NWB5keGrkAnUTNku1GbLYJPU3tfuPj on7kwcOT9xmp98KjVTSxvkCkNFxrqTpw0P4KhRYOLnWcpaZvK7wXe+HH/uf4Fg== Received: from hermann (dynamic-2a01-0c23-5de7-7300-fc43-3716-9269-5894.c23.pool.telefonica.de [IPv6:2a01:c23:5de7:7300:fc43:3716:9269:5894]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by hub1.goneo.de (Postfix) with ESMTPSA id A129F10A330F for ; Sat, 14 May 2022 11:39:24 +0200 (CEST) Date: Sat, 14 May 2022 11:39:23 +0200 From: FreeBSD User To: FreeBSD CURRENT Subject: vnet/if_bridge: ARP issues: connection failure Message-ID: <20220514113923.523f1ea9@hermann> List-Id: Discussions about the use of FreeBSD-current List-Archive: https://lists.freebsd.org/archives/freebsd-current List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-current@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Rspamd-UID: ce95d9 X-Rspamd-UID: 3ea6f9 X-Rspamd-Queue-Id: 4L0gTK4gLQz3L6w X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=walstatt-de.de header.s=DKIM001 header.b=J1+0Tpyp; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd@walstatt-de.de has no SPF policy when checking 85.220.129.31) smtp.mailfrom=freebsd@walstatt-de.de X-Spamd-Result: default: False [-2.15 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[walstatt-de.de:s=DKIM001]; NEURAL_HAM_MEDIUM(-0.99)[-0.990]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-current@freebsd.org]; DMARC_NA(0.00)[walstatt-de.de]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[walstatt-de.de:+]; NEURAL_HAM_SHORT(-0.26)[-0.262]; MLMMJ_DEST(0.00)[freebsd-current]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; MID_RHS_NOT_FQDN(0.50)[]; ASN(0.00)[asn:25394, ipnet:85.220.128.0/17, country:DE]; RCVD_TLS_ALL(0.00)[]; RCVD_IN_DNSWL_LOW(-0.10)[85.220.129.31:from] X-ThisMailContainsUnwantedMimeParts: N Hello, the problem I'm about to report is not specific to CURRENT, I report this to CURRENT because I'm list member. The problem occurs on FreeBSD 12.3-RELEASE-p5. Setup: connecting to vnet jails bound to a dedicated NIC via if_bridge results in an erratic behaviour (from my point of view). The box has two NICs, em0 which is dedicated for managing the host and igb0 for services like NFS, SMB and jails (the host is de facto a Xigmanas box). The NIC igb0 also has an IPv4 which is accessible without problem (sshd is listening on em0 and igb0 and any service bound to igb0 and its IP itself works like a charme, execept anything that is connected via if_bridge/vnet). Both, em0 and igb0, are member of the same network and connected to the same switch (I guess, it's the campus infrastructure, I have no access to that). Phenomenon: trying to ping a jail results initially in a long term waiting and often in "Host is down" - but then, sudenly, the jail is responding. Trying to connect to port 22/tcp of any jail doesn't work in 90% of the cases, but randomly, a host (out of five) does respond and the connection can be established. Terminating the connection and tryin again is in 99% then a fail. Once connected the ssh connection fries after a couple of seconds of inactivity. Checking ARP on the jail (login via host and jexec) and listening via tcpdump -XXvi vnet0 arp on a jail while pinging the jail from the netowrk shows up the typical requests, but not every request is then acompanied by a reply. I'm not firm in terms of networking and investigating ARP issues, so I followed some instructions found with ARP issues on FBSD, vnet and routing. MIB settings (on the host itself, vnet untouched): net.inet.ip.forwarding: 0 net.link.bridge.ipfw: 0 net.link.bridge.allow_llz_overlap: 0 net.link.bridge.inherit_mac: 0 net.link.bridge.log_stp: 0 net.link.bridge.pfil_local_phys: 0 net.link.bridge.pfil_member: 0 net.link.bridge.ipfw_arp: 0 net.link.bridge.pfil_bridge: 0 net.link.bridge.pfil_onlyip: 0 I also realised that on the igb0 interace checksum errors occured while rxcsum is enabled. I disbaled special features via "ifconfig igb0 -rxcsum -txcsum -tso -lro I'm out of ideas here. Another box, the same base OS, similar setup (two NICs, same ambition), but with the difference that the second NIC resides on a different network and is connected to a different switch, also if_bridge and vnet attached jails, there is no problem. Either there is a serious bug in 12.3-p5 (haven't had the chance to check on 13/CURRENT) or I'm doing something terribly wrong. Some technical details: em0@pci0:0:25:0: class=0x020000 card=0x29828016 chip=0x153b8086 rev=0x04 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Connection I217-V' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xf7d00000, size 131072, enabled bar [14] = type Memory, range 32, base 0xf7d35000, size 4096, enabled bar [18] = type I/O Port, range 32, base 0xf080, size 32, enabled cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 13[e0] = PCI Advanced Features: FLR TP gb0@pci0:4:0:0: class=0x020000 card=0x00028086 chip=0x15848086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = 'I210 Gigabit Network Connection' class = network subclass = ethernet bar [10] = type Memory, range 32, base 0xf7900000, size 1048576, enabled bar [1c] = type Memory, range 32, base 0xf7a00000, size 16384, enabled cap 01[40] = powerspec 3 supports D0 D3 current D0 cap 05[50] = MSI supports 1 message, 64 bit, vector masks cap 11[70] = MSI-X supports 5 messages, enabled Table in map 0x1c[0x0], PBA in map 0x1c[0x2000] cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR NS max read 512 link x1(x1) speed 2.5(2.5) ASPM L1(L0s/L1) ecap 0001[100] = AER 2 0 fatal 0 non-fatal 0 corrected ecap 0003[140] = Serial 1 xxxxxxxxxxxxxxxxxxxxx ecap 0017[1a0] = TPH Requester 1 Kind regards, oh