From nobody Mon Mar 6 22:51:19 2023 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PVv2K4MZnz3wVXD for ; Mon, 6 Mar 2023 22:51:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PVv2K3cNyz3NDc for ; Mon, 6 Mar 2023 22:51:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1678143081; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=MlSZ+/wV4XaoT0d4EABJqt4nbzciQWymzCsBun4LqZY=; b=wkM8gaJgm/qTssmLHCn5cIEq5Bqm0YkFwPl4nS1b4C1YTOKN+lL468PhzZEk/u8GWl7yn2 9VUjPRlk0+4UYB+DusmQO5uWsizB6TFJUd/Xn1q/njwelvzZXxhKFjo9nZmiM16IW9uSsL tu+b7ooD29+knYBMcdyb+d3wzD494RtB3s2GARdqayHOozZjYgLJY9xs5b7O2Ri/xRgQ8k D7VxDbO9md46aVBIwulXzar3ljtGRLyxRAxCV34Nuhq4TEUxcPUcqyJQZWBdomUovVD5/R gFc5z6508h6YWpWtinQmThQQ1CHAm+xgfdONLgPtVmH5LAbqX4QNRgeGvISzsQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1678143081; a=rsa-sha256; cv=none; b=IH+hFCrHYC7OTlfwK4xea2sKWaGuXWDunnfNIX37BoikF1WKUC9XKuDbjyhcFJMdqk4GXZ z45kghZmP4rCWqcDv/SGpcKERlQZlarTgDoACj8PCuWT3SbQJL+aYYgwLPt9cdc5LZUiFJ zAMHWKb0uzxPRe7Th+GMkBh3y27MYBFqXc1d/EBj9KwGKGHcBAuTYZsEB3auJzxIk5ppRJ QgbQS2P14b9zLDxNVmPijt/ul09Ih0G6frEQD0Cv1Arx+KQBitYYl8S2weJcs5xzUrRjf+ NTOUYNsdch3+7DacXyGAio+vQsdx93BTAz+u+wDBy7T3B2Pi+aqzmjAVSQqJEQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PVv2K2hD9zKm3 for ; Mon, 6 Mar 2023 22:51:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 326MpLDH058259 for ; Mon, 6 Mar 2023 22:51:21 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 326MpLlm058258 for jail@FreeBSD.org; Mon, 6 Mar 2023 22:51:21 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Mon, 06 Mar 2023 22:51:19 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: overwatch@lab.kyngin.net X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 kvs changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |overwatch@lab.kyngin.net --- Comment #26 from kvs --- Hello Everyone! I believe I have hit the same bug, though I believe my issue is specifically related to lagg/lacp. I can confirm this problem affects tap as well as ep= air interfaces on a bridge when attempting to send over a vlan interface that h= as a lagg parent. System Description: FreeBSD 13.1 w/ Chelsio T6225-SO-CR NIC, identified by = cc0 / cc1 (confirmed up and operational), host25 is the system name. Network is 10.20.20.0/24, gateway is 10.20.20.254 (mac: 02:11:22:33:44:55), host is assigned 10.20.20.5, epair0 is assigned to jail-10-20-20-6 (with matching I= P of 10.20.20.6 on epair0b). Switch is set to accept tagged frames only for vlan 2020. All mtu's 1500. When adding a vlan interface child of cc0 to the bridge, I do not have any trouble passing data over the lagg. host25# ifconfig cc0.2020 create up host25# ifconfig bridge2020 create up host25# ifconfig bridge2020 addm cc0.2020 host25# ifconfig bridge2020 addm epair0a host25# ifconfig bridge2020 inet 10.20.20.25/24 (pings from host -> gateway works fine) host25# ping 10.20.20.254 success! (pings from jail -> gateway also work) host25# jexec jail-10-20-20-6 sh jail-10-20-20-6# ping 10.20.20.254 success! (I now reset bridge2020 to use a lagg interface.) host25# ifconfig bridge2020 destroy host25# ifconfig cc0.2020 destroy host25# ifconfig lagg0 create laggproto lacp laggport cc0 laggport cc1 up host25# ifconfig lagg0.2020 create up host25# ifconfig bridge2020 create up host25# ifconfig bridge2020 addm lagg0.2020 addm epair0a host25# ifconfig bridge2020 inet 10.20.20.25/24 (pings from host -> gateway work fine) host25# ping 10.20.20.254 success! (pings from jail -> gateway timeout) host25# jexec jail-10-20-20-6 sh jail-10-20-20-6# ping 10.20.20.254 ping: sendto: Host is down (arp cache from jail appears to not include gateway mac) jail-10-20-20-6# arp -an ? (10.20.20.6) at 02:07:f0:80:de:0b on epair0b permanent [ethernet] ? (10.20.20.254) at (incomplete) on epair0b expired [ethernet] (I assign mac statically.) jail-10-20-20-6# arp -s 10.20.20.254 02:11:22:33:44:55 jail-10-20-20-6# arp -an ? (10.20.20.6) at 02:07:f0:80:de:0b on epair0b permanent [ethernet] ? (10.20.20.254) at 02:11:22:33:44:55 on epair0b permanent [ethernet] (attempt ping again after static arp assignment) jail-10-20-20-6# ping 10.20.20.254 success! What comes next is a reasonably big presumption on my part, so hopefully someone more educated on the topic kindly corrects me where I'm wrong. See= ing that the vlan interface of cc0.2020 works in the bridge when lagg0.2020 is removed/destroyed. I believe it's possible that the issue is related to arp responses being sent down one of the two lagg members and the host OS not b= eing aware of that. Although the reply does come inbound on one of the host OS interfaces, it doesn't propagate that down across the epair / tap. The VM/= Jail then never sees the arp reply, and keeps the arp as "(incomplete)" in it's cache. When using a single interface, or a lagg with only a single interfa= ce active, arp appears to work as expected. To help observe this, I did the following: 1) From host25, I watched epair0a, cc0, and cc1 using host25# tcpdump -e -vvv -XX -i [interface] 2) inside jail-10-20-20-6, I attempted to ping the gateway to generate the = arp traffic: ping -c 1 -t 1 -q 10.20.20.254 PING 10.20.20.254 (10.20.20.254): 56 data bytes --- 10.20.20.254 ping statistics --- 1 packets transmitted, 0 packets received, 100.0% packet loss 3) Results follow: # tcpdump -e -vvv -XX -i epair0a tcpdump: listening on epair0a, link-type EN10MB (Ethernet), capture size 26= 2144 bytes 01:43:54.768801 02:07:f0:80:de:0b (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.20.20.254 tell 10.20.20.6, length 28 0x0000: ffff ffff ffff 0207 f080 de0b 0806 0001=20 ................ 0x0010: 0800 0604 0001 0207 f080 de0b 0a14 1406=20 ................ 0x0020: 0000 0000 0000 0a14 14fe .......... 01:43:54.768936 02:07:f0:80:de:0b (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 56: Ethernet (len 6), IPv4 (len 4), Request who-has 10.20.20.254 tell 10.20.20.6, length 42 0x0000: ffff ffff ffff 0207 f080 de0b 0806 0001=20 ................ 0x0010: 0800 0604 0001 0207 f080 de0b 0a14 1406=20 ................ 0x0020: 0000 0000 0000 0a14 14fe 0000 0000 0000=20 ................ 0x0030: 0000 0000 0000 0000 ........ 01:43:54.768969 02:07:f0:80:de:0b (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Request who-has 10.20.20.254 tell 10.20.20.6, length 46 0x0000: ffff ffff ffff 0207 f080 de0b 0806 0001=20 ................ 0x0010: 0800 0604 0001 0207 f080 de0b 0a14 1406=20 ................ 0x0020: 0000 0000 0000 0a14 14fe 0000 0000 0000=20 ................ 0x0030: 0000 0000 0000 0000 0000 0000 .........= ... # tcpdump -e -vvv -XX -i cc0 tcpdump: listening on cc0, link-type EN10MB (Ethernet), capture size 262144 bytes 01:43:54.768822 02:07:f0:80:de:0b (oui Unknown) > Broadcast, ethertype 802.= 1Q (0x8100), length 46: vlan 2020, p 0, ethertype ARP, Ethernet (len 6), IPv4 = (len 4), Request who-has 10.20.20.254 tell 10.20.20.6, length 28 0x0000: ffff ffff ffff 0207 f080 de0b 8100 07e4 ................ 0x0010: 0806 0001 0800 0604 0001 0207 f080 de0b ................ 0x0020: 0a14 1406 0000 0000 0000 0a14 14fe .............. 01:43:54.769126 02:11:22:33:44:55 (oui Unknown) > 02:07:f0:80:de:0b (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 2020, p 0, ethertype A= RP, Ethernet (len 6), IPv4 (len 4), Reply 10.20.20.254 is-at 02:11:22:33:44:55 = (oui Unknown), length 46 0x0000: 0207 f080 de0b 0211 2233 4455 8100 07e4 ........"3DU.... 0x0010: 0806 0001 0800 0604 0002 0211 2233 4455 ............"3DU 0x0020: 0a14 14fe 0207 f080 de0b 0a14 1406 0000 ................ 0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 01:43:54.769171 02:11:22:33:44:55 (oui Unknown) > 02:07:f0:80:de:0b (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 2020, p 0, ethertype A= RP, Ethernet (len 6), IPv4 (len 4), Reply 10.20.20.254 is-at 02:11:22:33:44:55 = (oui Unknown), length 46 0x0000: 0207 f080 de0b 0211 2233 4455 8100 07e4 ........"3DU.... 0x0010: 0806 0001 0800 0604 0002 0211 2233 4455 ............"3DU 0x0020: 0a14 14fe 0207 f080 de0b 0a14 1406 0000 ................ 0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 01:43:54.769221 02:11:22:33:44:55 (oui Unknown) > 02:07:f0:80:de:0b (oui Unknown), ethertype 802.1Q (0x8100), length 64: vlan 2020, p 0, ethertype A= RP, Ethernet (len 6), IPv4 (len 4), Reply 10.20.20.254 is-at 02:11:22:33:44:55 = (oui Unknown), length 46 0x0000: 0207 f080 de0b 0211 2233 4455 8100 07e4 ........"3DU.... 0x0010: 0806 0001 0800 0604 0002 0211 2233 4455 ............"3DU 0x0020: 0a14 14fe 0207 f080 de0b 0a14 1406 0000 ................ 0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ # tcpdump -e -vvv -XX -i cc1 tcpdump: listening on cc1, link-type EN10MB (Ethernet), capture size 262144 bytes 01:43:54.768876 02:07:f0:80:de:0b (oui Unknown) > Broadcast, ethertype 802.= 1Q (0x8100), length 60: vlan 2020, p 0, ethertype ARP, Ethernet (len 6), IPv4 = (len 4), Request who-has 10.20.20.254 tell 10.20.20.6, length 42 0x0000: ffff ffff ffff 0207 f080 de0b 8100 07e4 ................ 0x0010: 0806 0001 0800 0604 0001 0207 f080 de0b ................ 0x0020: 0a14 1406 0000 0000 0000 0a14 14fe 0000 ................ 0x0030: 0000 0000 0000 0000 0000 0000 ............ 01:43:54.768965 02:07:f0:80:de:0b (oui Unknown) > Broadcast, ethertype 802.= 1Q (0x8100), length 64: vlan 2020, p 0, ethertype ARP, Ethernet (len 6), IPv4 = (len 4), Request who-has 10.20.20.254 tell 10.20.20.6, length 46 0x0000: ffff ffff ffff 0207 f080 de0b 8100 07e4 ................ 0x0010: 0806 0001 0800 0604 0001 0207 f080 de0b ................ 0x0020: 0a14 1406 0000 0000 0000 0a14 14fe 0000 ................ 0x0030: 0000 0000 0000 0000 0000 0000 0000 0000 ................ Apparently 1 arp request is sent over cc0, and 2 over cc1, all 3 replies co= me back over cc0. None of them appear to enter epair0a. I've not had any luck changing lagg hashes at this stage to try to force requests down one of the= two lagg members, so instead I downed one of the interfaces in the lagg. (bridge2020 is still up with epair0a and lagg0.2020 (lagg0 contains cc0+cc1 both up)) jail-10-20-20-6# ping 10.20.20.254 ping: sendto: Host is down host25# ifconfig cc1 down (confirm arp cache is empty in jail) jail-10-20-20-6# arp -da jail-10-20-20-6# ping 10.20.20.254 success! (using tcpdump, epair0a now sees the arp replies as well (I excluded the tcpdump for cc0 here because it's largely identical)) # tcpdump -e -vvv -XX -i epair0a 15:23:10.623560 02:07:f0:80:de:0b (oui Unknown) > Broadcast, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.20.20.254 tell 10.20.20.6, length 28 0x0000: 0001 0800 0604 0001 0207 f080 de0b 0a14 ................ 0x0010: 1406 0000 0000 0000 0a14 14fe ............ 15:23:10.623916 02:11:22:33:44:55 (oui Unknown) > 02:07:f0:80:de:0b (oui Unknown), ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknown), length 46 0x0000: 0001 0800 0604 0002 0211 2233 4455 0a14 .........."3DU.. 0x0010: 14fe 0207 f080 de0b 0a14 1406 0000 0000 ................ 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 15:23:10.623924 02:11:22:33:44:55 (oui Unknown) > 02:07:f0:80:de:0b (oui Unknown), ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknown), length 46 0x0000: 0001 0800 0604 0002 0211 2233 4455 0a14 .........."3DU.. 0x0010: 14fe 0207 f080 de0b 0a14 1406 0000 0000 ................ 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 15:23:10.623926 02:11:22:33:44:55 (oui Unknown) > 02:07:f0:80:de:0b (oui Unknown), ethertype ARP (0x0806), length 60: Ethernet (len 6), IPv4 (len 4), Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknown), length 46 0x0000: 0001 0800 0604 0002 0211 2233 4455 0a14 .........."3DU.. 0x0010: 14fe 0207 f080 de0b 0a14 1406 0000 0000 ................ 0x0020: 0000 0000 0000 0000 0000 0000 0000 .............. 15:23:10.623943 02:07:f0:80:de:0b (oui Unknown) > 02:11:22:33:44:55 (oui Unknown), ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 56841, offset 0, flags [none], proto ICMP (1), length 84) 10.20.20.6 > 10.20.20.254: ICMP echo request, id 22927, seq 0, leng= th 64 0x0000: 4500 0054 de09 0000 4001 5f74 0a14 1406 E..T....@._t.... 0x0010: 0a14 14fe 0800 8750 598f 0000 0006 2ec0 .......PY....... 0x0020: 15c1 e795 0809 0a0b 0c0d 0e0f 1011 1213 ................ 0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"# 0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123 0x0050: 3435 3637 4567 15:23:10.624147 02:11:22:33:44:55 (oui Unknown) > 02:07:f0:80:de:0b (oui Unknown), ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 54016, offset 0, flags [none], proto ICMP (1), length 84) 10.20.20.254 > 10.20.20.6: ICMP echo reply, id 22927, seq 0, length= 64 0x0000: 4500 0054 d300 0000 4001 6a7d 0a14 14fe E..T....@.j}.... 0x0010: 0a14 1406 0000 8f50 598f 0000 0006 2ec0 .......PY....... 0x0020: 15c1 e795 0809 0a0b 0c0d 0e0f 1011 1213 ................ 0x0030: 1415 1617 1819 1a1b 1c1d 1e1f 2021 2223 .............!"# 0x0040: 2425 2627 2829 2a2b 2c2d 2e2f 3031 3233 $%&'()*+,-./0123 0x0050: 3435 3637 4567 (arp cache seems valid as well) jail-10-20-20-6# arp -na ? (10.20.20.6) at 02:07:f0:80:de:0b on epair0b permanent [ethernet] ? (10.20.20.254) at 02:11:22:33:44:55 on epair0b expires in 1085 seconds [ethernet] Additional thoughts: 1) With lagg0, cc0, and cc1 up, I created a second jail on host25 using 10.20.20.7 (epair1). I add epair1a to bridge2020 (now including epair0a, epair1a and lagg0.2020). When I attempt to ping from jail-10-20-20-6 to .254 I get a timeout as previously experienced. Pinging from .6 to .7 appears to work without any trouble, if lagg0 has any cc0/1 members up or down. This was expected, as packets should never trave= rse lagg0.2020, but I did want to test/confirm. 2) I did run some ping tests with untagged lagg0 in the bridge, and it does appear it's working without trouble. I removed lagg0.2020 from bridge2020, then added lagg0 to bridge2020, and set the switch ports as untagged in the switch. The packets appear to move without trouble even with both cc0+cc1 = up.=20 I need to further test this to be conclusive, but this felt less important = to perform at this time as it doesn't solve the requirement I need of tagged ports. 3) I have a few bhyve vm's that I've added as tests, tap0, tap1, etc to the bridge2020. The results seem to be largely consistent with jails. You cou= ld replace jail-10-20-20-6, with vm-10-20-20-11 (tested freebsd / openbsd / windows) for instance, and these same results appear. Packets fail when originating from tap/vnet and traversing lagg0.2020. (again, lagg0/lacp is up, includes cc0+cc1, bridge2020 includes lagg0.2020, tap0, and epair0a devices) host25# ping 10.20.20.254 success! vm-10-20-20-11# arp -da (attempt traverse lagg0.2020) vm-10-20-20-11# ping 10.20.20.254 ping: sendto: Host is down (try tap0 -> epair0) vm-10-20-20-11# ping 10.20.20.6 success! (try tests again with lagg0 member cc1 down) host25# cc1 down (tap0 -> lagg0.2020 -> 10.20.20.254) vm-10-20-20-11# ping 10.20.20.254 success! (again tap0 -> epair0, works as expected) vm-10-20-20-11# ping 10.20.20.6 success! (turn cc1 back up, wait about 10 seconds for both laggports to be distribut= ing) host25# cc1 up vm-10-20-20-11# arp -da vm-10-20-20-11# ping 10.20.20.254 ping: sendto: Host is down (again, only lagg is preventing arp, tap <-> epair in bridge still works fi= ne) vm-10-20-20-11# ping 10.20.20.6 success! jail-10-20-20-6# ping 10.20.20.11 success! Conclusion: When bridging a vnet/tap interface with a lagg.vlan interface (= vlan interface with lagg [laggproto lacp] parent) arp replies do not enter the vnet/tap interface on the bridge when *both* lagg members are up. By downi= ng one of the two interfaces in the lagg group, arp replies enter the vnet/tap interface as expected. Final notes: I've not included it in this post, but I've attempted to remove all the hardware offloading features from the interfaces lagg0/lagg0.2020/cc0/cc1 as well as toggled lagg0 lagghash, toggled sysctls net.link.lagg.* and net.link.bridge.*, as well as upgraded to 13-STABLE. No luck moving data o= ver the lagg until I down one of the two lagg0 interfaces. For brevity, I used= the command 'ping host-ip' in the examples above, and only displayed a simple response of success/fail. In testing I mostly performed pings for reasonab= ly long periods (ex: -c 10 -t 2), to confirm the above examples. I'd be happy to help test further if anyone has any suggestions. Thank you! -kvs --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Tue Mar 21 23:41:08 2023 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Ph7Qs3nGyz410Hv for ; Tue, 21 Mar 2023 23:41:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ph7Qs2bkNz4M0k for ; Tue, 21 Mar 2023 23:41:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679442069; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DqyNwGkJAO9qPnm6h88YI71vlxBLGHGMTOfvxGpDfIM=; b=MKOd0H/SNC6DOXwcAqvbKayS5cyuYN2tN0e32OWDFct7ZBHKO1ZI+oXvzqalWBOqfrS7zd xaGq1wlGvV/EFjWYqm7r/52hx0fDbanzs+/WJBfoV4sPjAgt/WS+6kuYsxnxTySAlZZQ11 zjFW7ptKMY/UyWWEgB5AeBoTwfxmXbl5q51fXSCyNpsApNYXilae5acISvQ5VS8GiBmxq0 TiGkX6P4tQcCiYOpxgawl/ocGDyNPCCzaL67T6Lgzn6cjaz4auOlw397ZJ4WU8DtfnAe9n QhU1zE0a80KYTzsBtsD1c96bc8TCoHLH3AGXpwXDowUeUUj2J3QdET6lr01gIw== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679442069; a=rsa-sha256; cv=none; b=Ygjga93iIRsDlCgcCUaEEIjwBFrlT4evCDcXuVtofuf1FNSVLtk/MXKB6RoJzS1wGXAZjS CcMk5HoI0kB9nsHpJD5uN/olisKqgox+R+xbm+1MIimgPV1dtHokm7BbOQncWKTDJ5OGR8 BqhvyOzNStQ2tSam8dY281fZWTREVT4+c28Vk533eqvMls9HMLM7TH5b+CXMLJj7+j4IAB 74VLtK+U2DHRvysPemooyVCFJ6/TIgLMTb8OFLW0Yr0eHu4FXWkiPKeewwRHZFfxD9/9mr AzSRjQ+EkFF67rCxiNsnml07EFIFR5DfH5rK+VFLu7WVgq/k0TDUfVNI7+hApg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4Ph7Qs1gY0zXWj for ; Tue, 21 Mar 2023 23:41:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 32LNf9BD090008 for ; Tue, 21 Mar 2023 23:41:09 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 32LNf9H8090006 for jail@FreeBSD.org; Tue, 21 Mar 2023 23:41:09 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Tue, 21 Mar 2023 23:41:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: vermaden@interia.pl X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 Slawomir Wojciech Wojtczak changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |vermaden@interia.pl --- Comment #27 from Slawomir Wojciech Wojtczak --- Hi, any progress on this one? Will it be fixed in 13.2-RELEASE or the little later upcoming 14.0-RELEASE? I ask because my buddy just hit it again with 13.1-RELEASE today ... Regards, vermaden --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 22 03:23:58 2023 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhDN444lQz41CHg for ; Wed, 22 Mar 2023 03:24:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhDN431YKz3hfm for ; Wed, 22 Mar 2023 03:24:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679455444; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dKgZ9Zquwwtxgjk4IoudgzJ6V3RjuhgtdTjQ1BP5hcs=; b=MfV1Rib11FgeS4hNk9gVS+I5cj79RituRt+11vL/23V5NgPJbRvk34EmAfwSPltjUbogYq Ptg9ksLVfehIpN05nFeZTEd35ALbjw/YPkPk6KCaMxTWsxTdOEsbJ3J3NIjMPecA97gGPZ tp0bKMjJi4+e3pRFrdCIEmaSk6dMPzmqM1dX6+b7jrgFZOlWMv2+o9F/R+9k3qWvNzsZRw rAisZ/t23u0no1ReFAleRaMZicq9ouxMKkdJd0nJhkfA+AJlXp6whOReAjMjbdcYzD52mk 4Th/Xt53+xeBanLzje/hh1l+BjAfpoP9BLI4LGy9fq+PatYI6dMTOXikBWBqhA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679455444; a=rsa-sha256; cv=none; b=hDMhGL3PZD2LvQO+uL/c/bMG19cmMN3L/wLiuMTwc/lPQGgqhuVqj7RUM21YxL7IHDHzKs czg+Zc7CH2a2q8MlxeSq+thbttXCCCudaMdv38bo9hwPmg0emx/V6BoPipiE4sOMRP9DJC KjefzIoHP26UtCD2EsomSNnmFUUi3k3GeGBgJeoX9WFOGijj12AI8s21tzSqs8bMP0SIq7 rhFXATKmtyjdaNOkPjJH/plM9A2c0e9fxrJHBVl8CC263yaWponngjZbcOnvY5SGPFitgv L3smeVbx6y5K0dSppVmLndM9/7U7erhoTq8WsPM0HSu96gfvKZOSX8GEihPRAw== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PhDN424nYzdDh for ; Wed, 22 Mar 2023 03:24:04 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 32M3O4R3023437 for ; Wed, 22 Mar 2023 03:24:04 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 32M3O4cN023436 for jail@FreeBSD.org; Wed, 22 Mar 2023 03:24:04 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Wed, 22 Mar 2023 03:23:58 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: overwatch@lab.kyngin.net X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 --- Comment #28 from kvs --- (In reply to Slawomir Wojciech Wojtczak from comment #27) I have some headway on my end, though I don't know how much it's related to= the earlier bugs at this point. After further testing, vlans apparently aren't related to my problem. The problem occurs on lagg without vlan interfaces.=20=20 When a jail+VNET (on bridge) sends an ARP request it traverses the bridge a= nd exits both interfaces in the host lagg group. When the ARP reply comes bac= k, it appears it will only ever enter the host bridge if it comes in on the primary lagg member. I'm not certain this is exclusive to vnets, also poss= ibly this is normal operation for laggs using lacp? Lab test: lagg0 (ports cc0 + cc1), bridge2020 (members epair0a & lagg0) ping from jail+VNET to switch (10.20.20.254), using source epair0b (10.20.20.77) (epair0b -> epair0a -> bridge2020 -> lagg0 -> cc0/cc1 -> switch) tcpdump -i epair0a 10:00:17.981011 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length = 28 tcpdump -i bridge2020 10:00:17.981051 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length = 28 tcpdump -i lagg0 10:00:17.981030 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length = 28 10:00:17.981282 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknow= n), length 46 tcpdump -i cc0: 10:00:17.981050 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length = 42 tcpdump -i cc1: 10:00:17.981041 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length = 28 10:00:17.981282 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknow= n), length 46 Arp table is not populated on VM, as bridge2020 and epair0a/b never sees ARP reply come in over cc1. I believe in my case specifically the switch is se= eing cc1 as the primary lagg member while the FreeBSD server sees cc0 as the pri= mary lagg member. When ARP replies manage to come in over cc0, the ARP replies make it to the vnet interface and the jail populates its ARP table. I can force this even= t by downing cc1 or shutting down the cc1 switch port (in both cases it appears = the switch then identifies cc0 as the primary lagg member over which it sends A= RP replies). Alternatively, if both cc0 and cc1 are up, and the switch sends = an ARP reply over cc0 (has happened randomly), the ARP reply does makes it thr= ough the bridge/epair and populates the ARP cache on the VM. Example after ifconfig cc1 down: tcpdump -i epair0a 10:48:18.949695 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length = 28 10:48:18.950041 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknow= n), length 46 tcpdump -i bridge2020 10:48:18.949731 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length = 28 10:48:18.950041 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknow= n), length 46 tcpdump -i lagg0 10:48:18.949711 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length = 28 10:48:18.950041 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknow= n), length 46 tcpdump -i cc0 10:48:18.949722 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, length = 28 10:48:18.950041 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unknow= n), length 46 ARP table on VM is now populated with switch address, and everything appear= s to work as normal over lagg0 (with cc0 up / cc1 down).=20=20 In the mean time I've managed to get the switch configured to send L2/ARP o= ver both lagg members which has fixed the immediate problem. Though I do think it's strange that FreeBSD populates the ARP table just fine on the host over cc1, but just wont send that ARP reply over the bridge interface unless it comes in on cc0. That *feels* like a bug, as it only seems to affect the second interface on a lagg that's in a bridge, and quite possibly only for layer 2 (L2/3 needs further testing - I've not lost packets once the arp ta= ble is populated, but it's possible the switch was handling layer 3 differently= and always using the cc0 port, in which case FreeBSD would probably send over t= he bridge without trouble). Testing has been performed on 14-CURRENT and 13-STABLE with identical resul= ts. --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Wed Mar 22 09:44:08 2023 X-Original-To: jail@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4PhNpj36Q3z40Mfs for ; Wed, 22 Mar 2023 09:44:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4PhNpj1zDRz4Gtk for ; Wed, 22 Mar 2023 09:44:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1679478253; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YGvynQvy8rsiC4Nz+2zTRGLfQtXC8mCfrE0d/LXezSE=; b=Ns09u5dtp5J5HzFeHE7zzIT6R5laWSp0vP9iGK+ipvWTP2R5YTn02xgjxSNDwGON/jZMze b7fZK7gZwIEmWZ8bmCYzWm2AGhKHqJdwDxLMadFXdWIt+0a7qqQtCC5gIee5ZMIoAl/c3W 9bwhu0Vhah5kzBNKO5nCZv4ZyoMteOwvdqrzol+sJkg23liEJ4fnSjhcYeduveZTS5xgwL ZXNm3g1y1hXPmR47hiBS7EzpAfwOlDLEwSFiR/4yJ52tJHebuklpLx2Q1ye3EEwtsqTcYn OFxjgrqIwkv/dVIgKVYGYCX8Ecr62jRXixWX7Zad2J4yKJcr4NeRXnfwftlvhA== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1679478253; a=rsa-sha256; cv=none; b=rN3+GfQShZQUnCcAWfe4DrTHdw3wYdzpsLli/m6e8N1CKR9i6oZtGAw7sM00UtXBOl/v4J SnlnAQ2so4exgre4UcXChhkdn7Ii23bNWv1qz74oyB3+KT6/tas0th9T+KKwXoM69SLsx6 uxcYeI/ZbdeiC7IxTx41pcoNwxDfvUZUJLIQQcFhBnhQKzUNKJ2cjBp5dozBh32zwPQ334 Q80LJqt2pB2zOhO/CUqVBfg2YxC+JPB3Si8h2As5kBeXB9QzsK9SqKRU3TcEpt/1cnhJdV nzS9kX4OBebtOM4ixtgXpfOweTMVLwTVp/OL82RQ17teHV4XzataiqspQ3z9ww== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4PhNpj13KFzpfl for ; Wed, 22 Mar 2023 09:44:13 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 32M9iDND071833 for ; Wed, 22 Mar 2023 09:44:13 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 32M9iD6F071832 for jail@FreeBSD.org; Wed, 22 Mar 2023 09:44:13 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: jail@FreeBSD.org Subject: [Bug 240106] VNET issue with ARP and routing sockets in jails Date: Wed, 22 Mar 2023 09:44:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: zlei@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion about FreeBSD jail(8) List-Archive: https://lists.freebsd.org/archives/freebsd-jail List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-jail@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D240106 --- Comment #29 from Zhenlei Huang --- (In reply to kvs from comment #28) I think your should open a separate PR, as you have different setup with th= at of the original PR by John Westbrook. He has SR-IOV configured. I managed to repeat with cxl / lagg / bridge / epair (vnet) on 13.2-RC3. Al= so tried re / ue . > tcpdump -i cc0: > 10:00:17.981050 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, lengt= h 42 > tcpdump -i cc1: > 10:00:17.981041 ARP, Request who-has 10.20.20.254 tell 10.20.20.77, lengt= h 28 > 10:00:17.981282 ARP, Reply 10.20.20.254 is-at 02:11:22:33:44:55 (oui Unkn= own), length 46 You might want to tcpdump on cc0 with `--direction=3Din` to filter ARP requ= est send out from cc1 and then come back to cc0 (the switch forwarded it). The IF_BRIDGE(4) seems to hide some thing to protect itself get confused. If you can confirm, then please config you switch properly. The two ports c= c0 and cc1 connected should be in same link aggregation group. I'll see if I can teach IF_BRIDGE(4) to emit warnings in case it get ARP request packet sent from it self. --=20 You are receiving this mail because: You are the assignee for the bug.=