From owner-freebsd-stable@freebsd.org Sun May 31 16:07:51 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 96C85338070 for ; Sun, 31 May 2020 16:07:51 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from elf.hq.norma.perm.ru (mail.norma.perm.ru [128.127.146.8]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.norma.perm.ru", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 49ZjrN6Q4nz4KT1 for ; Sun, 31 May 2020 16:07:48 +0000 (UTC) (envelope-from emz@norma.perm.ru) Received: from [192.168.243.9] ([192.168.243.9]) by elf.hq.norma.perm.ru (8.15.2/8.15.2) with ESMTP id 04VG9aPv014193 for ; Sun, 31 May 2020 21:09:36 +0500 (+05) (envelope-from emz@norma.perm.ru) To: freebsd-stable@freebsd.org From: "Eugene M. Zheganin" Subject: CARP under Hyper-V: weird things happen Message-ID: Date: Sun, 31 May 2020 21:07:37 +0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.8.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-Rspamd-Queue-Id: 49ZjrN6Q4nz4KT1 X-Spamd-Bar: ++++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=perm.ru; spf=pass (mx1.freebsd.org: domain of emz@norma.perm.ru designates 128.127.146.8 as permitted sender) smtp.mailfrom=emz@norma.perm.ru X-Spamd-Result: default: False [6.35 / 15.00]; ARC_NA(0.00)[]; NEURAL_SPAM_SHORT(1.02)[1.024]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(0.00)[+a]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; NEURAL_SPAM_MEDIUM(1.04)[1.036]; RCPT_COUNT_ONE(0.00)[1]; BAD_REP_POLICIES(0.10)[]; HFILTER_HELO_IP_A(1.00)[elf.hq.norma.perm.ru]; RBL_VIRUSFREE_BOTNET(2.00)[128.127.146.8:from]; DMARC_POLICY_ALLOW(0.00)[perm.ru,none]; HFILTER_HELO_NORES_A_OR_MX(0.30)[elf.hq.norma.perm.ru]; NEURAL_SPAM_LONG(0.99)[0.991]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:57401, ipnet:128.127.146.0/24, country:RU]; MIME_TRACE(0.00)[0:+]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-Spam: Yes X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 May 2020 16:07:51 -0000 Hello, I'm Running 12.0-REL in a VM under W2016S with CARP enabled and paired to a baremetal FreeBSD server. All of a sudden I realized that thjis machine is unable to become a CARP MASTER - because it sees it's own ACRP announces, but instead of seeing them from a CARP synthetic MAC address only, it sees additional extra packets with several MACs derived from the original one (I'm well awared about the -MacAddressSpoof on SetVmNetworkAdapterVlan switch, and it's running with this thingg on, but still). These packets always almost (but not 100%) accompany each valid CARP advertisement. Say, we have a CARP-enabled interface: vlan2: flags=8943 metric 0 mtu 1500         description: AS WAN         options=80000         ether 00:15:5d:0a:79:12         inet 91.206.242.9/28 broadcast 91.206.242.15         inet 91.206.242.12/28 broadcast 91.206.242.15 vhid 3         groups: vlan         carp: BACKUP vhid 3 advbase 1 advskew 250         vlan: 2 vlanpcp: 0 parent interface: hn1         media: Ethernet autoselect (10Gbase-T )         status: active         nd6 options=29 Notice the MAC and now look at this: ===Cut=== [root@gw1:~]# tcpdump -T carp -nepi vlan2 carp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vlan2, link-type EN10MB (Ethernet), capture size 262144 bytes 20:45:54.152619 00:00:5e:00:01:03 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 ^^^ this is the ordinary and valid CARP advertisement, notice the synthetic MAC which is requiring setting mac address spoofing. 20:45:54.152880 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 ^^^ this is some insanity happening 20:45:54.153234 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 ^^^ and again 20:45:54.153401 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227035 ^^^ and again 20:45:57.562470 00:00:5e:00:01:03 > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 ^^^ valid CARP advertisement, next one-second advbase cycle 20:45:57.562874 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 ^^^ more insane stuff, notice the NEW (sic !) MAC-address 20:45:57.562955 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 20:45:57.562989 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: vhid=3 advbase=1 advskew=100 authlen=7 counter=13769798250643227036 ^C 8 packets captured 3195 packets received by filter ===Cut=== Does anyone has, by any chance, some idea about what's happening ? As soon as I stop CARP stack on this VM these "mad" MACs aren't received anymore, so I'm pretty confident these are somehow procuced on the Hyper-V side. Another weird this is that vlan1  is refusing to work (seems like packets are never received on the VM side) unless its configured on another adapter in the -Untagged (once again powershell term for SetVmNetworkAdapterVlan). Thanks. Eugene.