From owner-freebsd-stable@freebsd.org Wed Jun 10 09:46:56 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4B72032F03F for ; Wed, 10 Jun 2020 09:46:56 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (zm1.digsys.bg [193.68.21.128]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp-sofia.digsys.bg", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49hhwH1hxDz3T3l for ; Wed, 10 Jun 2020 09:46:54 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from [193.68.6.100] ([193.68.6.100]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.15.2/8.15.2) with ESMTPSA id 05A9kh6Z079897 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 10 Jun 2020 12:46:44 +0300 (EEST) (envelope-from daniel@digsys.bg) From: Daniel Kalchev Message-Id: <717DD022-DB2E-4DAD-8504-09B67CE344A4@digsys.bg> Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: CARP under Hyper-V: weird things happen Date: Wed, 10 Jun 2020 12:46:43 +0300 In-Reply-To: Cc: freebsd-stable@freebsd.org To: "Eugene M. Zheganin" References: X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Rspamd-Queue-Id: 49hhwH1hxDz3T3l X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of daniel@digsys.bg designates 193.68.21.128 as permitted sender) smtp.mailfrom=daniel@digsys.bg X-Spamd-Result: default: False [-2.10 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:193.68.21.128]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[digsys.bg]; NEURAL_HAM_LONG(-1.00)[-1.005]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.36)[-0.358]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.94)[-0.938]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:3245, ipnet:193.68.0.0/19, country:BG]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Jun 2020 09:46:56 -0000 Hi Eugene, Might it be the Hyper-V doesn=E2=80=99t properly implement multicast? Or = there is perhaps some setting in there to let it work. =46rom memory = CARP is not trivial on vmware as well, unless you make special settings. = Some ideas here: = https://docs.netgate.com/pfsense/en/latest/highavailability/troubleshootin= g-high-availability-clusters.html#hypervisor-users-especially-vmware-esx-e= sxi = Daniel > On 31 May 2020, at 19:07, Eugene M. Zheganin = wrote: >=20 > Hello, >=20 > I'm Running 12.0-REL in a VM under W2016S with CARP enabled and paired = to a baremetal FreeBSD server. >=20 > All of a sudden I realized that thjis machine is unable to become a = CARP MASTER - because it sees it's own ACRP announces, but instead of = seeing them from a CARP synthetic MAC address only, it sees additional = extra packets with several MACs derived from the original one (I'm well = awared about the -MacAddressSpoof on SetVmNetworkAdapterVlan switch, and = it's running with this thingg on, but still). These packets always = almost (but not 100%) accompany each valid CARP advertisement. >=20 > Say, we have a CARP-enabled interface: >=20 > vlan2: flags=3D8943 = metric 0 mtu 1500 > description: AS WAN > options=3D80000 > ether 00:15:5d:0a:79:12 > inet 91.206.242.9/28 broadcast 91.206.242.15 > inet 91.206.242.12/28 broadcast 91.206.242.15 vhid 3 > groups: vlan > carp: BACKUP vhid 3 advbase 1 advskew 250 > vlan: 2 vlanpcp: 0 parent interface: hn1 > media: Ethernet autoselect (10Gbase-T ) > status: active > nd6 options=3D29 >=20 > Notice the MAC and now look at this: >=20 > =3D=3D=3DCut=3D=3D=3D >=20 > [root@gw1:~]# tcpdump -T carp -nepi vlan2 carp > tcpdump: verbose output suppressed, use -v or -vv for full protocol = decode > listening on vlan2, link-type EN10MB (Ethernet), capture size 262144 = bytes > 20:45:54.152619 00:00:5e:00:01:03 > 01:00:5e:00:00:12, ethertype IPv4 = (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: = vhid=3D3 advbase=3D1 advskew=3D100 authlen=3D7 = counter=3D13769798250643227035 >=20 > ^^^ this is the ordinary and valid CARP advertisement, notice the = synthetic MAC which is requiring setting mac address spoofing. >=20 > 20:45:54.152880 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 = (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: = vhid=3D3 advbase=3D1 advskew=3D100 authlen=3D7 = counter=3D13769798250643227035 >=20 > ^^^ this is some insanity happening >=20 > 20:45:54.153234 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 = (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: = vhid=3D3 advbase=3D1 advskew=3D100 authlen=3D7 = counter=3D13769798250643227035 >=20 > ^^^ and again >=20 > 20:45:54.153401 9c:8e:99:0f:79:42 > 01:00:5e:00:00:12, ethertype IPv4 = (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: = vhid=3D3 advbase=3D1 advskew=3D100 authlen=3D7 = counter=3D13769798250643227035 >=20 > ^^^ and again >=20 > 20:45:57.562470 00:00:5e:00:01:03 > 01:00:5e:00:00:12, ethertype IPv4 = (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: = vhid=3D3 advbase=3D1 advskew=3D100 authlen=3D7 = counter=3D13769798250643227036 >=20 > ^^^ valid CARP advertisement, next one-second advbase cycle >=20 > 20:45:57.562874 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 = (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: = vhid=3D3 advbase=3D1 advskew=3D100 authlen=3D7 = counter=3D13769798250643227036 >=20 > ^^^ more insane stuff, notice the NEW (sic !) MAC-address >=20 > 20:45:57.562955 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 = (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: = vhid=3D3 advbase=3D1 advskew=3D100 authlen=3D7 = counter=3D13769798250643227036 > 20:45:57.562989 9c:8e:99:0f:79:3c > 01:00:5e:00:00:12, ethertype IPv4 = (0x0800), length 70: 91.206.242.9 > 224.0.0.18: CARPv2-advertise 36: = vhid=3D3 advbase=3D1 advskew=3D100 authlen=3D7 = counter=3D13769798250643227036 > ^C > 8 packets captured > 3195 packets received by filter >=20 > =3D=3D=3DCut=3D=3D=3D >=20 >=20 > Does anyone has, by any chance, some idea about what's happening ? As = soon as I stop CARP stack on this VM these "mad" MACs aren't received = anymore, so I'm pretty confident these are somehow procuced on the = Hyper-V side. >=20 > Another weird this is that vlan1 is refusing to work (seems like = packets are never received on the VM side) unless its configured on = another adapter in the -Untagged (once again powershell term for = SetVmNetworkAdapterVlan). >=20 >=20 > Thanks. >=20 > Eugene. >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org"