From owner-freebsd-stable@freebsd.org Wed Jan 16 14:14:36 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E6EF1489992 for ; Wed, 16 Jan 2019 14:14:36 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from constantine.ingresso.co.uk (unknown [IPv6:2a02:b90:3002:411::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 171636F5CA for ; Wed, 16 Jan 2019 14:14:35 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from [2a02:b90:3002:411::6] (helo=dilbert.ingresso.co.uk) by constantine.ingresso.co.uk with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1gjlxg-0004w5-Rb for freebsd-stable@freebsd.org; Wed, 16 Jan 2019 14:14:32 +0000 Received: from petefrench by dilbert.ingresso.co.uk with local (Exim 4.91 (FreeBSD)) (envelope-from ) id 1gjlxg-000DSh-Oi for freebsd-stable@freebsd.org; Wed, 16 Jan 2019 14:14:32 +0000 To: freebsd-stable@freebsd.org Subject: CARP stopped working after upgrade from 11 to 12 Message-Id: From: Pete French Date: Wed, 16 Jan 2019 14:14:32 +0000 X-Rspamd-Queue-Id: 171636F5CA X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=ingresso.co.uk; spf=pass (mx1.freebsd.org: domain of petefrench@ingresso.co.uk designates 2a02:b90:3002:411::3 as permitted sender) smtp.mailfrom=petefrench@ingresso.co.uk X-Spamd-Result: default: False [2.74 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RDNS_NONE(1.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a02:b90:3002:411::3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-0.29)[-0.290,0]; NEURAL_SPAM_MEDIUM(0.16)[0.158,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_SHORT(0.20)[0.200,0]; MX_GOOD(-0.01)[ingresso-co-uk.mail.protection.outlook.com,ingresso-co-uk.mail.protection.outlook.com]; DMARC_POLICY_ALLOW(-0.50)[ingresso.co.uk,none]; IP_SCORE(-0.02)[country: GB(-0.09)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16082, ipnet:2a02:b90::/32, country:GB]; HFILTER_HOSTNAME_UNKNOWN(2.50)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2019 14:14:36 -0000 I just upgraded my pair of firewalls from 11 to 12, and am now in the situation where CARP no longer works between them to faiilover the virtual addresse. Both machines come up thinking that they are the master. If I manually set the advskew on the interfaces to a high number on what should be passive then it briefly goes to backup mode, but then goes back to master with the message: BACKUP -> MASTER (preempting a slower master) This is kind of a big problem! Its also unexpected as I tested CARP on 12 in my development environment and it works here - though here we only have one address insetad of several. But this has worked fine for a very long time until now. The setup looks like this: ifconfig_em0="inet 10.32.10.1/16" ifconfig_em0_ipv6="inet6 2a02:1658:1:2:e550::1/64" ifconfig_em0_alias0="inet 10.32.10.6/16 vhid 10 advskew 10 pass redacted" ifconfig_em0_alias1="inet6 2a02:1658:1:2:e550::6/64 vhid 30 advskew 10 pass redacted" ifconfig_em1="inet 178.250.73.196/26" ifconfig_em1_ipv6="inet6 2a02:1658:1:1::1:2/64" ifconfig_em1_alias0="inet 178.250.73.198/26 vhid 20 advskew 10 pass redacted" ifconfig_em1_alias1="inet6 2a02:1658:1:1::1:1/64 vhid 40 advskew 10 pass redacted" ifconfig_em1_alias2="inet 178.250.73.199/26 vhid 20 advskew 10 pass redacted" ifconfig_em1_alias3="inet 178.250.73.200/26 vhid 20 advskew 10 pass redacted" ifconfig_em1_alias4="inet 178.250.73.221/26 vhid 20 advskew 10 pass redacted" ...and on the passive side almost identical except for the real IP's and the advskew which is set to 128. I have PF enables with pfsync as well, and I have set net.inet.carp.preempt=1 in systctl.conf. PF is configured to allow protocol 'carp' on both ether interfaces and 'pfsync' on the internal one. I did wonder if having the same vhid for a number of the addresse might be the issue so I then changed the config to have them all on separate vhid numbers, but the problem persists. This is now a bit of a major problem for me, as I am running on a single firewall with no faulover (which I dont like) and dont really know what the path forward is. As ever, all advice is welcome! -pete.