From owner-freebsd-net@freebsd.org Tue Sep 8 13:38:24 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B66B83C99BA for ; Tue, 8 Sep 2020 13:38:24 +0000 (UTC) (envelope-from freebsd@grem.de) Received: from mail.evolve.de (mail.evolve.de [213.239.217.29]) (using TLSv1.2 with cipher ECDHE-RSA-CHACHA20-POLY1305 (256/256 bits)) (Client CN "mail.evolve.de", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bm5nq57tpz3Wfh for ; Tue, 8 Sep 2020 13:38:23 +0000 (UTC) (envelope-from freebsd@grem.de) Received: by mail.evolve.de (OpenSMTPD) with ESMTP id b412a9f9; Tue, 8 Sep 2020 13:38:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=grem.de; h=content-type :content-transfer-encoding:mime-version:subject:from:in-reply-to :date:cc:message-id:references:to; s=20180501; bh=LKcZQ14AkSk+xk 16r+ukHxq65/w=; b=BjsmdQSAP6HuIpGIjhJxID1ZSNdtYaIrzAMGKfnk1TsHPa CrcQi9s5I1ppH07Kh0JFFWv1/8spt7GiFG6ntzYNaG1Pjzs46AXwJbPUrCJsFw7Z 3TXtg/hZ4+iYWg5isrqA8QkeQivFo8dUI1017Cg2KLnL59leL0PwmJHa2ZaUi/+Z XjvJddlepHnGlR5RyWN2ad9Uzg5D2s7BXOhhqXu3Oujq161VrlmJV8ePS7nCRk6w igzU0cQppsPLRysiD+Fz/nc7eaRoe3BFvSJH43oP2ifHBJrcbT+0taxBE0gGaLcN hnheQAbUYi6CvWcvmSGEm1QUxc5asZT/J+mgpPug== DomainKey-Signature: a=rsa-sha1; c=nofws; d=grem.de; h=content-type :content-transfer-encoding:mime-version:subject:from:in-reply-to :date:cc:message-id:references:to; q=dns; s=20180501; b=OPi9eSoD vSEC62nUj7Khv/ZRPcOTT8Y0dry0NZE9pxEi0qBsKS65IHvvz1Xgc0VMoW/QJgLr iLavlmZiV3QuzHVvLBI3ufVuDX++KTx0KJsbuh1Q2dGsds/QWQ02eUpx3okSg0/R 6RwltNnAttOWLTn76m1NWOXdYNRH1JP7A873RVyKeXh9arTQvfFpLFMfnertuBwZ VOkChctI7xEucYJkvjO3i+ExgATjCZRpLWs9XecRVE2kdk+5yHcyKaPoSXBkx7r5 hMXmstenAQhZwitwa9mofobKUFxB1Kx5R16HySZ4I+e5VRO4erqSRAY1yokml7eq DOqpztYznT8jDQ== Received: by mail.evolve.de (OpenSMTPD) with ESMTPSA id 7422ddcc (TLSv1.2:ECDHE-RSA-CHACHA20-POLY1305:256:NO); Tue, 8 Sep 2020 13:38:14 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Subject: Re: CARP over VLAN over LAGG From: Michael Gmelin In-Reply-To: <20200908132314.2txabgcuz4wmsq7n@x1> Date: Tue, 8 Sep 2020 15:38:14 +0200 Cc: freebsd-net@freebsd.org Message-Id: References: <20200908132314.2txabgcuz4wmsq7n@x1> To: Julien Cigar X-Mailer: iPhone Mail (17H35) X-Rspamd-Queue-Id: 4Bm5nq57tpz3Wfh X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=grem.de header.s=20180501 header.b=BjsmdQSA; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@grem.de designates 213.239.217.29 as permitted sender) smtp.mailfrom=freebsd@grem.de X-Spamd-Result: default: False [-2.48 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[grem.de:s=20180501]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+ip4:213.239.217.29/32]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grem.de]; NEURAL_HAM_LONG(-0.98)[-0.979]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[grem.de:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.52)[-0.524]; NEURAL_HAM_MEDIUM(-0.98)[-0.981]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:213.239.192.0/18, country:DE]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Sep 2020 13:38:24 -0000 > On 8. Sep 2020, at 15:23, Julien Cigar wrote: >=20 > =EF=BB=BFOn Tue, Sep 01, 2020 at 10:13:23AM +0200, Julien Cigar wrote: >>> On Mon, Aug 31, 2020 at 01:55:52PM +0200, Michael Gmelin wrote: >>>=20 >>>=20 >>>> On 31. Aug 2020, at 10:37, Julien Cigar wrote: >>>>=20 >>>> =EF=BB=BFOn Fri, Aug 28, 2020 at 04:52:01PM +0200, Julien Cigar wrote: >>>>> Hello, >>>>>=20 >>>>> I have a "highly available" router/firewall with the following >>>>> configuration (1). Those are plugged in two 2930F (with VSF) using LAC= P. >>>>> It works well, except that I have some weird issues with the CARP=20 >>>>> demotion counter when I'm unplugging some interfaces involved in the=20= >>>>> lagg/carp setup, for example if I unplug/replug igb0 and igb1 in this=20= >>>>> case: >>>>>=20 >>>>> (dmesg): >>>>> igb0: link state changed to DOWN >>>>> igb1: link state changed to DOWN >>>>> carp: demoted by 240 to 240 (send error 50 on vlan11) >>>>> carp: 11@vlan11: MASTER -> BACKUP (more frequent advertisement receive= d) >>>>> vlan11: deletion failed: 3 >>>>> igb1: link state changed to UP >>>>> igb0: link state changed to UP >>>>>=20 >>>>> then the CARP status stays to BACKUP unless I demote the CARP demotion= >>>>> counter manually with: sudo sysctl net.inet.carp.demotion=3D-240: >>>>>=20 >>>>> (dmesg): >>>>> carp: demoted by -240 to 0 (sysctl) >>>>> carp: 11@vlan11: BACKUP -> MASTER (preempting a slower master) >>>>>=20 >>>>> I guess this is because it takes some time for lagg/lacp to converge a= nd >>>>> thus carp thinks that there is a problematic condition as it experienc= es >>>>> problems with sending announcements.. >>>>>=20 >>>>> What it the best way to handle this? >>>>=20 >>>> I'm wondering if setting net.inet.carp.senderr_demotion_factor to "0" >>>> could be a solution? Are there any downsides of setting this to "0" >>>> instead of "240"? >>>>=20 >>>=20 >>> Sharing your pf.conf from both hosts could be helpful analyzing the issu= e. >>=20 >> Here is my pf.conf (it's the same on both host): >> https://gist.github.com/silenius/b758851f03c28ef8caaa53cfe381c455 >>=20 >> However, I don't think pf is the issue here, the problem is that there >> is a slight delay when LAGG/LACP converge and thus CARP increase the >> demotion counter by net.inet.carp.senderr_demotion_factor (240). >=20 > I can confirm that after setting net.inet.carp.senderr_demotion_factor=3D0= > (instead of 240) it works as expected. >=20 Cool, thanks for posting the solution. I don=E2=80=99t think I ever used CAR= P with lagg (usually I had redundant firewalls with CARP and servers using l= agg connected to both of them). Cheers, Michael >>=20 >>>=20 >>> -m >>>=20 >>>=20 >>=20 >> --=20 >> Julien Cigar >> Belgian Biodiversity Platform (http://www.biodiversity.be) >> PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 >> No trees were killed in the creation of this message. >> However, many electrons were terribly inconvenienced. >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >=20 > --=20 > Julien Cigar > Belgian Biodiversity Platform (http://www.biodiversity.be) > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > No trees were killed in the creation of this message. > However, many electrons were terribly inconvenienced.