From owner-freebsd-stable@freebsd.org Wed Jan 16 14:59:51 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68917148B0D2 for ; Wed, 16 Jan 2019 14:59:51 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from constantine.ingresso.co.uk (unknown [IPv6:2a02:b90:3002:411::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5C3A370F38 for ; Wed, 16 Jan 2019 14:59:50 +0000 (UTC) (envelope-from petefrench@ingresso.co.uk) Received: from [2a02:b90:3002:411::6] (helo=dilbert.ingresso.co.uk) by constantine.ingresso.co.uk with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.91 (FreeBSD)) (envelope-from ) id 1gjmfV-0006Cx-7V; Wed, 16 Jan 2019 14:59:49 +0000 Received: from petefrench by dilbert.ingresso.co.uk with local (Exim 4.91 (FreeBSD)) (envelope-from ) id 1gjmfV-000DjX-4E; Wed, 16 Jan 2019 14:59:49 +0000 To: freebsd-stable@freebsd.org, petefrench@ingresso.co.uk, thomas@gibfest.dk Subject: Re: CARP stopped working after upgrade from 11 to 12 In-Reply-To: Message-Id: From: Pete French Date: Wed, 16 Jan 2019 14:59:49 +0000 X-Rspamd-Queue-Id: 5C3A370F38 X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=ingresso.co.uk; spf=pass (mx1.freebsd.org: domain of petefrench@ingresso.co.uk designates 2a02:b90:3002:411::3 as permitted sender) smtp.mailfrom=petefrench@ingresso.co.uk X-Spamd-Result: default: False [1.97 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RDNS_NONE(1.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a02:b90:3002:411::3]; NEURAL_HAM_LONG(-0.72)[-0.717,0]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_MEDIUM(-0.33)[-0.327,0]; NEURAL_SPAM_SHORT(0.34)[0.341,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: ingresso-co-uk.mail.protection.outlook.com]; DMARC_POLICY_ALLOW(-0.50)[ingresso.co.uk,none]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16082, ipnet:2a02:b90::/32, country:GB]; HFILTER_HOSTNAME_UNKNOWN(2.50)[]; IP_SCORE(-0.02)[country: GB(-0.09)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2019 14:59:51 -0000 > Indeed. I am seeing the same thing. Which revision of 12 are you running? Ah, now that is very interesting - I wasnt expecting a reply so fast! I am running r342847 - not though, that this is also the version I am running on the two test systems which do work. > I am currently (yesterday and today) bisecting revisions to find the > commit which broke this, because it worked in 12-BETA2 but doesn't work > on latest 12-STABLE. Well done, thats takes a lot of effort to do. Thankyou for doing this. > MFC r340394: ipfw.8: Fix part of the SYNOPSIS documenting > LIST OF RULES AND PREPROCESSING that is still referred > as last section of the SYNOPSIS later but was erroneously situated > in the section IN-KERNEL NAT. Docs only, so cant be this one I think. > MFC r341638: > Let kern.trap_enotcap be set as a tunable. Also cant be this one from eyeballing the code. It simply makes it writeable. > MFC r340405: > Add accounting to per-domain UMA full bucket caches. This is not touching networking, so seems unlikely, though it is an actaul significant code chnage. > r342051 | kp | 2018-12-13 20:00:11 +0000 (Thu, 13 Dec 2018) | 20 lines > > pfsync: Performance improvement ahh..... now, this is where things like likely, as the difference between my test amchines and my live machines is that the live machines are using pf with pfsync enabled. > Of these I thought r342051 sounded most likely, so I am currently > building r342050. Thats my feeling too going through the above. Have you also tried simply disabling pfsync to see if CARP returns to normal ? I could live without pfsync to be honest, if thats what it takes to make this work. > I will write again in a few hours when I have isolated the commit. Thankyou again for outting in the effort to bisect this - if you can isolate it then we can back the change out and try it again and see if that helps. -pete.