From owner-freebsd-hackers@freebsd.org Thu Aug 29 14:54:29 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 90849D8174; Thu, 29 Aug 2019 14:54:29 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5H92v8Mz3HSC; Thu, 29 Aug 2019 14:54:29 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 162311BEA9; Thu, 29 Aug 2019 14:54:29 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [10.0.2.193] (ptr-8rh08k12jlyqauo4swr.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:240e:402:c59e:f85c:1ed8:db5b]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id 798F73858E; Thu, 29 Aug 2019 16:54:27 +0200 (CEST) From: "Kristof Provost" To: "Ian Lepore" Cc: "Konstantin Belousov" , "Li-Wen Hsu" , "FreeBSD Hackers" , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Date: Thu, 29 Aug 2019 16:54:26 +0200 X-Mailer: MailMate (2.0BETAr6137) Message-ID: In-Reply-To: <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> References: <20190829114057.GZ71821@kib.kiev.ua> <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 14:54:29 -0000 On 29 Aug 2019, at 16:42, Ian Lepore wrote: > (And I don't think breaking a test counts as > breaking the build.) > I fundamentally disagree on this point. A test failure is, just like a compiler warning, a precious gift that should not be ignored. The more distance (both in terms of time, and in terms of the people involved) there is between a bug being introduced and it being detected the harder it is to fix it. Test accelerate detection of bugs. If we do not take test failures seriously (i.e. as an indication something is wrong and should be fixed) the tests will inevitable become useless in one of two ways: we’ll either disable failing tests (which is what we tend to do now) reducing test coverage or we’ll have a test suite with many failures in it, which makes it useless as well. (As with compiler warnings, the best way to keep them under control is to consider them to be fatal errors.) In either scenario we end up reducing test coverage, which means we’re going to push more bugs towards users. > I totally agree. This is an overly-bureaucratic solution in search of > a problem. > > If this needs to be addressed at all (and I'm not sure it does), then > another sentence or two in bullet item 10 in section 18.1 [*] of the > committer's guide should be enough. And even then it needn't be > overly-formal and should just mention that if a commit does break the > build the committer is expected to be responsive to that problem and > the commit might get reverted if they're unresponsive. I don't think > we need schedules. > I do feel that’s a better argument. We’ve always had a policy of reverting on request (AIUI), so this is more or less trying to be a strong restatement of that, more than a fundamental shift in policy. Best regards, Kristof