From owner-freebsd-fcp@freebsd.org Thu Aug 29 14:42:41 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 38E2FD7918 for ; Thu, 29 Aug 2019 14:42:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K51X3xVxz3GGx for ; Thu, 29 Aug 2019 14:42:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 86D74D7914; Thu, 29 Aug 2019 14:42:40 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 85F9CD7913; Thu, 29 Aug 2019 14:42:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46K51W4wk8z3GG9; Thu, 29 Aug 2019 14:42:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x7TEgSpf005877 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 29 Aug 2019 17:42:31 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x7TEgSpf005877 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x7TEgS0o005876; Thu, 29 Aug 2019 17:42:28 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 29 Aug 2019 17:42:28 +0300 From: Konstantin Belousov To: Kristof Provost Cc: Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190829144228.GA71821@kib.kiev.ua> References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46K51W4wk8z3GG9 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.91 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.91)[-0.910,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 14:42:41 -0000 On Thu, Aug 29, 2019 at 02:03:00PM +0200, Kristof Provost wrote: > On 29 Aug 2019, at 13:40, Konstantin Belousov wrote: > > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: > >> It seems I was doing wrong that just changed the content of this FCP > >> to "feedback", but did not send to the right mailing lists. > >> > >> So I would like to make an announcement that the FCP > >> 20190401-ci_policy "CI policy": > >> > >> https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md > >> > >> is officially in "feedback" state to hopefully receive more comments > >> and suggestions, then we can move on for the next FCP state. > > > > What problem does the document tries to solve ? Or rather, do we > > really > > have the problem that it claims to solve ? > > > There are, somewhat regularly, commits which break functionality, or at > the very least tests. > The main objective of this policy proposal is to try to improve overall > code quality by encouraging and empowering all committers to investigate > and fix test failures. But this policy does not encourage, if anything. It gives a free ticket to revert, discouraging committers. > > >> From my experience, normal peer pressure is enough to get things > >> fixed > > quickly when it is possible to fix them quickly. If there is something > > more non-trivial, esp. in the tests and not the build, I am sure that > > a rule allowing anybody to do blind revert is much more harmful than > > having a test broken. > > > > More, I know that tests are of very low quality, which means that > > brokeness of the tests is not an indicator of anything until root > > cause > > is identified. > > > I’m not sure I agree with the characterisation that the tests are of > low quality. My own experience with the pf tests is that they test a > large section of the network stack and firewall code. They’ve > identified several very really issues (both pre- and post commit on the > epoch-isatin of the network stack, for example, as well as a fairly > important issue with IPv6 reassembly). This is my experience with the tests for kernel/libc/libthr, most of which comes from contrib/netbsd-tests, as I understand. > It’s certainly true that the pf tests often reveal issues that are not > in pf but in other code. I wouldn’t agree that this is a sign of low > quality tests, but instead I consider it a sign that we don’t have > enough tests for the network stack itself. > > > Can we rely on the common sense of developers until there is indeed > > the > > visible problem ? > > > I don’t want to suggest that people simply don’t care about test > failures, because that’s clearly not true. > > On the other hand, I do think we can do better. There are at least two > open problem in the network stack that I currently can’t get anyone to > look at, and where I personally do not have sufficient context (or time) > to fix them myself. (#239380, #238870). > > This proposal isn’t a silver bullet, I don’t think there is such a > thing, but I do believe that elevating the visibility and importance of > test failures can help us improve overall quality. This is not what was proposed. I fully agree that build failures should be fixed ASAP, and the each test results change (note the formulation) should be examined. But I do not think we have the problem right now of a scale which requires the free pass to revert with 4h timer.