Date: Thu, 29 Aug 2019 17:42:28 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Kristof Provost <kp@FreeBSD.org> Cc: Li-Wen Hsu <lwhsu@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org>, fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190829144228.GA71821@kib.kiev.ua> In-Reply-To: <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> References: <CAKBkRUwKKPKwRvUs00ja0%2BG9vCBB1pKhv6zBS-F-hb=pqMzSxQ@mail.gmail.com> <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Aug 29, 2019 at 02:03:00PM +0200, Kristof Provost wrote: > On 29 Aug 2019, at 13:40, Konstantin Belousov wrote: > > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: > >> It seems I was doing wrong that just changed the content of this FCP > >> to "feedback", but did not send to the right mailing lists. > >> > >> So I would like to make an announcement that the FCP > >> 20190401-ci_policy "CI policy": > >> > >> https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md > >> > >> is officially in "feedback" state to hopefully receive more comments > >> and suggestions, then we can move on for the next FCP state. > > > > What problem does the document tries to solve ? Or rather, do we > > really > > have the problem that it claims to solve ? > > > There are, somewhat regularly, commits which break functionality, or at > the very least tests. > The main objective of this policy proposal is to try to improve overall > code quality by encouraging and empowering all committers to investigate > and fix test failures. But this policy does not encourage, if anything. It gives a free ticket to revert, discouraging committers. > > >> From my experience, normal peer pressure is enough to get things > >> fixed > > quickly when it is possible to fix them quickly. If there is something > > more non-trivial, esp. in the tests and not the build, I am sure that > > a rule allowing anybody to do blind revert is much more harmful than > > having a test broken. > > > > More, I know that tests are of very low quality, which means that > > brokeness of the tests is not an indicator of anything until root > > cause > > is identified. > > > I’m not sure I agree with the characterisation that the tests are of > low quality. My own experience with the pf tests is that they test a > large section of the network stack and firewall code. They’ve > identified several very really issues (both pre- and post commit on the > epoch-isatin of the network stack, for example, as well as a fairly > important issue with IPv6 reassembly). This is my experience with the tests for kernel/libc/libthr, most of which comes from contrib/netbsd-tests, as I understand. > It’s certainly true that the pf tests often reveal issues that are not > in pf but in other code. I wouldn’t agree that this is a sign of low > quality tests, but instead I consider it a sign that we don’t have > enough tests for the network stack itself. > > > Can we rely on the common sense of developers until there is indeed > > the > > visible problem ? > > > I don’t want to suggest that people simply don’t care about test > failures, because that’s clearly not true. > > On the other hand, I do think we can do better. There are at least two > open problem in the network stack that I currently can’t get anyone to > look at, and where I personally do not have sufficient context (or time) > to fix them myself. (#239380, #238870). > > This proposal isn’t a silver bullet, I don’t think there is such a > thing, but I do believe that elevating the visibility and importance of > test failures can help us improve overall quality. This is not what was proposed. I fully agree that build failures should be fixed ASAP, and the each test results change (note the formulation) should be examined. But I do not think we have the problem right now of a scale which requires the free pass to revert with 4h timer.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190829144228.GA71821>