Date: Thu, 29 Aug 2019 14:03:00 +0200 From: "Kristof Provost" <kp@FreeBSD.org> To: "Konstantin Belousov" <kostikbel@gmail.com> Cc: "Li-Wen Hsu" <lwhsu@freebsd.org>, "FreeBSD Hackers" <freebsd-hackers@freebsd.org>, fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> In-Reply-To: <20190829114057.GZ71821@kib.kiev.ua> References: <CAKBkRUwKKPKwRvUs00ja0%2BG9vCBB1pKhv6zBS-F-hb=pqMzSxQ@mail.gmail.com> <20190829114057.GZ71821@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On 29 Aug 2019, at 13:40, Konstantin Belousov wrote: > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: >> It seems I was doing wrong that just changed the content of this FCP >> to "feedback", but did not send to the right mailing lists. >> >> So I would like to make an announcement that the FCP >> 20190401-ci_policy "CI policy": >> >> https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md >> >> is officially in "feedback" state to hopefully receive more comments >> and suggestions, then we can move on for the next FCP state. > > What problem does the document tries to solve ? Or rather, do we > really > have the problem that it claims to solve ? > There are, somewhat regularly, commits which break functionality, or at the very least tests. The main objective of this policy proposal is to try to improve overall code quality by encouraging and empowering all committers to investigate and fix test failures. >> From my experience, normal peer pressure is enough to get things >> fixed > quickly when it is possible to fix them quickly. If there is something > more non-trivial, esp. in the tests and not the build, I am sure that > a rule allowing anybody to do blind revert is much more harmful than > having a test broken. > > More, I know that tests are of very low quality, which means that > brokeness of the tests is not an indicator of anything until root > cause > is identified. > I’m not sure I agree with the characterisation that the tests are of low quality. My own experience with the pf tests is that they test a large section of the network stack and firewall code. They’ve identified several very really issues (both pre- and post commit on the epoch-isatin of the network stack, for example, as well as a fairly important issue with IPv6 reassembly). It’s certainly true that the pf tests often reveal issues that are not in pf but in other code. I wouldn’t agree that this is a sign of low quality tests, but instead I consider it a sign that we don’t have enough tests for the network stack itself. > Can we rely on the common sense of developers until there is indeed > the > visible problem ? > I don’t want to suggest that people simply don’t care about test failures, because that’s clearly not true. On the other hand, I do think we can do better. There are at least two open problem in the network stack that I currently can’t get anyone to look at, and where I personally do not have sufficient context (or time) to fix them myself. (#239380, #238870). This proposal isn’t a silver bullet, I don’t think there is such a thing, but I do believe that elevating the visibility and importance of test failures can help us improve overall quality. Best regards, Kristof From owner-freebsd-fcp@freebsd.org Thu Aug 29 14:42:30 2019 Return-Path: <owner-freebsd-fcp@freebsd.org> Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6D661D78E7 for <freebsd-fcp@mailman.nyi.freebsd.org>; Thu, 29 Aug 2019 14:42:30 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K51L2KPTz3GCP for <freebsd-fcp@freebsd.org>; Thu, 29 Aug 2019 14:42:30 +0000 (UTC) (envelope-from ian@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 4F99FD78E5; Thu, 29 Aug 2019 14:42:30 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4F482D78E3 for <fcp@mailman.nyi.freebsd.org>; Thu, 29 Aug 2019 14:42:30 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound2m.ore.mailhop.org (outbound2m.ore.mailhop.org [54.149.155.156]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46K51L0h8Kz3GCL for <fcp@freebsd.org>; Thu, 29 Aug 2019 14:42:29 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1567089748; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=a/iixmRyANIz2XYKkMo+f6254i6U+Dj868uVtSpXjInXC50OteB6mcgXHseF4aKxlx59DaSO/LxHI WFlOWcyQVh3gQwBmVxoRllA0F8GdpXXZp3fPgWR09cBZjwYLGlMbJ3GqUypmRrToZSp0oGZomy82LA sAemnjbRJfsWI+Lpl0ADj1omv9lR28DiX9dYCaF+zOG4I0vZ0jkHwkApYXjwx2PBvxPhG/Ym4VsP3x lD/tynr3MChPNKUnT5NClfH5fq8mMvYGvDLvjCeUVF8/RUWDFeD0p09AAj1nRVHTNKEeXfeTqgzxEe EKa0MyJeLFJo17p7K8N9l6C0bu4e+7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:dkim-signature:from; bh=q0A5Af+FjYXzUKhHiX5T2y2vMEbwVSRnnMr5Zyj6Azc=; b=hgA0dvajySNQtFfXVtA3Y5N0iCTIJg+xGYAUYP7lpYNaDDdc5wW+rngfoMGY22zq0ZgNS3AGRv7xW xJE2CgvvbH9jm8J5iMRPFGE6TuUybtdVY2s5JWFaNUz+HvNtRosdmMWqcvT8xGD/xKMT3Dv+RACH3+ F9hVo+xUMvBpTQQm7OXVQB3kUpeUkIO36vilvo9RZDuQnsGVM9b3SDgdoF/rKXsLe6MWD3bBg+KcxA 2xkouo6xC/FUrkWaKgrdM2747hIPN/nqnKpwxg0eYUJ4ZriUik+UyqlenjqyvMt3S43HXUa1iNbvHe wNd9u9er+dBiEWL7gmpTsKOx1vcwjqw== ARC-Authentication-Results: i=1; outbound4.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:from; bh=q0A5Af+FjYXzUKhHiX5T2y2vMEbwVSRnnMr5Zyj6Azc=; b=aQKiD4SUotZnnBulgPsFnO8j2mZHNV946DsBaHL7Fbvhzw4aaGrwPlkECSX2kYKd/nhC2Vmb5wDBu u55DpxWn+kAamJV9FoH1Wnp3D/TTcQmtNM4yoJbqb171jcS7EDKobQQU/SPBrzQAO48/yaY3WowBZR SzD9B9VKG+knKCLkZ2nmMAG8OS3GLVe0Xpot4P0c3W7te8XsLaxkOpJsn0zf3ATFuweU51qPVUXyNP 9Rg0Epo/Ra4WTrvWMVHX6VLoy/vpu4KTcdvVlCW3aI9b88AroL/pzn2Lx/lBc4mmDpfuQVpNQaMY8P +txhFAbszz3GsQlfhhoQYo6f5LPQNQA== X-MHO-RoutePath: aGlwcGll X-MHO-User: 3795f44d-ca6b-11e9-85ed-13b9aae3a1d2 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound4.ore.mailhop.org (Halon) with ESMTPSA id 3795f44d-ca6b-11e9-85ed-13b9aae3a1d2; Thu, 29 Aug 2019 14:42:26 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id x7TEgP25017390; Thu, 29 Aug 2019 08:42:25 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> Subject: Re: FCP 20190401-ci_policy: CI policy From: Ian Lepore <ian@freebsd.org> To: Konstantin Belousov <kostikbel@gmail.com>, Li-Wen Hsu <lwhsu@freebsd.org> Cc: FreeBSD Hackers <freebsd-hackers@freebsd.org>, fcp@freebsd.org Date: Thu, 29 Aug 2019 08:42:25 -0600 In-Reply-To: <20190829114057.GZ71821@kib.kiev.ua> References: <CAKBkRUwKKPKwRvUs00ja0+G9vCBB1pKhv6zBS-F-hb=pqMzSxQ@mail.gmail.com> <20190829114057.GZ71821@kib.kiev.ua> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 46K51L0h8Kz3GCL X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; ASN(0.00)[asn:16509, ipnet:54.148.0.0/15, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List <freebsd-fcp.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fcp>, <mailto:freebsd-fcp-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fcp/>; List-Post: <mailto:freebsd-fcp@freebsd.org> List-Help: <mailto:freebsd-fcp-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fcp>, <mailto:freebsd-fcp-request@freebsd.org?subject=subscribe> X-List-Received-Date: Thu, 29 Aug 2019 14:42:30 -0000 On Thu, 2019-08-29 at 14:40 +0300, Konstantin Belousov wrote: > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: > > It seems I was doing wrong that just changed the content of this FCP > > to "feedback", but did not send to the right mailing lists. > > > > So I would like to make an announcement that the FCP > > 20190401-ci_policy "CI policy": > > > > https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md > > > > is officially in "feedback" state to hopefully receive more comments > > and suggestions, then we can move on for the next FCP state. > > What problem does the document tries to solve ? Or rather, do we really > have the problem that it claims to solve ? > > From my experience, normal peer pressure is enough to get things fixed > quickly when it is possible to fix them quickly. If there is something > more non-trivial, esp. in the tests and not the build, I am sure that > a rule allowing anybody to do blind revert is much more harmful than > having a test broken. > > More, I know that tests are of very low quality, which means that > brokeness of the tests is not an indicator of anything until root cause > is identified. > > Can we rely on the common sense of developers until there is indeed the > visible problem ? > I totally agree. This is an overly-bureaucratic solution in search of a problem. If this needs to be addressed at all (and I'm not sure it does), then another sentence or two in bullet item 10 in section 18.1 [*] of the committer's guide should be enough. And even then it needn't be overly-formal and should just mention that if a commit does break the build the committer is expected to be responsive to that problem and the commit might get reverted if they're unresponsive. I don't think we need schedules. (And I don't think breaking a test counts as breaking the build.) [*] https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/rules.html -- Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?412537DD-D98F-4B92-85F5-CB93CF33F281>