Date: Thu, 29 Aug 2019 17:02:47 +0200 From: "Kristof Provost" <kp@FreeBSD.org> To: "Konstantin Belousov" <kostikbel@gmail.com> Cc: "Li-Wen Hsu" <lwhsu@freebsd.org>, "FreeBSD Hackers" <freebsd-hackers@freebsd.org>, fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <B8B361D5-A41E-4A40-91CC-A7E170457257@FreeBSD.org> In-Reply-To: <20190829144228.GA71821@kib.kiev.ua> References: <CAKBkRUwKKPKwRvUs00ja0%2BG9vCBB1pKhv6zBS-F-hb=pqMzSxQ@mail.gmail.com> <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> <20190829144228.GA71821@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On 29 Aug 2019, at 16:42, Konstantin Belousov wrote: > On Thu, Aug 29, 2019 at 02:03:00PM +0200, Kristof Provost wrote: >> There are, somewhat regularly, commits which break functionality, or >> at >> the very least tests. >> The main objective of this policy proposal is to try to improve >> overall >> code quality by encouraging and empowering all committers to >> investigate >> and fix test failures. > But this policy does not encourage, if anything. > It gives a free ticket to revert, discouraging committers. > To provide a counterpoint here: my personal frustration right now is that I’ve spent a good bit of time adding tests for pf and fixing bugs for it, only to see the tests having to be disabled because of unrelated (to pf) changes in the network stack. Either through lack of visibility, or lack of time, or because people assume pf tests failures must by definition be the responsibility of the pf maintainer, these failures have not been investigated by anyone other than me, and I lack the time and subject matter expertise to fix them. I’m desperately afraid that if/when these bugs do get fixed we’re going to discover that other things have broken in the mean time, and the tests are still going to fail, for different reasons. These are bugs. They’re the best case scenario for bug reports even, because they come with a reproduction case built-in, and yet they’re still not getting fixed. This too is discouraging. I’m open to alternative proposals for how to address that problem, but I don’t think that “continue on as we always have” is the correct answer. Best regards, Kristof From owner-freebsd-fcp@freebsd.org Thu Aug 29 15:08:50 2019 Return-Path: <owner-freebsd-fcp@freebsd.org> Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6E934D8ABD for <freebsd-fcp@mailman.nyi.freebsd.org>; Thu, 29 Aug 2019 15:08:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K5bk1bJfz3JZD for <freebsd-fcp@freebsd.org>; Thu, 29 Aug 2019 15:08:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.nyi.freebsd.org (Postfix) id 349BCD8ABC; Thu, 29 Aug 2019 15:08:50 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 345DFD8ABB for <fcp@mailman.nyi.freebsd.org>; Thu, 29 Aug 2019 15:08:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5bj43gmz3JZ8 for <fcp@freebsd.org>; Thu, 29 Aug 2019 15:08:49 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x831.google.com with SMTP id b11so4007322qtp.10 for <fcp@freebsd.org>; Thu, 29 Aug 2019 08:08:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=2Wc2bEFcg8n1w6AcR/y2lTDsyKi+cVQ4C3w3oWA0M4U=; b=PgDmVkDkJHnFZ1HuuhjK5rT3egZ6RQDY8qFiw6tRNUmX5Y+Q/xd7JHcymuynH7dX0c EiDyXRsKF3tPgjYALwY1x8gTZK8Tur5O3BkrN7DYuQ9yDfgIlSIePKTD905Kty6en3NV n3kyPJ5dC703+lJudbChMUHR1+GpQ0kKaVLq9cDx97SewRIUs1Fw6rFwJpG487ffRXXR Iquw6yXFLQTcAcD14Tm6/UHeCwnq1FJuK7tTSLS/hoEwOauGXMHQ5t61wUs7z2rSJzW4 nHQAwSmdpwM3E90mENjXVAbSsba46+lLlWl/jqJPzDNeLqkZvut1wOi8W9lObR/JJGWP WUxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=2Wc2bEFcg8n1w6AcR/y2lTDsyKi+cVQ4C3w3oWA0M4U=; b=Cy6BzmI18TbLQugGWztu/srStOnOFTz2glkovm+E265y36JgR/VM1DjQMPWrGaJzUC 5EfwIG4xIilBErJY3m1BwABD+Tn6Obo4usZcrgmIIit20RX+2miTh1AVUM4s+3/XDNUX LY5a9ElLslbGj2f0AV44gsOzXawUEJyaDNLBpw64e6lQPAwFY1jiC5jzo4XZjORWqNbt AAwfgSYtjPBiSZHbVxF5oqLIgp7xUs2AYgWi2vx/qvfIP4HChkRWjd42YVWFgeTN5cGm vCxN3PcCLlOaC3xdTWZ4X3aaeMfTtAlnNRgCacAINRTy/wU4YwpsPQ7GXBnMYs1tqJbk +o6w== X-Gm-Message-State: APjAAAWqrTexXMEKex/IWtf/wfEja33B5Wy6goVp5/PqMKGeIEDFQlWx bbOy6N1PNer+SyOyHS+OczF37i1Fe9aeyNM6QualUatoME0= X-Google-Smtp-Source: APXvYqw6prGe0bR/jCOJ4nFhjgXq7WN7WbMmL5Wx3W/Pfn7FVwkvIS8gEJxjw0w98CtkLmv8JQLzXHUuxSy9EVJy/Lc= X-Received: by 2002:ac8:4602:: with SMTP id p2mr10203695qtn.291.1567091328412; Thu, 29 Aug 2019 08:08:48 -0700 (PDT) MIME-Version: 1.0 References: <CAKBkRUwKKPKwRvUs00ja0+G9vCBB1pKhv6zBS-F-hb=pqMzSxQ@mail.gmail.com> <20190829114057.GZ71821@kib.kiev.ua> <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> In-Reply-To: <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> From: Warner Losh <imp@bsdimp.com> Date: Thu, 29 Aug 2019 09:08:37 -0600 Message-ID: <CANCZdfqwnVD91Nv4JraUUnPWmmy-e3BV0k01xK+SNNfzLJQu6A@mail.gmail.com> Subject: Re: FCP 20190401-ci_policy: CI policy To: Ian Lepore <ian@freebsd.org> Cc: Konstantin Belousov <kostikbel@gmail.com>, Li-Wen Hsu <lwhsu@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org>, fcp@freebsd.org X-Rspamd-Queue-Id: 46K5bj43gmz3JZ8 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=PgDmVkDk; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::831) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-5.91 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[fcp@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; RCVD_IN_DNSWL_NONE(0.00)[1.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.91)[ip: (-9.34), ipnet: 2607:f8b0::/32(-2.85), asn: 15169(-2.32), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_CC(0.00)[gmail.com] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List <freebsd-fcp.freebsd.org> List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-fcp>, <mailto:freebsd-fcp-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fcp/>; List-Post: <mailto:freebsd-fcp@freebsd.org> List-Help: <mailto:freebsd-fcp-request@freebsd.org?subject=help> List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-fcp>, <mailto:freebsd-fcp-request@freebsd.org?subject=subscribe> X-List-Received-Date: Thu, 29 Aug 2019 15:08:50 -0000 On Thu, Aug 29, 2019 at 8:42 AM Ian Lepore <ian@freebsd.org> wrote: > On Thu, 2019-08-29 at 14:40 +0300, Konstantin Belousov wrote: > > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: > > > It seems I was doing wrong that just changed the content of this FCP > > > to "feedback", but did not send to the right mailing lists. > > > > > > So I would like to make an announcement that the FCP > > > 20190401-ci_policy "CI policy": > > > > > > https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md > > > > > > is officially in "feedback" state to hopefully receive more comments > > > and suggestions, then we can move on for the next FCP state. > > > > What problem does the document tries to solve ? Or rather, do we really > > have the problem that it claims to solve ? > > > > From my experience, normal peer pressure is enough to get things fixed > > quickly when it is possible to fix them quickly. If there is something > > more non-trivial, esp. in the tests and not the build, I am sure that > > a rule allowing anybody to do blind revert is much more harmful than > > having a test broken. > > > > More, I know that tests are of very low quality, which means that > > brokeness of the tests is not an indicator of anything until root cause > > is identified. > > > > Can we rely on the common sense of developers until there is indeed the > > visible problem ? > > > > I totally agree. This is an overly-bureaucratic solution in search of > a problem. > > If this needs to be addressed at all (and I'm not sure it does), then > another sentence or two in bullet item 10 in section 18.1 [*] of the > committer's guide should be enough. And even then it needn't be > overly-formal and should just mention that if a commit does break the > build the committer is expected to be responsive to that problem and > the commit might get reverted if they're unresponsive. I don't think > we need schedules. (And I don't think breaking a test counts as > breaking the build.) > > [*] > https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/rules.html There's been growing friction around these points in the past several years. We should document that tree breakages aren't acceptable, committers are expected to fix things as soon as possible, and that new broken tests have to be investigated quickly, or the change should be reverted. This document tries to attach some suggested time frames on different types of breakage we've seen and set the expectations for people when things go wrong. Having different categories also allows us to set expectations for external toolchain breakage that's more lax than in-tree toolchain breakage, for example. The future will have external toolchain CI and notifications will likely be turned on when there's breakage. While some tests are poorly written, we should disable / remove the ones that are actually bad rather than create a policy that tolerates bad tests by setting no expectation around that. Now, one may quibble over the timelines, but that discussion helps everybody in the community know what the expectations are when committing and enables people that want to scrimp on testing to do so if they know they will be around for any unanticipated fallout, but may also encourage others to test more because they know they might not want to be. Warner
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B8B361D5-A41E-4A40-91CC-A7E170457257>