From owner-freebsd-hackers@freebsd.org Thu Aug 29 15:02:50 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 944C8D88C1; Thu, 29 Aug 2019 15:02:50 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5Sp3LJKz3JJC; Thu, 29 Aug 2019 15:02:50 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 25CDC1BFE8; Thu, 29 Aug 2019 15:02:50 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [10.0.2.193] (ptr-8rh08k12jlyqauo4swr.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:240e:402:c59e:f85c:1ed8:db5b]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id A83F6385CC; Thu, 29 Aug 2019 17:02:48 +0200 (CEST) From: "Kristof Provost" To: "Konstantin Belousov" Cc: "Li-Wen Hsu" , "FreeBSD Hackers" , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Date: Thu, 29 Aug 2019 17:02:47 +0200 X-Mailer: MailMate (2.0BETAr6137) Message-ID: In-Reply-To: <20190829144228.GA71821@kib.kiev.ua> References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> <20190829144228.GA71821@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed; markup=markdown Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 15:02:50 -0000 On 29 Aug 2019, at 16:42, Konstantin Belousov wrote: > On Thu, Aug 29, 2019 at 02:03:00PM +0200, Kristof Provost wrote: >> There are, somewhat regularly, commits which break functionality, or >> at >> the very least tests. >> The main objective of this policy proposal is to try to improve >> overall >> code quality by encouraging and empowering all committers to >> investigate >> and fix test failures. > But this policy does not encourage, if anything. > It gives a free ticket to revert, discouraging committers. > To provide a counterpoint here: my personal frustration right now is that I’ve spent a good bit of time adding tests for pf and fixing bugs for it, only to see the tests having to be disabled because of unrelated (to pf) changes in the network stack. Either through lack of visibility, or lack of time, or because people assume pf tests failures must by definition be the responsibility of the pf maintainer, these failures have not been investigated by anyone other than me, and I lack the time and subject matter expertise to fix them. I’m desperately afraid that if/when these bugs do get fixed we’re going to discover that other things have broken in the mean time, and the tests are still going to fail, for different reasons. These are bugs. They’re the best case scenario for bug reports even, because they come with a reproduction case built-in, and yet they’re still not getting fixed. This too is discouraging. I’m open to alternative proposals for how to address that problem, but I don’t think that “continue on as we always have” is the correct answer. Best regards, Kristof From owner-freebsd-hackers@freebsd.org Thu Aug 29 15:08:50 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6EDF7D8ABE for ; Thu, 29 Aug 2019 15:08:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5bj4qwcz3JZB for ; Thu, 29 Aug 2019 15:08:49 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x833.google.com with SMTP id y26so4047409qto.4 for ; Thu, 29 Aug 2019 08:08:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=2Wc2bEFcg8n1w6AcR/y2lTDsyKi+cVQ4C3w3oWA0M4U=; b=PgDmVkDkJHnFZ1HuuhjK5rT3egZ6RQDY8qFiw6tRNUmX5Y+Q/xd7JHcymuynH7dX0c EiDyXRsKF3tPgjYALwY1x8gTZK8Tur5O3BkrN7DYuQ9yDfgIlSIePKTD905Kty6en3NV n3kyPJ5dC703+lJudbChMUHR1+GpQ0kKaVLq9cDx97SewRIUs1Fw6rFwJpG487ffRXXR Iquw6yXFLQTcAcD14Tm6/UHeCwnq1FJuK7tTSLS/hoEwOauGXMHQ5t61wUs7z2rSJzW4 nHQAwSmdpwM3E90mENjXVAbSsba46+lLlWl/jqJPzDNeLqkZvut1wOi8W9lObR/JJGWP WUxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=2Wc2bEFcg8n1w6AcR/y2lTDsyKi+cVQ4C3w3oWA0M4U=; b=m9sIXUUDfExfLiDmuzq1zCsQZReWO4qNi8TwD/9yFfoZsM8IBYrwLY4c15Ft18Ddd4 KD3uDSk4TyDrKNrfRlVysrdZN2EC2Wm2b0OXX+3aSNRntzE4pR6r/DUW7Z9YYq0VyzRs OvpOu82X0FQJj0Em2pyqC3GgYWCdmy8bW83pGBlSFjyaalMoEMhanLY7n/5pbFCtVMhL syVtjoLGw7T9MBeNZwLjyWrl5XhaKws5+NZoPSPKQZPzBk27x0c0MwH+1+20y9AuJhFP +k1HB4hZPWBrgkya22PeeVmy7J6cxhVg3k+6XgGEiAxmSiORBXPDx/txKcH9KN2jrgFf lNcQ== X-Gm-Message-State: APjAAAV0Qfmo3nNtbeRwu7DW3lD8LWskbW4ONat1z0XzjHWiM+ki7Nyv xgTT0mxKhuuT2yiE/4sJSPQYUtOM2T88rbIKOLVPvQ== X-Google-Smtp-Source: APXvYqw6prGe0bR/jCOJ4nFhjgXq7WN7WbMmL5Wx3W/Pfn7FVwkvIS8gEJxjw0w98CtkLmv8JQLzXHUuxSy9EVJy/Lc= X-Received: by 2002:ac8:4602:: with SMTP id p2mr10203695qtn.291.1567091328412; Thu, 29 Aug 2019 08:08:48 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> In-Reply-To: <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> From: Warner Losh Date: Thu, 29 Aug 2019 09:08:37 -0600 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Ian Lepore Cc: Konstantin Belousov , Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org X-Rspamd-Queue-Id: 46K5bj4qwcz3JZB X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=PgDmVkDk; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::833) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-5.91 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; RCVD_IN_DNSWL_NONE(0.00)[3.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.91)[ip: (-9.33), ipnet: 2607:f8b0::/32(-2.85), asn: 15169(-2.32), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_CC(0.00)[gmail.com] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 15:08:50 -0000 On Thu, Aug 29, 2019 at 8:42 AM Ian Lepore wrote: > On Thu, 2019-08-29 at 14:40 +0300, Konstantin Belousov wrote: > > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: > > > It seems I was doing wrong that just changed the content of this FCP > > > to "feedback", but did not send to the right mailing lists. > > > > > > So I would like to make an announcement that the FCP > > > 20190401-ci_policy "CI policy": > > > > > > https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md > > > > > > is officially in "feedback" state to hopefully receive more comments > > > and suggestions, then we can move on for the next FCP state. > > > > What problem does the document tries to solve ? Or rather, do we really > > have the problem that it claims to solve ? > > > > From my experience, normal peer pressure is enough to get things fixed > > quickly when it is possible to fix them quickly. If there is something > > more non-trivial, esp. in the tests and not the build, I am sure that > > a rule allowing anybody to do blind revert is much more harmful than > > having a test broken. > > > > More, I know that tests are of very low quality, which means that > > brokeness of the tests is not an indicator of anything until root cause > > is identified. > > > > Can we rely on the common sense of developers until there is indeed the > > visible problem ? > > > > I totally agree. This is an overly-bureaucratic solution in search of > a problem. > > If this needs to be addressed at all (and I'm not sure it does), then > another sentence or two in bullet item 10 in section 18.1 [*] of the > committer's guide should be enough. And even then it needn't be > overly-formal and should just mention that if a commit does break the > build the committer is expected to be responsive to that problem and > the commit might get reverted if they're unresponsive. I don't think > we need schedules. (And I don't think breaking a test counts as > breaking the build.) > > [*] > https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/rules.html There's been growing friction around these points in the past several years. We should document that tree breakages aren't acceptable, committers are expected to fix things as soon as possible, and that new broken tests have to be investigated quickly, or the change should be reverted. This document tries to attach some suggested time frames on different types of breakage we've seen and set the expectations for people when things go wrong. Having different categories also allows us to set expectations for external toolchain breakage that's more lax than in-tree toolchain breakage, for example. The future will have external toolchain CI and notifications will likely be turned on when there's breakage. While some tests are poorly written, we should disable / remove the ones that are actually bad rather than create a policy that tolerates bad tests by setting no expectation around that. Now, one may quibble over the timelines, but that discussion helps everybody in the community know what the expectations are when committing and enables people that want to scrimp on testing to do so if they know they will be around for any unanticipated fallout, but may also encourage others to test more because they know they might not want to be. Warner