From owner-freebsd-hackers@freebsd.org Sat Aug 31 17:01:14 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6F662D6B32 for ; Sat, 31 Aug 2019 17:01:14 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x744.google.com (mail-qk1-x744.google.com [IPv6:2607:f8b0:4864:20::744]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46LN0T3D89z3C6n for ; Sat, 31 Aug 2019 17:01:13 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x744.google.com with SMTP id d26so1376077qkk.2 for ; Sat, 31 Aug 2019 10:01:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:cc; bh=TASPvrVnu4iidRxLP6ME7du/hTU8PZXSNp7Ts/viXls=; b=yKMyZw4IFNu8VkgDQHXh+LnnVs4lz4Pk+4TReFK0z1F4WzrxtFBMKgbMEWHJSiyZHk Y1YufKB3bFjFt6V7yLzgrOjbRbSqbmyJpMOwjNlRTajGRP7LbmVhmbhBHIR8TsU/Is4H kVl1bFV9apG4SuxynEqW/JM+7Yqf1mLJK3bXyYdnLgZUagjYibBMDb0NVU9jRThZ5ox4 dS4krOj2XPgF/W/c9t9UF38ujf2LKZ6QqqRWKcMVzr/LLUu2iBLd/3SzE+3mXDH2ms5/ 14b5A+SYDZ8QuR1RkiCNWcTJLxlmMHMMKMKs8IbV5PIHG8aDIppQ4uAL8qlW5qHNmKa8 Mo0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:cc; bh=TASPvrVnu4iidRxLP6ME7du/hTU8PZXSNp7Ts/viXls=; b=YKACLWOZZYhokdLoq9Apvo1xt4cIYRYp1rH/NAaumOnJodUcMZUvvkiqnvaqOcDGAi /C0kOel2TILPsg8n9sLT3r1jPAVCjw2KIA00QAdNVHYkvahwiAVq5ilXUhop+FRSuHAo r+Jh48LS2JRbYN796+kAW5tmnW9/QRvXDlKZslQjyStGCeQ/clDJ/9Jxvir3sjcD7XXb +D/bzn6r+W2rEk2XDOgqHbbJjps2oI/7jhBIDuuZf/HJohQs/PijnFK/yGzLyNLdpbzO R2P4vdZeEWhSpDnnoIMaJgNX3ItSAC9o5LU5Xv9AeNE85b7qZAZ3rH6/f7RFTVtmokft JmKg== X-Gm-Message-State: APjAAAU0bagAcf1xtOntwF+TzAmN/0+lQdiUdgLCg9hSZBTVoT66PE3a +JBIr4DH2JBu4Vf4g5mQx/VHvyo4UUhpIWx96vHcZAs6lds= X-Received: by 2002:a37:30f:: with SMTP id 15mt11586980qkd.240.1567270871089; Sat, 31 Aug 2019 10:01:11 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <20190830065534.GC71821@kib.kiev.ua> <73111.1567263258@kaos.jnpr.net> In-Reply-To: <73111.1567263258@kaos.jnpr.net> From: Warner Losh Date: Sat, 31 Aug 2019 11:00:59 -0600 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy Cc: FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org X-Rspamd-Queue-Id: 46LN0T3D89z3C6n X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=yKMyZw4I; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::744) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-1.46 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.979,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.978,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-0.95)[-0.951,0]; RCVD_IN_DNSWL_NONE(0.00)[4.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MISSING_TO(2.00)[]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-0.56)[ip: (2.42), ipnet: 2607:f8b0::/32(-2.83), asn: 15169(-2.31), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 17:01:14 -0000 After this weeks discussions, I got to thinking about where we were on this topic. I thought I'd spend a few minutes summarizing my impressions so that we don't get bogged down in the details of disagreement, but rather start from where we agree. Here's my take on what that is, but I write this down as the basis for discussion, not to lay down the law. I think there's consensus on the following points: (1) We want CI (2) We want developers to be responsive to breakage in CI (3) At the moment, we have some sub-optimal tools, and when we change those we should evolve the process. (4) Build breakage is something that should be fixed very quickly. (5) People should reach out to the original developer who committed the change when there's breakage ASAP (6) If the original developer can't timely fix the problem, it's OK to either back out the change, or perhaps commit a tiny fix if the original was huge and the fix needed to fix the build is tiny. (7) Breaking tests is a problem, but our tests need to evolve because we have too high a rate of false positives. (8) There's a sliding scale of urgency (Tier 1 build breakage needs to be fixed in a couple of hours, Tier 2 can go a day, Tier 3 can go longer but shouldn't linger). (9) The urgency for a Test regression also is a sliding scale, but given the current state of the tests we need to apply judgement on revert vs fix test vs disable test. (10) Reverts shouldn't be feared, but there's a cost to reverting automatically and there's some desire for developers to have a chance to be in the loop (11) We need to work on the social aspect of reverts to destigmatize them. We might quibble a bit over timelines for the different pieces, but here's what I've noticed the approximate timelines are today (there are exceptions, and I don't have hard data, just my sense from watching the tree, but I think they form a reasonable basis absent better data): * Build system breakage usually is fixed within an hour (eg, I screwed up a Makefile or bsd.foo.mk file somehow). * x86 build breakages are usually fixed in an hour or two (longer over the weekend) * arm and arm64 build breakages are usually fixed within 4-8 hours * Other build breakages are usually fixed within a day or two. * out of tree compiler breakages are fixed on the order of a week. * I have little data on test breakage, but it's my sense most issues are resolved in less than a week. We've been quite reluctant to do reverts to date. They happen, but have usually been initiated by the committer. Li-Wen and others would like to change that to setting firm timelines; start to reset the social aspect of reverts and document the social norms with an eye towards improving things, either within the SVN framework, or the coming git framework. Finally, there's a number of ways we can do this, not limited to the FCP in question. However, nobody has stepped up to drive that initiative. It would be great to have someone sign up to drive revisions of the various internal developer guides to modernize their content to reflect how norms have evolved since the were written (including this topic and the related topic of responsiveness to @freebsd.org email and others...). Warner