From owner-freebsd-fcp@freebsd.org Wed Aug 28 04:30:17 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 07297D0D15 for ; Wed, 28 Aug 2019 04:30:17 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46JCTN5Q8Jz3DlH for ; Wed, 28 Aug 2019 04:30:16 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id B8013D0D12; Wed, 28 Aug 2019 04:30:16 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B7A93D0D11; Wed, 28 Aug 2019 04:30:16 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-yb1-f172.google.com (mail-yb1-f172.google.com [209.85.219.172]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46JCTM36jtz3DlG; Wed, 28 Aug 2019 04:30:12 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-yb1-f172.google.com with SMTP id z2so355118ybp.9; Tue, 27 Aug 2019 21:30:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=QYAnd6yKa5/PNe/LsQCvO+Q0ux1DvZfWAhvQc/CmPD8=; b=H3RApEg1RsOEC5KBy0eZABl+zjDWvmicgNmJXhBWPkpK0ww8iOe9pxbsRQFQeSPJBP qJkZR4Rv27dRNabZtsWYEeVIVlF5j7Lq3RloPvM31SfO/RMfpj4iJa+NU8kzlCtH2NLX xkcHpZaCMuRAgwAMK9weSZeperI9nS92PgUCrLJbmXJ6JHpYeMfRPqSUqGXgoXhHtNvL /VIw5G+QjxbQAvM2xeSiSvtp9oTiv3z9ZU+WkcD7Ln56ioD0SYVA5fe/WqsBmmVs6A6N BTHUVCrdCsXh8YQnfWwufFzjc7pRoxvM2oHwTBmFq3iMbjkWkGPJHHIhYPWP6+GppLrV dnEQ== X-Gm-Message-State: APjAAAXoVoK5LNPEXpQI3X7YJzywDvqEA4q4oDl4fWcJXErCkOMHy47Y osno0WiT20TAWDvFuDb4D18fxbn7J+TsI+zDmVcGSVaS X-Google-Smtp-Source: APXvYqwosnRMDk0Oi+gCXcuRruoHUyL4Wp5xhPCxgQyHBuBxpiubCemNl1fnMU7Q5LDnwH0dWZrA0Q5zIz3VfUS8+/8= X-Received: by 2002:a25:7089:: with SMTP id l131mr1629577ybc.110.1566966610613; Tue, 27 Aug 2019 21:30:10 -0700 (PDT) MIME-Version: 1.0 From: Li-Wen Hsu Date: Wed, 28 Aug 2019 12:29:58 +0800 Message-ID: Subject: FCP 20190401-ci_policy: CI policy To: fcp@freebsd.org, FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46JCTM36jtz3DlG X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of lwhsufreebsd@gmail.com designates 209.85.219.172 as permitted sender) smtp.mailfrom=lwhsufreebsd@gmail.com X-Spamd-Result: default: False [-5.98 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-2.98)[ip: (-9.20), ipnet: 209.85.128.0/17(-3.34), asn: 15169(-2.33), country: US(-0.05)]; NEURAL_HAM_SHORT(-0.99)[-0.993,0]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[172.219.85.209.list.dnswl.org : 127.0.5.0]; FORGED_SENDER(0.30)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; TAGGED_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Aug 2019 04:30:17 -0000 It seems I was doing wrong that just changed the content of this FCP to "feedback", but did not send to the right mailing lists. So I would like to make an announcement that the FCP 20190401-ci_policy "CI policy": https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md is officially in "feedback" state to hopefully receive more comments and suggestions, then we can move on for the next FCP state. Thanks, Li-Wen From owner-freebsd-fcp@freebsd.org Wed Aug 28 11:30:28 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3EAB2D9A0A for ; Wed, 28 Aug 2019 11:30:28 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46JNpC5v3yz45PP for ; Wed, 28 Aug 2019 11:30:27 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: by mailman.nyi.freebsd.org (Postfix) id CA3D8D9A08; Wed, 28 Aug 2019 11:30:27 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C9E07D9A07; Wed, 28 Aug 2019 11:30:27 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from mailgate.Leidinger.net (mailgate.leidinger.net [IPv6:2a00:1828:2000:313::1:5]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46JNpB02r7z45PN; Wed, 28 Aug 2019 11:30:25 +0000 (UTC) (envelope-from Alexander@leidinger.net) Received: from outgoing.leidinger.net (p5B1657F2.dip0.t-ipconnect.de [91.22.87.242]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by mailgate.Leidinger.net (Postfix) with ESMTPSA id 90A9F14CB; Wed, 28 Aug 2019 13:20:22 +0200 (CEST) Received: from webmail.leidinger.net (webmail.Leidinger.net [IPv6:fd73:10c7:2053:1::3:102]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) by outgoing.leidinger.net (Postfix) with ESMTPS id BF40C936C; Wed, 28 Aug 2019 13:20:19 +0200 (CEST) Received: (from www@localhost) by webmail.leidinger.net (8.15.2/8.14.4/Submit) id x7SBKJnj025847; Wed, 28 Aug 2019 13:20:19 +0200 (CEST) (envelope-from Alexander@leidinger.net) X-Authentication-Warning: webmail.leidinger.net: www set sender to Alexander@leidinger.net using -f Received: from [::ffff:31.3.144.27] ([::ffff:31.3.144.27]) by webmail.leidinger.net (Horde Framework) with HTTPS; Wed, 28 Aug 2019 13:20:19 +0200 Date: Wed, 28 Aug 2019 13:20:19 +0200 Message-ID: <20190828132019.Horde.u4cw2bP8C-GNjeokW3YPrg9@webmail.leidinger.net> From: Alexander Leidinger To: Li-Wen Hsu Cc: fcp@freebsd.org, FreeBSD Hackers Subject: Re: FCP 20190401-ci_policy: CI policy In-Reply-To: User-Agent: Horde Application Framework 5 Accept-Language: de,en Content-Type: multipart/signed; boundary="=_Pc7I2F4qlUaL4QpRwd9TYBt"; protocol="application/pgp-signature"; micalg=pgp-sha1 MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=leidinger.net; s=outgoing-alex; t=1566991222; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to; bh=VDN9vUTW2Fy9Z1wtBl+AxkR7Odxzd2veiInw+VEJJaI=; b=xtHWJZ6yXa5EYEzqzGQ8Kv/EgHK3M7/rRDhWJCpUzAc+yOqmZS+crc1lnlbVcu9LZM9OOn 7mXc1ez8W1H5AXVMJVHe0GEMQlNhq0XZn5Whn3qBEt5R/57CZFgFFppYHdCBeQCbuAN5Y1 Fj4c4l22VH95BvBi0Y0SZrqtWlC87k8csYqFK27CheO8gJwEnwoQ+UA0EpHgYpMLAb2qtq voo0MptlsaZ8VhuaGH2S3U48tvBMCUC8SWiJXvcUo3iO5wYHNL91yAkapL5e9BqNLr/uP6 W/YojhxXIAXzDCNz0OYdggwCE0YmBVU8H+J/ku/61a7Q/K5sKU0aE8gZLqwwtg== ARC-Seal: i=1; s=outgoing-alex; d=leidinger.net; t=1566991222; a=rsa-sha256; cv=none; b=Vh0mtNCX9O6mxzzRlUfEpNc4MO42s5kf3pprYhU3wRx4zAnsTfxyWgaMC+MdOpTirAMKAM mca9/2nQvsXolN+22KJEd7rZYiEiSk+CJdX9xhZfeUFiqg5d60w5ICAwC4eBqn2m38GaRR n05xImAWJDAQsOTbjh+QURn7iQfM0r30nGbMptc8FZgKPj86VXpeyHWPkXjCLWQ8K0r7xC CJUjXIKcjxJzqJ4Fs++PpRfaEvCaRQxInY+pH8eVYYPZoy0m2SHcc3k19I/bZGTOixA+rk ZLCfOEelmdyJKNB4qHQ3vuqX7mqEue4pUbU4WuJ5I0Z/32j9DS4viKZ9ADJ5Dg== ARC-Authentication-Results: i=1; mailgate.Leidinger.net; auth=pass smtp.auth=netchild@leidinger.net smtp.mailfrom=Alexander@leidinger.net X-Rspamd-Queue-Id: 46JNpB02r7z45PN X-Spamd-Bar: --------- X-Spamd-Result: default: False [-9.79 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[leidinger.net:s=outgoing-alex]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; XAW_SERVICE_ACCT(1.00)[]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[leidinger.net:+]; DMARC_POLICY_ALLOW(-0.50)[leidinger.net,quarantine]; NEURAL_HAM_SHORT(-0.98)[-0.977,0]; IP_SCORE(-3.72)[ip: (-9.80), ipnet: 2a00:1828::/32(-4.89), asn: 34240(-3.88), country: DE(-0.01)]; SIGNED_PGP(-2.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[242.87.22.91.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:34240, ipnet:2a00:1828::/32, country:DE]; ARC_ALLOW(-1.00)[i=1]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Aug 2019 11:30:28 -0000 This message is in MIME format and has been PGP signed. --=_Pc7I2F4qlUaL4QpRwd9TYBt Content-Type: text/plain; charset=utf-8; format=flowed; DelSp=Yes Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Quoting Li-Wen Hsu (from Wed, 28 Aug 2019 12:29:58 +080= 0): > It seems I was doing wrong that just changed the content of this FCP > to "feedback", but did not send to the right mailing lists. > > So I would like to make an announcement that the FCP > 20190401-ci_policy "CI policy": > > https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md > > is officially in "feedback" state to hopefully receive more comments > and suggestions, then we can move on for the next FCP state. Last sentence in the paragraph of "Notification" needs a content fix,=20=20 there=20are several superfluous words... Waiting period: Why do you differentiate between "all platforms" and "a tier 1=20=20 platform"?=20By definition a tier 1 platform is the highest level of=20=20 platform=20we have and as such it has the highest priority, isn't it? 15 minutes is a little bit short in my opinion. Both items together: I suggest to remove the "all platforms"=20=20 completely=20without replacement and keep the rest as is. Bye, Alexander. --=20 http://www.Leidinger.net=20Alexander@Leidinger.net: PGP 0x8F31830F9F2772BF http://www.FreeBSD.org netchild@FreeBSD.org : PGP 0x8F31830F9F2772BF --=_Pc7I2F4qlUaL4QpRwd9TYBt Content-Type: application/pgp-signature Content-Description: Digitale PGP-Signatur Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAABAgAGBQJdZmNzAAoJEBINsJsD+NiG+sEP/iaMi2BTAEuaEkzI6O/HeY2R eNnxoRyp4mMnATq8jUe4eBZt2BQ1Qi6LPMcJpbMEmSOuteWS7VhuO4RuGJTzw9I5 PRSkEIX963rih/ktcrWfEMhbssOCykd2UEBMC71bQKmRtw1kpgso/EXJq21d6W1v /5ZQIMUTBm+X12zegW/PSm+DtrxefFq5fS3P1RGTVGTdzyFRQuj0+wUTD7zHaz6P 8SGw9IrIuWQDL/84EdnCMq44EyrGeF/rQnpVpFmmwNiR7Kl2POl1ZKWACdgSpmFR m7K9Xjjz97ec8oHBFYonUJoysAYZo40lkZvt3YcgCL64DCAeATdYLZjA0+f9QQkO O3E6gdaec/3/GF4r/pkYPzLNWdan80QuvbLj/tHmPTmRwYdpAKatY7ICty+T5sX2 nx/Ht7YSNJBkN0MmNS92bh7i+vNErOLqiQIkZEvUw+V+ysiMisFwDe8BqHl+pf61 T1Jw+kSy3Y/wwFYa3eMOy45lTTDDMz67Tg75XUfw8YXhgS1Kbgb0dK/GNxg7b3jT 8bDYD0i2KEQeOJ73YotFFe893W1qPibJngfW7YvKwuV4H1ncGDkvSdsLSuh+hgEB 2fW29krTt5BBCzfNTUpSfGPBGlGR9OgofN8O4QV8yL4ZIN8ATqSBNZd18DHleFcw V9AgGXNS9CHMAhlOkkdw =PU8L -----END PGP SIGNATURE----- --=_Pc7I2F4qlUaL4QpRwd9TYBt-- From owner-freebsd-fcp@freebsd.org Thu Aug 29 11:41:07 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C4AB2D3032 for ; Thu, 29 Aug 2019 11:41:07 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46K1033tXkz4XmP for ; Thu, 29 Aug 2019 11:41:07 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 8370CD3030; Thu, 29 Aug 2019 11:41:07 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8329DD302F; Thu, 29 Aug 2019 11:41:07 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46K1022Lh0z4XmM; Thu, 29 Aug 2019 11:41:05 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x7TBevg5062817 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 29 Aug 2019 14:41:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x7TBevg5062817 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x7TBevqY062816; Thu, 29 Aug 2019 14:40:57 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 29 Aug 2019 14:40:57 +0300 From: Konstantin Belousov To: Li-Wen Hsu Cc: fcp@freebsd.org, FreeBSD Hackers Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190829114057.GZ71821@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46K1022Lh0z4XmM X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.93 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; R_SPF_SOFTFAIL(0.00)[~all:c]; IP_SCORE_FREEMAIL(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.93)[-0.928,0]; IP_SCORE(0.00)[ip: (-2.60), ipnet: 2001:470::/32(-4.45), asn: 6939(-3.10), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 11:41:07 -0000 On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: > It seems I was doing wrong that just changed the content of this FCP > to "feedback", but did not send to the right mailing lists. > > So I would like to make an announcement that the FCP > 20190401-ci_policy "CI policy": > > https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md > > is officially in "feedback" state to hopefully receive more comments > and suggestions, then we can move on for the next FCP state. What problem does the document tries to solve ? Or rather, do we really have the problem that it claims to solve ? >From my experience, normal peer pressure is enough to get things fixed quickly when it is possible to fix them quickly. If there is something more non-trivial, esp. in the tests and not the build, I am sure that a rule allowing anybody to do blind revert is much more harmful than having a test broken. More, I know that tests are of very low quality, which means that brokeness of the tests is not an indicator of anything until root cause is identified. Can we rely on the common sense of developers until there is indeed the visible problem ? From owner-freebsd-fcp@freebsd.org Thu Aug 29 12:03:03 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 66C14D4545 for ; Thu, 29 Aug 2019 12:03:03 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K1TM1xwGz4ZQG for ; Thu, 29 Aug 2019 12:03:03 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 42B0AD4543; Thu, 29 Aug 2019 12:03:03 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4258ED4542; Thu, 29 Aug 2019 12:03:03 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K1TM0Ssxz4ZQF; Thu, 29 Aug 2019 12:03:03 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id BD55A1A4ED; Thu, 29 Aug 2019 12:03:02 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [10.0.2.193] (ptr-8rh08k12jlyqauo4swr.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:240e:402:c59e:f85c:1ed8:db5b]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id 59CDD3817F; Thu, 29 Aug 2019 14:03:01 +0200 (CEST) From: "Kristof Provost" To: "Konstantin Belousov" Cc: "Li-Wen Hsu" , "FreeBSD Hackers" , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Date: Thu, 29 Aug 2019 14:03:00 +0200 X-Mailer: MailMate (2.0BETAr6137) Message-ID: <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> In-Reply-To: <20190829114057.GZ71821@kib.kiev.ua> References: <20190829114057.GZ71821@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 12:03:03 -0000 On 29 Aug 2019, at 13:40, Konstantin Belousov wrote: > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: >> It seems I was doing wrong that just changed the content of this FCP >> to "feedback", but did not send to the right mailing lists. >> >> So I would like to make an announcement that the FCP >> 20190401-ci_policy "CI policy": >> >> https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md >> >> is officially in "feedback" state to hopefully receive more comments >> and suggestions, then we can move on for the next FCP state. > > What problem does the document tries to solve ? Or rather, do we > really > have the problem that it claims to solve ? > There are, somewhat regularly, commits which break functionality, or at the very least tests. The main objective of this policy proposal is to try to improve overall code quality by encouraging and empowering all committers to investigate and fix test failures. >> From my experience, normal peer pressure is enough to get things >> fixed > quickly when it is possible to fix them quickly. If there is something > more non-trivial, esp. in the tests and not the build, I am sure that > a rule allowing anybody to do blind revert is much more harmful than > having a test broken. > > More, I know that tests are of very low quality, which means that > brokeness of the tests is not an indicator of anything until root > cause > is identified. > I’m not sure I agree with the characterisation that the tests are of low quality. My own experience with the pf tests is that they test a large section of the network stack and firewall code. They’ve identified several very really issues (both pre- and post commit on the epoch-isatin of the network stack, for example, as well as a fairly important issue with IPv6 reassembly). It’s certainly true that the pf tests often reveal issues that are not in pf but in other code. I wouldn’t agree that this is a sign of low quality tests, but instead I consider it a sign that we don’t have enough tests for the network stack itself. > Can we rely on the common sense of developers until there is indeed > the > visible problem ? > I don’t want to suggest that people simply don’t care about test failures, because that’s clearly not true. On the other hand, I do think we can do better. There are at least two open problem in the network stack that I currently can’t get anyone to look at, and where I personally do not have sufficient context (or time) to fix them myself. (#239380, #238870). This proposal isn’t a silver bullet, I don’t think there is such a thing, but I do believe that elevating the visibility and importance of test failures can help us improve overall quality. Best regards, Kristof From owner-freebsd-fcp@freebsd.org Thu Aug 29 14:42:30 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6D661D78E7 for ; Thu, 29 Aug 2019 14:42:30 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K51L2KPTz3GCP for ; Thu, 29 Aug 2019 14:42:30 +0000 (UTC) (envelope-from ian@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 4F99FD78E5; Thu, 29 Aug 2019 14:42:30 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4F482D78E3 for ; Thu, 29 Aug 2019 14:42:30 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound2m.ore.mailhop.org (outbound2m.ore.mailhop.org [54.149.155.156]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46K51L0h8Kz3GCL for ; Thu, 29 Aug 2019 14:42:29 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1567089748; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=a/iixmRyANIz2XYKkMo+f6254i6U+Dj868uVtSpXjInXC50OteB6mcgXHseF4aKxlx59DaSO/LxHI WFlOWcyQVh3gQwBmVxoRllA0F8GdpXXZp3fPgWR09cBZjwYLGlMbJ3GqUypmRrToZSp0oGZomy82LA sAemnjbRJfsWI+Lpl0ADj1omv9lR28DiX9dYCaF+zOG4I0vZ0jkHwkApYXjwx2PBvxPhG/Ym4VsP3x lD/tynr3MChPNKUnT5NClfH5fq8mMvYGvDLvjCeUVF8/RUWDFeD0p09AAj1nRVHTNKEeXfeTqgzxEe EKa0MyJeLFJo17p7K8N9l6C0bu4e+7w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:dkim-signature:from; bh=q0A5Af+FjYXzUKhHiX5T2y2vMEbwVSRnnMr5Zyj6Azc=; b=hgA0dvajySNQtFfXVtA3Y5N0iCTIJg+xGYAUYP7lpYNaDDdc5wW+rngfoMGY22zq0ZgNS3AGRv7xW xJE2CgvvbH9jm8J5iMRPFGE6TuUybtdVY2s5JWFaNUz+HvNtRosdmMWqcvT8xGD/xKMT3Dv+RACH3+ F9hVo+xUMvBpTQQm7OXVQB3kUpeUkIO36vilvo9RZDuQnsGVM9b3SDgdoF/rKXsLe6MWD3bBg+KcxA 2xkouo6xC/FUrkWaKgrdM2747hIPN/nqnKpwxg0eYUJ4ZriUik+UyqlenjqyvMt3S43HXUa1iNbvHe wNd9u9er+dBiEWL7gmpTsKOx1vcwjqw== ARC-Authentication-Results: i=1; outbound4.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:cc:to:from:subject:message-id:from; bh=q0A5Af+FjYXzUKhHiX5T2y2vMEbwVSRnnMr5Zyj6Azc=; b=aQKiD4SUotZnnBulgPsFnO8j2mZHNV946DsBaHL7Fbvhzw4aaGrwPlkECSX2kYKd/nhC2Vmb5wDBu u55DpxWn+kAamJV9FoH1Wnp3D/TTcQmtNM4yoJbqb171jcS7EDKobQQU/SPBrzQAO48/yaY3WowBZR SzD9B9VKG+knKCLkZ2nmMAG8OS3GLVe0Xpot4P0c3W7te8XsLaxkOpJsn0zf3ATFuweU51qPVUXyNP 9Rg0Epo/Ra4WTrvWMVHX6VLoy/vpu4KTcdvVlCW3aI9b88AroL/pzn2Lx/lBc4mmDpfuQVpNQaMY8P +txhFAbszz3GsQlfhhoQYo6f5LPQNQA== X-MHO-RoutePath: aGlwcGll X-MHO-User: 3795f44d-ca6b-11e9-85ed-13b9aae3a1d2 X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (unknown [67.177.211.60]) by outbound4.ore.mailhop.org (Halon) with ESMTPSA id 3795f44d-ca6b-11e9-85ed-13b9aae3a1d2; Thu, 29 Aug 2019 14:42:26 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id x7TEgP25017390; Thu, 29 Aug 2019 08:42:25 -0600 (MDT) (envelope-from ian@freebsd.org) Message-ID: <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> Subject: Re: FCP 20190401-ci_policy: CI policy From: Ian Lepore To: Konstantin Belousov , Li-Wen Hsu Cc: FreeBSD Hackers , fcp@freebsd.org Date: Thu, 29 Aug 2019 08:42:25 -0600 In-Reply-To: <20190829114057.GZ71821@kib.kiev.ua> References: <20190829114057.GZ71821@kib.kiev.ua> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 46K51L0h8Kz3GCL X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-2.98 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; ASN(0.00)[asn:16509, ipnet:54.148.0.0/15, country:US]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 14:42:30 -0000 On Thu, 2019-08-29 at 14:40 +0300, Konstantin Belousov wrote: > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: > > It seems I was doing wrong that just changed the content of this FCP > > to "feedback", but did not send to the right mailing lists. > > > > So I would like to make an announcement that the FCP > > 20190401-ci_policy "CI policy": > > > > https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md > > > > is officially in "feedback" state to hopefully receive more comments > > and suggestions, then we can move on for the next FCP state. > > What problem does the document tries to solve ? Or rather, do we really > have the problem that it claims to solve ? > > From my experience, normal peer pressure is enough to get things fixed > quickly when it is possible to fix them quickly. If there is something > more non-trivial, esp. in the tests and not the build, I am sure that > a rule allowing anybody to do blind revert is much more harmful than > having a test broken. > > More, I know that tests are of very low quality, which means that > brokeness of the tests is not an indicator of anything until root cause > is identified. > > Can we rely on the common sense of developers until there is indeed the > visible problem ? > I totally agree. This is an overly-bureaucratic solution in search of a problem. If this needs to be addressed at all (and I'm not sure it does), then another sentence or two in bullet item 10 in section 18.1 [*] of the committer's guide should be enough. And even then it needn't be overly-formal and should just mention that if a commit does break the build the committer is expected to be responsive to that problem and the commit might get reverted if they're unresponsive. I don't think we need schedules. (And I don't think breaking a test counts as breaking the build.) [*] https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/rules.html -- Ian From owner-freebsd-fcp@freebsd.org Thu Aug 29 14:42:41 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 38E2FD7918 for ; Thu, 29 Aug 2019 14:42:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K51X3xVxz3GGx for ; Thu, 29 Aug 2019 14:42:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 86D74D7914; Thu, 29 Aug 2019 14:42:40 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 85F9CD7913; Thu, 29 Aug 2019 14:42:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46K51W4wk8z3GG9; Thu, 29 Aug 2019 14:42:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x7TEgSpf005877 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 29 Aug 2019 17:42:31 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x7TEgSpf005877 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x7TEgS0o005876; Thu, 29 Aug 2019 17:42:28 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 29 Aug 2019 17:42:28 +0300 From: Konstantin Belousov To: Kristof Provost Cc: Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190829144228.GA71821@kib.kiev.ua> References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46K51W4wk8z3GG9 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.91 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.91)[-0.910,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 14:42:41 -0000 On Thu, Aug 29, 2019 at 02:03:00PM +0200, Kristof Provost wrote: > On 29 Aug 2019, at 13:40, Konstantin Belousov wrote: > > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: > >> It seems I was doing wrong that just changed the content of this FCP > >> to "feedback", but did not send to the right mailing lists. > >> > >> So I would like to make an announcement that the FCP > >> 20190401-ci_policy "CI policy": > >> > >> https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md > >> > >> is officially in "feedback" state to hopefully receive more comments > >> and suggestions, then we can move on for the next FCP state. > > > > What problem does the document tries to solve ? Or rather, do we > > really > > have the problem that it claims to solve ? > > > There are, somewhat regularly, commits which break functionality, or at > the very least tests. > The main objective of this policy proposal is to try to improve overall > code quality by encouraging and empowering all committers to investigate > and fix test failures. But this policy does not encourage, if anything. It gives a free ticket to revert, discouraging committers. > > >> From my experience, normal peer pressure is enough to get things > >> fixed > > quickly when it is possible to fix them quickly. If there is something > > more non-trivial, esp. in the tests and not the build, I am sure that > > a rule allowing anybody to do blind revert is much more harmful than > > having a test broken. > > > > More, I know that tests are of very low quality, which means that > > brokeness of the tests is not an indicator of anything until root > > cause > > is identified. > > > I’m not sure I agree with the characterisation that the tests are of > low quality. My own experience with the pf tests is that they test a > large section of the network stack and firewall code. They’ve > identified several very really issues (both pre- and post commit on the > epoch-isatin of the network stack, for example, as well as a fairly > important issue with IPv6 reassembly). This is my experience with the tests for kernel/libc/libthr, most of which comes from contrib/netbsd-tests, as I understand. > It’s certainly true that the pf tests often reveal issues that are not > in pf but in other code. I wouldn’t agree that this is a sign of low > quality tests, but instead I consider it a sign that we don’t have > enough tests for the network stack itself. > > > Can we rely on the common sense of developers until there is indeed > > the > > visible problem ? > > > I don’t want to suggest that people simply don’t care about test > failures, because that’s clearly not true. > > On the other hand, I do think we can do better. There are at least two > open problem in the network stack that I currently can’t get anyone to > look at, and where I personally do not have sufficient context (or time) > to fix them myself. (#239380, #238870). > > This proposal isn’t a silver bullet, I don’t think there is such a > thing, but I do believe that elevating the visibility and importance of > test failures can help us improve overall quality. This is not what was proposed. I fully agree that build failures should be fixed ASAP, and the each test results change (note the formulation) should be examined. But I do not think we have the problem right now of a scale which requires the free pass to revert with 4h timer. From owner-freebsd-fcp@freebsd.org Thu Aug 29 14:44:51 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F315FD7BCE for ; Thu, 29 Aug 2019 14:44:51 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K5433xWQz3GZb for ; Thu, 29 Aug 2019 14:44:51 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 87177D7BC9; Thu, 29 Aug 2019 14:44:51 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 86BE9D7BC8; Thu, 29 Aug 2019 14:44:51 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-lf1-x143.google.com (mail-lf1-x143.google.com [IPv6:2a00:1450:4864:20::143]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K54331jvz3GZZ; Thu, 29 Aug 2019 14:44:51 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mail-lf1-x143.google.com with SMTP id r5so2760170lfc.3; Thu, 29 Aug 2019 07:44:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:reply-to:from:date:message-id :subject:to:cc; bh=vi6RluLvB3RVtOqMSwTBcCPHr3wMB6PjNtXI4mFu7xg=; b=CrqzX0hxzJTSQsxZAHboaUU8u3dqR11K6LpJ9jwdq73F9oUhj4CyNeuTNuEnZPnqRK VNrNsNiO9Qm25nw62K00nG3f3R2+b+Aq32tAYNPiRKyQq4WtEIvB2yb509mNJp2gyY4L B+djE2yXtmqcHFZmghtEOGO7kkbIQmhLMoNhaPEXuhMIydqrXASMw5eB9W3jMqZdGIAi w/UrGMPZpteOqaSkfbmWSG+iHfgqaY9HycWcvgv1ryyyGhB1LPEb2j2+uycQD68pOvA0 hyZcZ5JAUaSK0q+YdTafFjgB41LPbHDR3J8jU369ygq1BVxmdKsk+rw1CymZLtKWykru KrYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to:cc; bh=vi6RluLvB3RVtOqMSwTBcCPHr3wMB6PjNtXI4mFu7xg=; b=dAtz8ut1UbkjdaNDJROUY/QJezK9em42OwfEJSWYxW8k1iqYNU+shgfM9zkl3TWX9K n65hlFVIhevM0BvB6/b0D3BPFpBorIMSXp4hDg0/r8DhmmcxPewFj63xzO9+gaFq5rrN 60bcRhS2IETfrb9zTR9sHQprjjkQYYzxX688nE5Q3kWJB1CO4xeo/SGMCF7GH+OfNWwm NVUMq+PvywV1gHrbG29/N267IeA1k4xdHQrBAzK+YERaWjukgTDfuXL9Cx/eCBhjN5mB KVF9+mqUG8B4NRzimvqYzYnVcWIqkE5vJ2gRIjwHgz7d9m7ndvbtBORzb3cWoaW5nmB7 wBzQ== X-Gm-Message-State: APjAAAVHgt6kUJT5rxAtZ3mIe+GU145Df2W/VmLKyVynYR4fBGpEP4gn mJtFu/G0+gWVJk255DejNPVboxZ97yF8cugEkcdlFeJ7DtU= X-Google-Smtp-Source: APXvYqyms0ZjwLVbkRFOQGGIENLSitckxYGyqhD4idJWirjrbzZH5VtzzWPtEQ2ybke4wT7q0ovg6JMYc79rGdWApz0= X-Received: by 2002:a19:674d:: with SMTP id e13mr6463979lfj.176.1567089889352; Thu, 29 Aug 2019 07:44:49 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> In-Reply-To: <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> Reply-To: araujo@freebsd.org From: Marcelo Araujo Date: Thu, 29 Aug 2019 22:44:38 +0800 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Kristof Provost Cc: Konstantin Belousov , FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org X-Rspamd-Queue-Id: 46K54331jvz3GZZ X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.98)[-0.983,0] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 14:44:52 -0000 Em qui, 29 de ago de 2019 =C3=A0s 20:03, Kristof Provost escreveu: > On 29 Aug 2019, at 13:40, Konstantin Belousov wrote: > > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: > >> It seems I was doing wrong that just changed the content of this FCP > >> to "feedback", but did not send to the right mailing lists. > >> > >> So I would like to make an announcement that the FCP > >> 20190401-ci_policy "CI policy": > >> > >> https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md > >> > >> is officially in "feedback" state to hopefully receive more comments > >> and suggestions, then we can move on for the next FCP state. > > > > What problem does the document tries to solve ? Or rather, do we > > really > > have the problem that it claims to solve ? > > > There are, somewhat regularly, commits which break functionality, or at > the very least tests. > The main objective of this policy proposal is to try to improve overall > code quality by encouraging and empowering all committers to investigate > and fix test failures. > Sure, but it doesn't sounds like you are empowering people to works in their spare time for the project, quite the opposite. Could you show something more feasible than "somewhat" regularly breaks functionality? Most of the time personally I'm running HEAD, built daily and I can't see this. > > >> From my experience, normal peer pressure is enough to get things > >> fixed > > quickly when it is possible to fix them quickly. If there is something > > more non-trivial, esp. in the tests and not the build, I am sure that > > a rule allowing anybody to do blind revert is much more harmful than > > having a test broken. > > > > More, I know that tests are of very low quality, which means that > > brokeness of the tests is not an indicator of anything until root > > cause > > is identified. > > > I=E2=80=99m not sure I agree with the characterisation that the tests are= of > low quality. My own experience with the pf tests is that they test a > large section of the network stack and firewall code. They=E2=80=99ve > identified several very really issues (both pre- and post commit on the > epoch-isatin of the network stack, for example, as well as a fairly > important issue with IPv6 reassembly). > It=E2=80=99s certainly true that the pf tests often reveal issues that ar= e not > in pf but in other code. I wouldn=E2=80=99t agree that this is a sign of = low > quality tests, but instead I consider it a sign that we don=E2=80=99t hav= e > enough tests for the network stack itself. > All the tests and CI are pretty new, or at least it is something new for most of everybody and people are getting used to that. I stop here, I would elaborate more, but after Ian's email, I think I don't need anymore. > > > Can we rely on the common sense of developers until there is indeed > > the > > visible problem ? > > > I don=E2=80=99t want to suggest that people simply don=E2=80=99t care abo= ut test > failures, because that=E2=80=99s clearly not true. > > On the other hand, I do think we can do better. There are at least two > open problem in the network stack that I currently can=E2=80=99t get anyo= ne to > look at, and where I personally do not have sufficient context (or time) > to fix them myself. (#239380, #238870). > > This proposal isn=E2=80=99t a silver bullet, I don=E2=80=99t think there = is such a > thing, but I do believe that elevating the visibility and importance of > test failures can help us improve overall quality. > > Best regards, > Kristof > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " > --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-fcp@freebsd.org Thu Aug 29 14:47:24 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D40DCD7E2A for ; Thu, 29 Aug 2019 14:47:24 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K5704tlyz3Gsf for ; Thu, 29 Aug 2019 14:47:24 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id A5BEBD7E28; Thu, 29 Aug 2019 14:47:24 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A5632D7E27; Thu, 29 Aug 2019 14:47:24 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-lj1-x243.google.com (mail-lj1-x243.google.com [IPv6:2a00:1450:4864:20::243]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K56z687Cz3Gsd; Thu, 29 Aug 2019 14:47:23 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mail-lj1-x243.google.com with SMTP id x18so3352621ljh.1; Thu, 29 Aug 2019 07:47:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:reply-to:from:date:message-id :subject:to:cc; bh=qT+mBqubGxPBox80K474SvmrCGrQtlul6W9MHeIFwC8=; b=CytIhV5xnNr/yCULsKxKxTymlq6rTy8wwtnbqBsDp4YyMzNu3UDSaER18xohAb60NM g4mbQgmaJH87byh/ZWOBT3EdBs1Wj6EhC6O7VR3YMn+F5pjiGEkwBHQYr9KaOMrISAQH XIoZc5ZLsluAvkxXTh5NkDXqZMspQ3rbzSVhYwP/Jk4s0Xp4H9UfiyXRTH/9FoC95iIu CIBNK7KNDFMPmXPSzBcOSuIs9SbtVu5c1qfkTDRENP74lwt0klU49sx2P5A2dSzoUGp9 ZObLsEdvcdeFDW3BMhUbkEv00r9HUsW8VvngMhhGVusy7grwYaHkff7rhUMFsoDpdYSS l5Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to:cc; bh=qT+mBqubGxPBox80K474SvmrCGrQtlul6W9MHeIFwC8=; b=m0gQsAyR7SHyOORTzqDrhb1j1PtyYqRoK92Tm/ttv75tBpd6/AifN9yy3HTmhRPcBq rIn2LQ33gPWz5ht8F+KdLygcMbfz3i7JGsQYGCzGtCct69jCPkQBKD/OoTVkI5d+uSGG E6TkH59dKenKOD108n/LI9bwkfOOaVY8Fd2tSO8QCthZprKoXzi+78oHcpiAVQ+GEznP YmgEYyWfpvezwi5RI6AfFoh1lkXZvLkciZqsdc6BK3/GGyw0n+Kr6/faIe518J+048Rz sbdjbwnADTN4VVROF3sjgqzqfl+96VOpztGFJ2EBmDW+snI9SMGwjUEhCM1w6Oky3GpW Ru4Q== X-Gm-Message-State: APjAAAV4MsAFIgpyP9HQmlsAZm/kYxxNw/kwLz1OoI03cpTt6y0G9ssS YU1nTuO9T3wD/B+6rC2Uh+/DPfkOx+c2wYdBHG3DTJgzBQE= X-Google-Smtp-Source: APXvYqzhE8GwoiaGoth1QcocgSL9rLaPDRyZVlcvTprxsdI5quQEiQcpJHSVFcvcQEjYCKb5HJiF78TsSZxJWBUq1+o= X-Received: by 2002:a2e:81ca:: with SMTP id s10mr5649243ljg.181.1567090041764; Thu, 29 Aug 2019 07:47:21 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> In-Reply-To: Reply-To: araujo@freebsd.org From: Marcelo Araujo Date: Thu, 29 Aug 2019 22:47:10 +0800 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Kristof Provost Cc: Konstantin Belousov , FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org X-Rspamd-Queue-Id: 46K56z687Cz3Gsd X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=CytIhV5x; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of araujobsdport@gmail.com designates 2a00:1450:4864:20::243 as permitted sender) smtp.mailfrom=araujobsdport@gmail.com X-Spamd-Result: default: False [-3.97 / 15.00]; HAS_REPLYTO(0.00)[araujo@freebsd.org]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_FIVE(0.00)[5]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.97)[-0.968,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (3.15), ipnet: 2a00:1450::/32(-2.99), asn: 15169(-2.32), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[3.4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_CC(0.00)[gmail.com]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 14:47:24 -0000 Em qui, 29 de ago de 2019 =C3=A0s 22:44, Marcelo Araujo escreveu: > > > Em qui, 29 de ago de 2019 =C3=A0s 20:03, Kristof Provost > escreveu: > >> On 29 Aug 2019, at 13:40, Konstantin Belousov wrote: >> > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: >> >> It seems I was doing wrong that just changed the content of this FCP >> >> to "feedback", but did not send to the right mailing lists. >> >> >> >> So I would like to make an announcement that the FCP >> >> 20190401-ci_policy "CI policy": >> >> >> >> https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md >> >> >> >> is officially in "feedback" state to hopefully receive more comments >> >> and suggestions, then we can move on for the next FCP state. >> > >> > What problem does the document tries to solve ? Or rather, do we >> > really >> > have the problem that it claims to solve ? >> > >> There are, somewhat regularly, commits which break functionality, or at >> the very least tests. >> The main objective of this policy proposal is to try to improve overall >> code quality by encouraging and empowering all committers to investigate >> and fix test failures. >> > > Sure, but it doesn't sounds like you are empowering people to works in > their spare time for the project, quite the opposite. > Could you show something more feasible than "somewhat" regularly breaks > functionality? Most of the time personally I'm running HEAD, built daily > and I can't see this. > > >> >> >> From my experience, normal peer pressure is enough to get things >> >> fixed >> > quickly when it is possible to fix them quickly. If there is something >> > more non-trivial, esp. in the tests and not the build, I am sure that >> > a rule allowing anybody to do blind revert is much more harmful than >> > having a test broken. >> > >> > More, I know that tests are of very low quality, which means that >> > brokeness of the tests is not an indicator of anything until root >> > cause >> > is identified. >> > >> I=E2=80=99m not sure I agree with the characterisation that the tests ar= e of >> low quality. My own experience with the pf tests is that they test a >> large section of the network stack and firewall code. They=E2=80=99ve >> identified several very really issues (both pre- and post commit on the >> epoch-isatin of the network stack, for example, as well as a fairly >> important issue with IPv6 reassembly). >> It=E2=80=99s certainly true that the pf tests often reveal issues that a= re not >> in pf but in other code. I wouldn=E2=80=99t agree that this is a sign of= low >> quality tests, but instead I consider it a sign that we don=E2=80=99t ha= ve >> enough tests for the network stack itself. >> > > All the tests and CI are pretty new, or at least it is something new for > most of everybody and people are getting used to that. > > I stop here, I would elaborate more, but after Ian's email, I think I > don't need anymore. > One last thing I'd like to say: We should stop to try to control FreeBSD contributions, it is not doing good for the project. Let it be, let it go, there is no owner, just go and fix things. > > >> >> > Can we rely on the common sense of developers until there is indeed >> > the >> > visible problem ? >> > >> I don=E2=80=99t want to suggest that people simply don=E2=80=99t care ab= out test >> failures, because that=E2=80=99s clearly not true. >> >> On the other hand, I do think we can do better. There are at least two >> open problem in the network stack that I currently can=E2=80=99t get any= one to >> look at, and where I personally do not have sufficient context (or time) >> to fix them myself. (#239380, #238870). >> >> This proposal isn=E2=80=99t a silver bullet, I don=E2=80=99t think there= is such a >> thing, but I do believe that elevating the visibility and importance of >> test failures can help us improve overall quality. >> >> Best regards, >> Kristof >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.or= g >> " >> > > > -- > > -- > Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.Fr= eeBSD.org \/ \ ^ > Power To Server. .\. /_) > > --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-fcp@freebsd.org Thu Aug 29 14:54:29 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B397CD8177 for ; Thu, 29 Aug 2019 14:54:29 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46K5H94D05z3HSD for ; Thu, 29 Aug 2019 14:54:29 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 90CE2D8175; Thu, 29 Aug 2019 14:54:29 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 90849D8174; Thu, 29 Aug 2019 14:54:29 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5H92v8Mz3HSC; Thu, 29 Aug 2019 14:54:29 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 162311BEA9; Thu, 29 Aug 2019 14:54:29 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [10.0.2.193] (ptr-8rh08k12jlyqauo4swr.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:240e:402:c59e:f85c:1ed8:db5b]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id 798F73858E; Thu, 29 Aug 2019 16:54:27 +0200 (CEST) From: "Kristof Provost" To: "Ian Lepore" Cc: "Konstantin Belousov" , "Li-Wen Hsu" , "FreeBSD Hackers" , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Date: Thu, 29 Aug 2019 16:54:26 +0200 X-Mailer: MailMate (2.0BETAr6137) Message-ID: In-Reply-To: <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> References: <20190829114057.GZ71821@kib.kiev.ua> <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 14:54:29 -0000 On 29 Aug 2019, at 16:42, Ian Lepore wrote: > (And I don't think breaking a test counts as > breaking the build.) > I fundamentally disagree on this point. A test failure is, just like a compiler warning, a precious gift that should not be ignored. The more distance (both in terms of time, and in terms of the people involved) there is between a bug being introduced and it being detected the harder it is to fix it. Test accelerate detection of bugs. If we do not take test failures seriously (i.e. as an indication something is wrong and should be fixed) the tests will inevitable become useless in one of two ways: we’ll either disable failing tests (which is what we tend to do now) reducing test coverage or we’ll have a test suite with many failures in it, which makes it useless as well. (As with compiler warnings, the best way to keep them under control is to consider them to be fatal errors.) In either scenario we end up reducing test coverage, which means we’re going to push more bugs towards users. > I totally agree. This is an overly-bureaucratic solution in search of > a problem. > > If this needs to be addressed at all (and I'm not sure it does), then > another sentence or two in bullet item 10 in section 18.1 [*] of the > committer's guide should be enough. And even then it needn't be > overly-formal and should just mention that if a commit does break the > build the committer is expected to be responsive to that problem and > the commit might get reverted if they're unresponsive. I don't think > we need schedules. > I do feel that’s a better argument. We’ve always had a policy of reverting on request (AIUI), so this is more or less trying to be a strong restatement of that, more than a fundamental shift in policy. Best regards, Kristof From owner-freebsd-fcp@freebsd.org Thu Aug 29 15:01:31 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C9126D86A8 for ; Thu, 29 Aug 2019 15:01:31 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46K5RH3cwWz3J1T for ; Thu, 29 Aug 2019 15:01:31 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 7C32AD86A4; Thu, 29 Aug 2019 15:01:31 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7BDA7D86A3; Thu, 29 Aug 2019 15:01:31 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-lj1-x242.google.com (mail-lj1-x242.google.com [IPv6:2a00:1450:4864:20::242]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5RH2jN9z3J1S; Thu, 29 Aug 2019 15:01:30 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mail-lj1-x242.google.com with SMTP id x3so3371009lji.5; Thu, 29 Aug 2019 08:01:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:reply-to:from:date:message-id :subject:to:cc; bh=3ZnXP+zKpmB0LU+a5ZZE56+58oJErxdNO5Cw30lJ9Nc=; b=iyca442rNhBaC5GvY3I1J6rgHOTFCFbvwuCShgpJxVmZb74FqKKdh5IsPeyVEt6dCR oifbq6ShTCSnFsiQ07i3/j/FcQIGs7kzPfYWi5OmbQj6c51POInk6/2x3PstpgX3BKIR uXTDUv8qPwg726H4f+ZAkA99OXf2//jVd/TN4zRNHHJigUoKbiY+fzd5GuO7aWg7lCCB Y+MXnREBJ2nFOo/h1p1jU2Lbg+olpTuXjzUtvGFmLLRuMy9gEcG3AG/EvnkRCHmLsZA2 +LVmcv9XLTJkiCYgO7qQbCJFw/ZBmD4NTY8WUxsVRPbmtPcV3IKW/rMrYNCHc1mGi3mh EapQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to:cc; bh=3ZnXP+zKpmB0LU+a5ZZE56+58oJErxdNO5Cw30lJ9Nc=; b=hENPB6nko6iBiLvVo8/4XF3UQddmaLI6vzsy9wDfsTc65xx4SF5Pj4hjPOCJwM0PIm Xsm3AbxiuSkvUXOiyi0Pi5kmHwIHpRE3Ot//sWfc4TY87h4XX0mbF0mLwj7dMA4gJXpL 1PEC1E4x+vh7Qn+iiMp4nypDImsgO4SLndnP2gpKPMAHK1DoZkqisCly+WzdgT27M93C 3Gl5+xFh7pdINtw4TDI7XulxETwki2+2pkmXb13y2TgkXDRLqJfUYB3Ws2JvbUjAFqN3 D4dx4HMPsjRnBgAZy8IZI53xczNxSXPkZYUl9OMJ18Ph5gK/5mBlwaRaR6Qkc6Wsg8ML ig6Q== X-Gm-Message-State: APjAAAVmzLZcsSBnLBJkyV5M6AgDn7VpOmwpDe0iOV64TASvXXrob4Zd WKvze9TAG/K9uDA0Tiq7fhO9IzJIo3G5BSl7v06UnYdmnPY= X-Google-Smtp-Source: APXvYqzv8wycrFM8IQuvkvHTJcg4qdljRE7bJUUQ1m0UndetIGa91MEK55ybN55nm3tOrLGoUh5OdxFsxtcAOpMGeSw= X-Received: by 2002:a2e:81ca:: with SMTP id s10mr5692168ljg.181.1567090889056; Thu, 29 Aug 2019 08:01:29 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> In-Reply-To: Reply-To: araujo@freebsd.org From: Marcelo Araujo Date: Thu, 29 Aug 2019 23:01:17 +0800 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Kristof Provost Cc: Ian Lepore , Konstantin Belousov , FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org X-Rspamd-Queue-Id: 46K5RH2jN9z3J1S X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 15:01:31 -0000 Em qui, 29 de ago de 2019 =C3=A0s 22:54, Kristof Provost escreveu: > On 29 Aug 2019, at 16:42, Ian Lepore wrote: > > (And I don't think breaking a test counts as > > breaking the build.) > > > I fundamentally disagree on this point. A test failure is, just like a > compiler warning, a precious gift that should not be ignored. > The more distance (both in terms of time, and in terms of the people > involved) there is between a bug being introduced and it being detected > the harder it is to fix it. Test accelerate detection of bugs. If we do > not take test failures seriously (i.e. as an indication something is > wrong and should be fixed) the tests will inevitable become useless in > one of two ways: we=E2=80=99ll either disable failing tests (which is wha= t we > tend to do now) reducing test coverage or we=E2=80=99ll have a test suite= with > many failures in it, which makes it useless as well. (As with compiler > warnings, the best way to keep them under control is to consider them to > be fatal errors.) > Could you elaborate where is the "fundamentally" you disagree? Where is the fundament? You guys are introducing something new, yes everybody knows about test, it is year 2019, but nobody can come with new rules tha in hours we gonna revert if you "dare to don't fix it". Sorry, this is not how people test software and fix it. > > In either scenario we end up reducing test coverage, which means we=E2=80= =99re > going to push more bugs towards users. > > > I totally agree. This is an overly-bureaucratic solution in search of > > a problem. > > > > If this needs to be addressed at all (and I'm not sure it does), then > > another sentence or two in bullet item 10 in section 18.1 [*] of the > > committer's guide should be enough. And even then it needn't be > > overly-formal and should just mention that if a commit does break the > > build the committer is expected to be responsive to that problem and > > the commit might get reverted if they're unresponsive. I don't think > > we need schedules. > > > I do feel that=E2=80=99s a better argument. We=E2=80=99ve always had a po= licy of > reverting on request (AIUI), so this is more or less trying to be a > strong restatement of that, more than a fundamental shift in policy. > We don't have a policy to revert commit, actually revert commit is something bad, it is kind of punishment, I have been there, nobody wants to be there. Stop to push this non-sense argument. > > Best regards, > Kristof > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " > --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-fcp@freebsd.org Thu Aug 29 15:02:50 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D6B48D88C4 for ; Thu, 29 Aug 2019 15:02:50 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46K5Sp4LJ0z3JJD for ; Thu, 29 Aug 2019 15:02:50 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id 94A14D88C2; Thu, 29 Aug 2019 15:02:50 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 944C8D88C1; Thu, 29 Aug 2019 15:02:50 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5Sp3LJKz3JJC; Thu, 29 Aug 2019 15:02:50 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 25CDC1BFE8; Thu, 29 Aug 2019 15:02:50 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [10.0.2.193] (ptr-8rh08k12jlyqauo4swr.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:240e:402:c59e:f85c:1ed8:db5b]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id A83F6385CC; Thu, 29 Aug 2019 17:02:48 +0200 (CEST) From: "Kristof Provost" To: "Konstantin Belousov" Cc: "Li-Wen Hsu" , "FreeBSD Hackers" , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Date: Thu, 29 Aug 2019 17:02:47 +0200 X-Mailer: MailMate (2.0BETAr6137) Message-ID: In-Reply-To: <20190829144228.GA71821@kib.kiev.ua> References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> <20190829144228.GA71821@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed; markup=markdown Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 15:02:50 -0000 On 29 Aug 2019, at 16:42, Konstantin Belousov wrote: > On Thu, Aug 29, 2019 at 02:03:00PM +0200, Kristof Provost wrote: >> There are, somewhat regularly, commits which break functionality, or >> at >> the very least tests. >> The main objective of this policy proposal is to try to improve >> overall >> code quality by encouraging and empowering all committers to >> investigate >> and fix test failures. > But this policy does not encourage, if anything. > It gives a free ticket to revert, discouraging committers. > To provide a counterpoint here: my personal frustration right now is that I’ve spent a good bit of time adding tests for pf and fixing bugs for it, only to see the tests having to be disabled because of unrelated (to pf) changes in the network stack. Either through lack of visibility, or lack of time, or because people assume pf tests failures must by definition be the responsibility of the pf maintainer, these failures have not been investigated by anyone other than me, and I lack the time and subject matter expertise to fix them. I’m desperately afraid that if/when these bugs do get fixed we’re going to discover that other things have broken in the mean time, and the tests are still going to fail, for different reasons. These are bugs. They’re the best case scenario for bug reports even, because they come with a reproduction case built-in, and yet they’re still not getting fixed. This too is discouraging. I’m open to alternative proposals for how to address that problem, but I don’t think that “continue on as we always have” is the correct answer. Best regards, Kristof From owner-freebsd-fcp@freebsd.org Thu Aug 29 15:08:50 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6E934D8ABD for ; Thu, 29 Aug 2019 15:08:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K5bk1bJfz3JZD for ; Thu, 29 Aug 2019 15:08:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.nyi.freebsd.org (Postfix) id 349BCD8ABC; Thu, 29 Aug 2019 15:08:50 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 345DFD8ABB for ; Thu, 29 Aug 2019 15:08:50 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x831.google.com (mail-qt1-x831.google.com [IPv6:2607:f8b0:4864:20::831]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5bj43gmz3JZ8 for ; Thu, 29 Aug 2019 15:08:49 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x831.google.com with SMTP id b11so4007322qtp.10 for ; Thu, 29 Aug 2019 08:08:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=2Wc2bEFcg8n1w6AcR/y2lTDsyKi+cVQ4C3w3oWA0M4U=; b=PgDmVkDkJHnFZ1HuuhjK5rT3egZ6RQDY8qFiw6tRNUmX5Y+Q/xd7JHcymuynH7dX0c EiDyXRsKF3tPgjYALwY1x8gTZK8Tur5O3BkrN7DYuQ9yDfgIlSIePKTD905Kty6en3NV n3kyPJ5dC703+lJudbChMUHR1+GpQ0kKaVLq9cDx97SewRIUs1Fw6rFwJpG487ffRXXR Iquw6yXFLQTcAcD14Tm6/UHeCwnq1FJuK7tTSLS/hoEwOauGXMHQ5t61wUs7z2rSJzW4 nHQAwSmdpwM3E90mENjXVAbSsba46+lLlWl/jqJPzDNeLqkZvut1wOi8W9lObR/JJGWP WUxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=2Wc2bEFcg8n1w6AcR/y2lTDsyKi+cVQ4C3w3oWA0M4U=; b=Cy6BzmI18TbLQugGWztu/srStOnOFTz2glkovm+E265y36JgR/VM1DjQMPWrGaJzUC 5EfwIG4xIilBErJY3m1BwABD+Tn6Obo4usZcrgmIIit20RX+2miTh1AVUM4s+3/XDNUX LY5a9ElLslbGj2f0AV44gsOzXawUEJyaDNLBpw64e6lQPAwFY1jiC5jzo4XZjORWqNbt AAwfgSYtjPBiSZHbVxF5oqLIgp7xUs2AYgWi2vx/qvfIP4HChkRWjd42YVWFgeTN5cGm vCxN3PcCLlOaC3xdTWZ4X3aaeMfTtAlnNRgCacAINRTy/wU4YwpsPQ7GXBnMYs1tqJbk +o6w== X-Gm-Message-State: APjAAAWqrTexXMEKex/IWtf/wfEja33B5Wy6goVp5/PqMKGeIEDFQlWx bbOy6N1PNer+SyOyHS+OczF37i1Fe9aeyNM6QualUatoME0= X-Google-Smtp-Source: APXvYqw6prGe0bR/jCOJ4nFhjgXq7WN7WbMmL5Wx3W/Pfn7FVwkvIS8gEJxjw0w98CtkLmv8JQLzXHUuxSy9EVJy/Lc= X-Received: by 2002:ac8:4602:: with SMTP id p2mr10203695qtn.291.1567091328412; Thu, 29 Aug 2019 08:08:48 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> In-Reply-To: <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> From: Warner Losh Date: Thu, 29 Aug 2019 09:08:37 -0600 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Ian Lepore Cc: Konstantin Belousov , Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org X-Rspamd-Queue-Id: 46K5bj43gmz3JZ8 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=PgDmVkDk; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::831) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-5.91 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[fcp@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; RCVD_IN_DNSWL_NONE(0.00)[1.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.91)[ip: (-9.34), ipnet: 2607:f8b0::/32(-2.85), asn: 15169(-2.32), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; FREEMAIL_CC(0.00)[gmail.com] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 15:08:50 -0000 On Thu, Aug 29, 2019 at 8:42 AM Ian Lepore wrote: > On Thu, 2019-08-29 at 14:40 +0300, Konstantin Belousov wrote: > > On Wed, Aug 28, 2019 at 12:29:58PM +0800, Li-Wen Hsu wrote: > > > It seems I was doing wrong that just changed the content of this FCP > > > to "feedback", but did not send to the right mailing lists. > > > > > > So I would like to make an announcement that the FCP > > > 20190401-ci_policy "CI policy": > > > > > > https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md > > > > > > is officially in "feedback" state to hopefully receive more comments > > > and suggestions, then we can move on for the next FCP state. > > > > What problem does the document tries to solve ? Or rather, do we really > > have the problem that it claims to solve ? > > > > From my experience, normal peer pressure is enough to get things fixed > > quickly when it is possible to fix them quickly. If there is something > > more non-trivial, esp. in the tests and not the build, I am sure that > > a rule allowing anybody to do blind revert is much more harmful than > > having a test broken. > > > > More, I know that tests are of very low quality, which means that > > brokeness of the tests is not an indicator of anything until root cause > > is identified. > > > > Can we rely on the common sense of developers until there is indeed the > > visible problem ? > > > > I totally agree. This is an overly-bureaucratic solution in search of > a problem. > > If this needs to be addressed at all (and I'm not sure it does), then > another sentence or two in bullet item 10 in section 18.1 [*] of the > committer's guide should be enough. And even then it needn't be > overly-formal and should just mention that if a commit does break the > build the committer is expected to be responsive to that problem and > the commit might get reverted if they're unresponsive. I don't think > we need schedules. (And I don't think breaking a test counts as > breaking the build.) > > [*] > https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/rules.html There's been growing friction around these points in the past several years. We should document that tree breakages aren't acceptable, committers are expected to fix things as soon as possible, and that new broken tests have to be investigated quickly, or the change should be reverted. This document tries to attach some suggested time frames on different types of breakage we've seen and set the expectations for people when things go wrong. Having different categories also allows us to set expectations for external toolchain breakage that's more lax than in-tree toolchain breakage, for example. The future will have external toolchain CI and notifications will likely be turned on when there's breakage. While some tests are poorly written, we should disable / remove the ones that are actually bad rather than create a policy that tolerates bad tests by setting no expectation around that. Now, one may quibble over the timelines, but that discussion helps everybody in the community know what the expectations are when committing and enables people that want to scrimp on testing to do so if they know they will be around for any unanticipated fallout, but may also encourage others to test more because they know they might not want to be. Warner From owner-freebsd-fcp@freebsd.org Thu Aug 29 15:09:03 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 23101D8B17 for ; Thu, 29 Aug 2019 15:09:03 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K5by67fcz3JfX for ; Thu, 29 Aug 2019 15:09:02 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id D282BD8B13; Thu, 29 Aug 2019 15:09:02 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D22BED8B11; Thu, 29 Aug 2019 15:09:02 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-lj1-x243.google.com (mail-lj1-x243.google.com [IPv6:2a00:1450:4864:20::243]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5by52Vkz3JfT; Thu, 29 Aug 2019 15:09:02 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mail-lj1-x243.google.com with SMTP id l14so3415718lje.2; Thu, 29 Aug 2019 08:09:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:reply-to:from:date:message-id :subject:to:cc; bh=q+v6xLmI+5EfAk0WfFddMXmOZ8cUNN7J5HBg/AFEXdc=; b=ICOsuU4Mgn6hQRvyzUxmu9xnbFJxGaVLjg5kh8Z21ZZo0XB4j84kQ9++iJw0jk96Ai zRri8oVYQIC1/6ZSxuo9kPD4MP2cMiRnzFgUfX69K6DD4Cs7syrO8Wy9MmUfRd+WJS6v GjYqcr18ut+ZuH3/rjzOkbjhko/9ahkCkSNfaGq7WveSYmZtnVL7ZkzpsL+wvVAA5GZq Uu6agEWtSJzHFB74imQVwCiw1VCJ+UoNzJZRmr+1f6YM9VeDSBRYItKI7yeHmZ2H391m +m3Vb2siWiKMX/QvcwXugKFzfLtgg13vJZeBRby/PtF+BnqFZjiuz/sBMyl6tZnb8SCW ByWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to:cc; bh=q+v6xLmI+5EfAk0WfFddMXmOZ8cUNN7J5HBg/AFEXdc=; b=GiNXpbB99NJ1BN2gafP7uTL0TLl8B+4BplopGJ1ioVb7Ga9V7/BT4YzNdGInG4iqv3 AmTbE9QJd2CpJYDvRcpzzhLuW5i1aaZP33uZAZFSKU+47Pqy4fjQ2tuwexQomQg5t9iw tykL6mmX5tJ4gU75jn50O/aY1najx2/kqXBgRbyGMvzOhcOHLvqW5JjW4prxdin6Z/EN 6QxFXs6KDX/kBsJlZYUnFOISPm5+kDNG/ufXQ2kjuFhx+2HRkDlfUn84X/PPfLbAS62X q341RREFG351sr3k3FtwMWtps4B6BDgeTyCeLmQWHrVKL+Hu+9SXcwx1+hEabzRV1Gio dLew== X-Gm-Message-State: APjAAAX+U/RbhZ7hQHNJcnLm8D6z3gmllS0OScwHPDn5rK1A52I1ziZF BjbIagXAo7Xy+Q12Qy9Yxr/HYcdqaZ0ZWh49Q+oub0iBuss= X-Google-Smtp-Source: APXvYqzgOgPvyT8Iruzjm0NRoATQ/vwYzJJVZAvZ+hu6Eb+DeduSjWQka4/IePMwIb72VKAhGaNQ41PiG79erNaR0Vc= X-Received: by 2002:a2e:81ca:: with SMTP id s10mr5721694ljg.181.1567091340970; Thu, 29 Aug 2019 08:09:00 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> <20190829144228.GA71821@kib.kiev.ua> In-Reply-To: Reply-To: araujo@freebsd.org From: Marcelo Araujo Date: Thu, 29 Aug 2019 23:08:49 +0800 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Kristof Provost Cc: Konstantin Belousov , FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org X-Rspamd-Queue-Id: 46K5by52Vkz3JfT X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 15:09:03 -0000 Em qui, 29 de ago de 2019 =C3=A0s 23:03, Kristof Provost escreveu: > On 29 Aug 2019, at 16:42, Konstantin Belousov wrote: > > On Thu, Aug 29, 2019 at 02:03:00PM +0200, Kristof Provost wrote: > >> There are, somewhat regularly, commits which break functionality, or > >> at > >> the very least tests. > >> The main objective of this policy proposal is to try to improve > >> overall > >> code quality by encouraging and empowering all committers to > >> investigate > >> and fix test failures. > > But this policy does not encourage, if anything. > > It gives a free ticket to revert, discouraging committers. > > > To provide a counterpoint here: my personal frustration right now is > that I=E2=80=99ve spent a good bit of time adding tests for pf and fixing= bugs > for it, only to see the tests having to be disabled because of unrelated > (to pf) changes in the network stack. > > Either through lack of visibility, or lack of time, or because people > assume pf tests failures must by definition be the responsibility of the > pf maintainer, these failures have not been investigated by anyone other > than me, and I lack the time and subject matter expertise to fix them. > > I=E2=80=99m desperately afraid that if/when these bugs do get fixed we=E2= =80=99re > going to discover that other things have broken in the mean time, and > the tests are still going to fail, for different reasons. > > These are bugs. They=E2=80=99re the best case scenario for bug reports ev= en, > because they come with a reproduction case built-in, and yet they=E2=80= =99re > still not getting fixed. This too is discouraging. > > I=E2=80=99m open to alternative proposals for how to address that problem= , but > I don=E2=80=99t think that =E2=80=9Ccontinue on as we always have=E2=80= =9D is the correct > OK, because of PF that is sort of deprecated on FreeBSD and it need some new rules to make it workable, everybody else need to abdicate to some new rules. Yes, right you are!!!! > answer. > > Best regards, > Kristof > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " > --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-fcp@freebsd.org Thu Aug 29 15:09:35 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 183C8D8C62 for ; Thu, 29 Aug 2019 15:09:35 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46K5cZ6mphz3Jph for ; Thu, 29 Aug 2019 15:09:34 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id E86C5D8C60; Thu, 29 Aug 2019 15:09:34 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E8166D8C5E; Thu, 29 Aug 2019 15:09:34 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5cZ5Yt8z3Jpg; Thu, 29 Aug 2019 15:09:34 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 76A2D1BFEA; Thu, 29 Aug 2019 15:09:34 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from [10.0.2.193] (ptr-8rh08k12jlyqauo4swr.18120a2.ip6.access.telenet.be [IPv6:2a02:1811:240e:402:c59e:f85c:1ed8:db5b]) (Authenticated sender: kp) by venus.codepro.be (Postfix) with ESMTPSA id EEAC138603; Thu, 29 Aug 2019 17:09:32 +0200 (CEST) From: "Kristof Provost" To: araujo@freebsd.org Cc: "Ian Lepore" , "Konstantin Belousov" , "FreeBSD Hackers" , "Li-Wen Hsu" , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Date: Thu, 29 Aug 2019 17:09:32 +0200 X-Mailer: MailMate (2.0BETAr6137) Message-ID: In-Reply-To: References: <20190829114057.GZ71821@kib.kiev.ua> <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 15:09:35 -0000 On 29 Aug 2019, at 17:01, Marcelo Araujo wrote: > Em qui, 29 de ago de 2019 =C3=A0s 22:54, Kristof Provost > escreveu: > >> On 29 Aug 2019, at 16:42, Ian Lepore wrote: >>> (And I don't think breaking a test counts as >>> breaking the build.) >>> >> I fundamentally disagree on this point. A test failure is, just like = >> a >> compiler warning, a precious gift that should not be ignored. >> The more distance (both in terms of time, and in terms of the people >> involved) there is between a bug being introduced and it being = >> detected >> the harder it is to fix it. Test accelerate detection of bugs. If we = >> do >> not take test failures seriously (i.e. as an indication something is >> wrong and should be fixed) the tests will inevitable become useless = >> in >> one of two ways: we=E2=80=99ll either disable failing tests (which is = what = >> we >> tend to do now) reducing test coverage or we=E2=80=99ll have a test su= ite = >> with >> many failures in it, which makes it useless as well. (As with = >> compiler >> warnings, the best way to keep them under control is to consider them = >> to >> be fatal errors.) >> > > Could you elaborate where is the "fundamentally" you disagree? Where = > is the > fundament? You guys are introducing something new, yes everybody knows > about test, it is year 2019, but nobody can come with new rules tha in > hours we gonna revert if you "dare to don't fix it". Sorry, this is = > not how > people test software and fix it. > I do think that breaking a test breaks the build. Something used to work = and now it doesn=E2=80=99t. That=E2=80=99s breakage, even if it=E2=80=99s= not as total as = it not compiling any more. >> In either scenario we end up reducing test coverage, which means = >> we=E2=80=99re >> going to push more bugs towards users. >> >>> I totally agree. This is an overly-bureaucratic solution in search = >>> of >>> a problem. >>> >>> If this needs to be addressed at all (and I'm not sure it does), = >>> then >>> another sentence or two in bullet item 10 in section 18.1 [*] of the >>> committer's guide should be enough. And even then it needn't be >>> overly-formal and should just mention that if a commit does break = >>> the >>> build the committer is expected to be responsive to that problem and >>> the commit might get reverted if they're unresponsive. I don't = >>> think >>> we need schedules. >>> >> I do feel that=E2=80=99s a better argument. We=E2=80=99ve always had a= policy of >> reverting on request (AIUI), so this is more or less trying to be a >> strong restatement of that, more than a fundamental shift in policy. >> > > We don't have a policy to revert commit, actually revert commit is > something bad, it is kind of punishment, I have been there, nobody = > wants to > be there. Stop to push this non-sense argument. > That=E2=80=99s how I=E2=80=99ve interpreted =E2=80=9911. Developer Relati= ons=E2=80=99 in in = https://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/art= icle.html#conventions = (Specifically, =E2=80=9C If a commit does results in controversy erupti= ng, = it may be advisable to consider backing the change out again until the = matter is settled.=E2=80=9D) I understand that it=E2=80=99s not fun to see changes reverted, and it=E2= =80=99s = certainly not the intention to make that the preferred solution. = That=E2=80=99s why the FCP discusses adds waiting periods and discussion = with = the committer. Best regards, Kristof From owner-freebsd-fcp@freebsd.org Thu Aug 29 15:10:06 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BF861D8DB4 for ; Thu, 29 Aug 2019 15:10:06 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K5dB3zfpz3JyD for ; Thu, 29 Aug 2019 15:10:06 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.nyi.freebsd.org (Postfix) id 88A4BD8DB3; Thu, 29 Aug 2019 15:10:06 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 886BFD8DB2 for ; Thu, 29 Aug 2019 15:10:06 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x735.google.com (mail-qk1-x735.google.com [IPv6:2607:f8b0:4864:20::735]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5d95Vq3z3Jy8 for ; Thu, 29 Aug 2019 15:10:05 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x735.google.com with SMTP id d23so3254273qko.3 for ; Thu, 29 Aug 2019 08:10:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=hPOjkvVLjUDz8PTEAgwVogrcGUwECCcytgUPPKENR04=; b=z7elCk/3resJpmTxkLZZf/zVFFXNsIcyCk03541nM4tV7nETNaysrc+u6ppfX2M/3B j7GR5LIaGBjunYT9o6PmjkZNQAhrNVus59xeU3J1WVWP9FetlRosPVCQdRIBCq545V2c LqhjIjTjF96xj8p1x5JLXW8RkJpsWOVZRS+Uqmnw21e23JbEOTrLtoM1tQgs42iz5mnO I/Hc6/UvePsJZE7fjyI4C65xkI0az6Ud7eFhgsisqW0dVxBgN17en537uBY6+VlqostM 9GhTMCq8rqZR6mUvTVk/tZjkNymuq7U8AeEI7EtS5cnv/El9Tnn+jxYhuV2TFfRTqiZg 05TQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=hPOjkvVLjUDz8PTEAgwVogrcGUwECCcytgUPPKENR04=; b=Uyftcp507ZUC+PRd6eETuySjwTuLn/Kdb3oUBQwghM3bZZcuLwdiHxQ5NQXtLobmA+ fAJ0igwo7iUwhrGQR09ogocqQ6JPM0CXHZP9ap1EzKHdERu6raO4CVbpAdOJn/ljyaAz oKqhE36qFi0CgaWvDPLz598LyzMuMXVQSYP/snD9xVexibhQ6hthtjbC6xjgv+N7iesx NgVFAHxP2ubIs9c0xzzzsqE8ikq6ocQzc2rRlMTwF0I4mmk0Xcj7cZDceYreVTQbSQSX TxAMcU9oTuoj4ZZiOZTsYKnHTo6qOR+9dqvvbBFo37D/Gm99513kmXCWqwohFjfOLG36 ZcSg== X-Gm-Message-State: APjAAAV0Q4nfdQD01Q+Te5yV1+tsjgLuml3JaiKCgY7nIlifqaiU+lcA QCWkJAv/LNK2H2Sigxo/uUywYtUINJJDsS+YL1gEhw== X-Google-Smtp-Source: APXvYqwFgJdDNLwBgKx77ozvLQ5s+GCLcQKQljZwWMUdZB+rpDKRK614tXrK+5zJwsjFkYCvzB0rXWTlo9yArBPkZC4= X-Received: by 2002:a37:2c07:: with SMTP id s7mr9818010qkh.495.1567091404529; Thu, 29 Aug 2019 08:10:04 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> <20190829144228.GA71821@kib.kiev.ua> In-Reply-To: From: Warner Losh Date: Thu, 29 Aug 2019 09:09:53 -0600 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Marcelo Araujo Cc: Kristof Provost , Konstantin Belousov , FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org X-Rspamd-Queue-Id: 46K5d95Vq3z3Jy8 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=z7elCk/3; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::735) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-5.93 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[fcp@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; RCPT_COUNT_FIVE(0.00)[6]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; RCVD_IN_DNSWL_NONE(0.00)[5.3.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.93)[ip: (-9.43), ipnet: 2607:f8b0::/32(-2.85), asn: 15169(-2.32), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 15:10:06 -0000 On Thu, Aug 29, 2019 at 9:09 AM Marcelo Araujo wrote: > Em qui, 29 de ago de 2019 =C3=A0s 23:03, Kristof Provost > escreveu: > > > On 29 Aug 2019, at 16:42, Konstantin Belousov wrote: > > > On Thu, Aug 29, 2019 at 02:03:00PM +0200, Kristof Provost wrote: > > >> There are, somewhat regularly, commits which break functionality, or > > >> at > > >> the very least tests. > > >> The main objective of this policy proposal is to try to improve > > >> overall > > >> code quality by encouraging and empowering all committers to > > >> investigate > > >> and fix test failures. > > > But this policy does not encourage, if anything. > > > It gives a free ticket to revert, discouraging committers. > > > > > To provide a counterpoint here: my personal frustration right now is > > that I=E2=80=99ve spent a good bit of time adding tests for pf and fixi= ng bugs > > for it, only to see the tests having to be disabled because of unrelate= d > > (to pf) changes in the network stack. > > > > Either through lack of visibility, or lack of time, or because people > > assume pf tests failures must by definition be the responsibility of th= e > > pf maintainer, these failures have not been investigated by anyone othe= r > > than me, and I lack the time and subject matter expertise to fix them. > > > > I=E2=80=99m desperately afraid that if/when these bugs do get fixed we= =E2=80=99re > > going to discover that other things have broken in the mean time, and > > the tests are still going to fail, for different reasons. > > > > These are bugs. They=E2=80=99re the best case scenario for bug reports = even, > > because they come with a reproduction case built-in, and yet they=E2=80= =99re > > still not getting fixed. This too is discouraging. > > > > I=E2=80=99m open to alternative proposals for how to address that probl= em, but > > I don=E2=80=99t think that =E2=80=9Ccontinue on as we always have=E2=80= =9D is the correct > > > > OK, because of PF that is sort of deprecated on FreeBSD and it need some > new rules to make it workable, everybody else need to abdicate to some ne= w > rules. Yes, right you are!!!! > Let's take every opportunity to clarify community norms and turn it into a federal case. That's productive. Warner From owner-freebsd-fcp@freebsd.org Thu Aug 29 15:13:07 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9434BD9124 for ; Thu, 29 Aug 2019 15:13:07 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46K5hg2M10z3KRl for ; Thu, 29 Aug 2019 15:13:07 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 4ECDAD9121; Thu, 29 Aug 2019 15:13:07 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4E772D9120; Thu, 29 Aug 2019 15:13:07 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K5hf10pQz3KRk; Thu, 29 Aug 2019 15:13:05 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mail-lf1-x132.google.com with SMTP id r134so1925338lff.12; Thu, 29 Aug 2019 08:13:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:reply-to:from:date:message-id :subject:to:cc; bh=h0QMz+Lm8FEUOwvzCgeLbqnlJx+2a0mAxRB2YpPElmc=; b=oeml8Be48kyt6U9nTOFc6KyjbIdOcnb3Q+4/ZzNCUn3tbB5g7n7qrKhGrPD21RqJEv Bo+EdopkKyhIR6Gt0u1wiJF+AkzHVZ56/7rY8z9svqj6YXyA4DpdB+YNFusLD8cWf8sJ Wf0YArIZ1tG0mNOKHH53j05ICTlqQkNc/3IKZ/5fHoqFcJrnHJte6Wf+bNi7K2gMSjFt cugSXQX0/4RWEK6Y/AApgynYB2SQG5D9HDOCgfLO2QX0jQTsCxZ7+paZDd2WU5Jb6zre nUv5sQ/mo5czl8hMwl0cNtxUieAaFBq9Jey4QhxQr1p0UWq+7k8HIUPHWTLtHyYfWfwc o6VA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:reply-to :from:date:message-id:subject:to:cc; bh=h0QMz+Lm8FEUOwvzCgeLbqnlJx+2a0mAxRB2YpPElmc=; b=n+zW64dil5giiguJ/4THzlLPAB2YRq5xl7UIAcKLXNgr11d42Fuv4HriBbBXXjfSwM LQ78n6vDprJ84iUe2b7gyKrArIouqY/9tU0nnc/3m20vH3VKuL0sHs66rofaQydcud0z GxDhKk+XogMffN2QPvEUh6CXDvTw0ZB/0GnFGFsYgsuPFF3vmlIYSGthD3TMPtisXXnI S3yS6pO1MGmyJjBIERnXWvHUzuw166KdzfaLZ27qx2/vB25X8fa3uTlVVpAYopHH/tE+ UjT6aCIN6RdKbRgdQI+9Y6qUOLEwjio0XbNUMXTIdlG+pDOtIHMXVll0stXB/9XfkLLj OFxQ== X-Gm-Message-State: APjAAAVMaNd3mr5bgPZZDW2TvUb8n3GTEqAaLtphlNFZhGBv2BSyOffe pT3qZCINYBsa9MOhmweAdA1Qa1RRVRnGHYM0JJc= X-Google-Smtp-Source: APXvYqyBoaxALyIyxqXLD1UbhXziGmNMbQfbPUoEla3gz2Djnd5WeIp9kPMzsG7nr1DoZrK+pWwGtz+vZZ9x0L2n784= X-Received: by 2002:ac2:5485:: with SMTP id t5mr4501980lfk.27.1567091582825; Thu, 29 Aug 2019 08:13:02 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> <20190829144228.GA71821@kib.kiev.ua> In-Reply-To: Reply-To: araujo@freebsd.org From: Marcelo Araujo Date: Thu, 29 Aug 2019 23:12:51 +0800 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Warner Losh Cc: Marcelo Araujo , Kristof Provost , Konstantin Belousov , FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org X-Rspamd-Queue-Id: 46K5hf10pQz3KRk X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=oeml8Be4; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of araujobsdport@gmail.com designates 2a00:1450:4864:20::132 as permitted sender) smtp.mailfrom=araujobsdport@gmail.com X-Spamd-Result: default: False [-3.99 / 15.00]; HAS_REPLYTO(0.00)[araujo@freebsd.org]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCPT_COUNT_SEVEN(0.00)[7]; NEURAL_HAM_SHORT(-0.99)[-0.993,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-9.54), ipnet: 2a00:1450::/32(-2.99), asn: 15169(-2.32), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2.3.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 15:13:07 -0000 Em qui, 29 de ago de 2019 =C3=A0s 23:10, Warner Losh escre= veu: > > > On Thu, Aug 29, 2019 at 9:09 AM Marcelo Araujo > wrote: > >> Em qui, 29 de ago de 2019 =C3=A0s 23:03, Kristof Provost >> escreveu: >> >> > On 29 Aug 2019, at 16:42, Konstantin Belousov wrote: >> > > On Thu, Aug 29, 2019 at 02:03:00PM +0200, Kristof Provost wrote: >> > >> There are, somewhat regularly, commits which break functionality, o= r >> > >> at >> > >> the very least tests. >> > >> The main objective of this policy proposal is to try to improve >> > >> overall >> > >> code quality by encouraging and empowering all committers to >> > >> investigate >> > >> and fix test failures. >> > > But this policy does not encourage, if anything. >> > > It gives a free ticket to revert, discouraging committers. >> > > >> > To provide a counterpoint here: my personal frustration right now is >> > that I=E2=80=99ve spent a good bit of time adding tests for pf and fix= ing bugs >> > for it, only to see the tests having to be disabled because of unrelat= ed >> > (to pf) changes in the network stack. >> > >> > Either through lack of visibility, or lack of time, or because people >> > assume pf tests failures must by definition be the responsibility of t= he >> > pf maintainer, these failures have not been investigated by anyone oth= er >> > than me, and I lack the time and subject matter expertise to fix them. >> > >> > I=E2=80=99m desperately afraid that if/when these bugs do get fixed we= =E2=80=99re >> > going to discover that other things have broken in the mean time, and >> > the tests are still going to fail, for different reasons. >> > >> > These are bugs. They=E2=80=99re the best case scenario for bug reports= even, >> > because they come with a reproduction case built-in, and yet they=E2= =80=99re >> > still not getting fixed. This too is discouraging. >> > >> > I=E2=80=99m open to alternative proposals for how to address that prob= lem, but >> > I don=E2=80=99t think that =E2=80=9Ccontinue on as we always have=E2= =80=9D is the correct >> > >> >> OK, because of PF that is sort of deprecated on FreeBSD and it need some >> new rules to make it workable, everybody else need to abdicate to some n= ew >> rules. Yes, right you are!!!! >> > > Let's take every opportunity to clarify community norms and turn it into = a > federal case. That's productive. > Yeah, that was my bad!!! Apologies for that if we still have time. Sorry for that. > > Warner > --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-fcp@freebsd.org Thu Aug 29 15:37:21 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 14ED8D9C9F for ; Thu, 29 Aug 2019 15:37:21 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46K6Dc5qrFz3Lwh for ; Thu, 29 Aug 2019 15:37:20 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id C65A1D9C9B; Thu, 29 Aug 2019 15:37:20 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C6052D9C9A; Thu, 29 Aug 2019 15:37:20 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46K6Dc3xYNz3Lwf; Thu, 29 Aug 2019 15:37:20 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x7TFbChh018671 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 29 Aug 2019 18:37:15 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x7TFbChh018671 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x7TFbCjH018670; Thu, 29 Aug 2019 18:37:12 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 29 Aug 2019 18:37:12 +0300 From: Konstantin Belousov To: Kristof Provost Cc: Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190829153712.GB71821@kib.kiev.ua> References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> <20190829144228.GA71821@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46K6Dc3xYNz3Lwf X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.90 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.91)[-0.906,0] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 15:37:21 -0000 On Thu, Aug 29, 2019 at 05:02:47PM +0200, Kristof Provost wrote: > On 29 Aug 2019, at 16:42, Konstantin Belousov wrote: > > On Thu, Aug 29, 2019 at 02:03:00PM +0200, Kristof Provost wrote: > >> There are, somewhat regularly, commits which break functionality, or > >> at > >> the very least tests. > >> The main objective of this policy proposal is to try to improve > >> overall > >> code quality by encouraging and empowering all committers to > >> investigate > >> and fix test failures. > > But this policy does not encourage, if anything. > > It gives a free ticket to revert, discouraging committers. > > > To provide a counterpoint here: my personal frustration right now is > that I’ve spent a good bit of time adding tests for pf and fixing bugs > for it, only to see the tests having to be disabled because of unrelated > (to pf) changes in the network stack. > > Either through lack of visibility, or lack of time, or because people > assume pf tests failures must by definition be the responsibility of the > pf maintainer, these failures have not been investigated by anyone other > than me, and I lack the time and subject matter expertise to fix them. > > I’m desperately afraid that if/when these bugs do get fixed we’re > going to discover that other things have broken in the mean time, and > the tests are still going to fail, for different reasons. > > These are bugs. They’re the best case scenario for bug reports even, > because they come with a reproduction case built-in, and yet they’re > still not getting fixed. This too is discouraging. I fully agree with your attitude there, and understand your frustration. IMO the right action would be to contact the committers who did the relevant changes, first. Was it done ? What was their response ? If they are silent, next action would be some public mail. Do you know where the bug is ? If yes, how hard is to fix it ? > > I’m open to alternative proposals for how to address that problem, but > I don’t think that “continue on as we always have” is the correct > answer. > > Best regards, > Kristof From owner-freebsd-fcp@freebsd.org Thu Aug 29 15:52:46 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D660DDA484 for ; Thu, 29 Aug 2019 15:52:46 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46K6ZQ4hQyz3N0M for ; Thu, 29 Aug 2019 15:52:46 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id A0E42DA482; Thu, 29 Aug 2019 15:52:46 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A090BDA481; Thu, 29 Aug 2019 15:52:46 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-yw1-f48.google.com (mail-yw1-f48.google.com [209.85.161.48]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46K6ZP5rFhz3N0J; Thu, 29 Aug 2019 15:52:45 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-yw1-f48.google.com with SMTP id i207so1296530ywc.9; Thu, 29 Aug 2019 08:52:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=wY3QqHOPLSUlCOb4QmA9woMOMytMceJZ9WNhi9JsvlM=; b=CYWnJ8vNpdiBVrHFHRT05p1bEF2BUGCKrxwembIDbWQtYG4SjLr/cYSHStplFMSU9c ySjsDcJpvm1Ev9lK8ROgBOwvoD8opCnLq+y3IfYIy0mqhdcWDmm0cY2zM+1TVFBE3Arl 4IgfBWCIv13a/NMLFK8TNwF3L4OJGhzLEYF60Qwk4sdT7oI3GT9SAmf1MCpiJO8w+vih a3E18wLVVx9ux2XAz0RMWIlApTLgLBvZlXmsFo+R9NIpsgsLJPXPU8mcuvcTWtr4GVhm lvDLFRSd/nt7D+7YKl8maTtsQbPMMPUU7kat2UDuwRjvaVEfg1r5jA4NnWmQCzCEpS+c 9c2A== X-Gm-Message-State: APjAAAUE9JMFS0i/ncc/76YqR4tgkmV3LDibGm51zzVDvPphKlMqU+j+ 4mFdzKJ7V4TUxHDuQV5fMlTbY3b9sz918vh75UN1FeV9 X-Google-Smtp-Source: APXvYqwSf91pWiPN9dYEGQSlr8I/x8+1FRo41YZhZ6mp+Ir88nYMezpj5tvO48mwXFlTygEOH5/EFE6wnhk3eQUzpmk= X-Received: by 2002:a0d:ca02:: with SMTP id m2mr7784840ywd.400.1567093964422; Thu, 29 Aug 2019 08:52:44 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> <20190829144228.GA71821@kib.kiev.ua> <20190829153712.GB71821@kib.kiev.ua> In-Reply-To: <20190829153712.GB71821@kib.kiev.ua> From: Li-Wen Hsu Date: Thu, 29 Aug 2019 23:52:29 +0800 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: FreeBSD Hackers , fcp@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46K6ZP5rFhz3N0J X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of lwhsufreebsd@gmail.com designates 209.85.161.48 as permitted sender) smtp.mailfrom=lwhsufreebsd@gmail.com X-Spamd-Result: default: False [-5.93 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; TAGGED_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; IP_SCORE(-2.93)[ip: (-8.95), ipnet: 209.85.128.0/17(-3.34), asn: 15169(-2.32), country: US(-0.05)]; NEURAL_HAM_SHORT(-0.99)[-0.994,0]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[48.161.85.209.list.dnswl.org : 127.0.5.0]; FORGED_SENDER(0.30)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 15:52:46 -0000 I was really hesitated to send out the previous mail because I understand this is a sensitive topic and some words might look scary. My apologies if the proposal makes people uncomfortable. Please calm down, this is just an idea, not call for vote or say it will proceed as-is. We want to point out there is a thing may worth to be discussed, and hope to collect the comments and suggestions, we know the proposed way is not perfect, and that's why we need better idea from more people. I am really bad at writing, but some items I hope people can check: - Yes we're doing good now, and we should keep it and let's try to do better. - Please check the weekly CI report to see the build (and test) statistics, we still have some rooms to be improved. Let's try to get bugs fixed before users asking on -current or -stable. Of course we cannot cover all parts of the system, but that means we can always do more contributions. - We have been running CI for few years, perhaps it's still too early, but let's try to pay more attention to it. If there are things imperfect, let's fix it. I cannot watch the results, do preliminary analysis and call people to check, just by myself along, at least not forever. I hope we can have a more automatic and scalable way. - There is no one want to "control" the contribution, instead, we hope this could make collaboration more smoothly. As there are more and more contributors, we should have a way to keep head and stable branches buildable and have less regressions as we can as possible, then everyone can work together with more confidence. - The "revert" part and the timeline looks scary, please think this is the last and unwanted solution. The description is try to limit its scope and encourage people do analysis, communication and fix first. The words here absolutely should be improved. Updating the committer guide is really a good idea, and we probably should define what means "unresponsive", like in ports we have maintainer timeout and even maintainer reset. Thanks the feedback from all of you. I think encouraging people to discuss and keep the discussion record is one of the purposes to have FCP process. Best, Li-Wen From owner-freebsd-fcp@freebsd.org Thu Aug 29 19:05:14 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8E4B6DDABF for ; Thu, 29 Aug 2019 19:05:14 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KBrV1stHz44l3 for ; Thu, 29 Aug 2019 19:05:14 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: by mailman.nyi.freebsd.org (Postfix) id 3E68DDDABD; Thu, 29 Aug 2019 19:05:14 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3E1C4DDABC; Thu, 29 Aug 2019 19:05:14 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46KBrT2SJ5z44l2; Thu, 29 Aug 2019 19:05:12 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id x7TJ5BRs091372; Thu, 29 Aug 2019 12:05:11 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id x7TJ5Bw8091371; Thu, 29 Aug 2019 12:05:11 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> Subject: Re: FCP 20190401-ci_policy: CI policy In-Reply-To: To: araujo@freebsd.org Date: Thu, 29 Aug 2019 12:05:11 -0700 (PDT) CC: Kristof Provost , Konstantin Belousov , FreeBSD Hackers , Li-Wen Hsu , Ian Lepore , fcp@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 46KBrT2SJ5z44l2 X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd-rwg@gndrsh.dnsmgr.net has no SPF policy when checking 69.59.192.140) smtp.mailfrom=freebsd-rwg@gndrsh.dnsmgr.net X-Spamd-Result: default: False [2.65 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.49)[0.485,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.95)[0.946,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.28)[0.278,0]; RCPT_COUNT_SEVEN(0.00)[7]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.04)[ip: (0.15), ipnet: 69.59.192.0/19(0.07), asn: 13868(0.05), country: US(-0.05)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 19:05:14 -0000 (unneeded context removed) > > In either scenario we end up reducing test coverage, which means we?re > > going to push more bugs towards users. > > > > > I totally agree. This is an overly-bureaucratic solution in search of > > > a problem. > > > > > > If this needs to be addressed at all (and I'm not sure it does), then > > > another sentence or two in bullet item 10 in section 18.1 [*] of the > > > committer's guide should be enough. And even then it needn't be > > > overly-formal and should just mention that if a commit does break the > > > build the committer is expected to be responsive to that problem and > > > the commit might get reverted if they're unresponsive. I don't think > > > we need schedules. > > > > > I do feel that?s a better argument. We?ve always had a policy of > > reverting on request (AIUI), so this is more or less trying to be a > > strong restatement of that, more than a fundamental shift in policy. > > > > We don't have a policy to revert commit, actually revert commit is > something bad, it is kind of punishment, I have been there, nobody wants to > be there. Stop to push this non-sense argument. Here in lies one of the fundemental problems, this view by some that a "revert commit is something bad, it is kind of punishment". That is not true. Reverts are GREAT things, they allow the tree to be returned to a known state, usually quicly. The original commit is STILL IN SVN, and a bad revert can guess what.. be reverted!. IMHO the project as a whole needs to overcome its fear of reverts and start to use them for the great and powerful things that they are. This connection of bad and punishment needs to stop, and the sooner the better. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-fcp@freebsd.org Thu Aug 29 21:26:31 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6B87AE04E9 for ; Thu, 29 Aug 2019 21:26:31 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KFzW0c1mz4Ctf for ; Thu, 29 Aug 2019 21:26:31 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.nyi.freebsd.org (Postfix) id 12E45E04E8; Thu, 29 Aug 2019 21:26:31 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 12AB9E04E7 for ; Thu, 29 Aug 2019 21:26:31 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x72b.google.com (mail-qk1-x72b.google.com [IPv6:2607:f8b0:4864:20::72b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KFzV0hpKz4Ctc for ; Thu, 29 Aug 2019 21:26:29 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x72b.google.com with SMTP id w18so4396873qki.0 for ; Thu, 29 Aug 2019 14:26:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=psQkg5NFhz4GN6E0Jq/7T4mBDwPB1ph0P/vT6e2lXuQ=; b=JZd6loCg+YQdQql8wgOiMWtOt5vOp+II5onQh8aReWkG7MvshujPEnEh9AJ54oKpFk ByuiRvoyLjTSkE4FiGYIUIq4P34CkF/y9IV1qDX7DC/DnnSvI91/X+IcW6j55wR8iyRx rJXROucxRC0RKNODzMr1G73wp0F/xgHEtM7GS7XmwS/kvLvzcfxeqL5mEGBQDPqik+nt mlhNyUmBl1CtiA5e3Y6kHABkKBiCt3lGjenwHDMutQRi+nI3Cul9buFSDGEaOzckEQXN xCRXbc/AOd+OEDe8ZtaVUBd0FwkwoWgCsn7/aocVL6Ri8WwOWI9MTEemQfAzSdq5fue4 tleg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=psQkg5NFhz4GN6E0Jq/7T4mBDwPB1ph0P/vT6e2lXuQ=; b=bk2wxB1zNJ1+10LRvJNU5jP8eFNduZRZ8lsqoArN+d1+vRCYHKDwomRZ3Ya8AkCUdt nkGJAfhASWkvpx94mQN8TFRWvIvdKb2QEuTZNoake/j8tj5dXOUTzYqk+TWctYimn+N/ bobdnoucG0OmmuRFVtnpLPkh21NsgIFj6I3UInysZlqgLIDTHwG6mMwJkxdPg51Df8ac AWNfE9QNp44g7Dqd35phHR3KtTnmdDkSw/70uC4s5uJJA7AMpa2P46iit/tfqcSKBzxD +bDhQwqHpi59FgEBLemY8ZJjF3fpFgDbn6XDw2xpT6qp789n2qaGRjd8ZI7Df57mGQdR 36Zg== X-Gm-Message-State: APjAAAVveGxfrEE+d06ofIHff47s3KMv86sdFLjHC/mh8+Wpphie1wZM EUqb+LjA6CV4XZPEfmkbSI9LFg+4+1zruztc02Hqbw== X-Google-Smtp-Source: APXvYqxPX6vTEYq6x5eAL4kdMewdUKcE6kOLkS1/XnuFVPYFqgjiLIDWu/r+la/Te+iNG3KoxtoAmIP0CU7xocCGeSQ= X-Received: by 2002:a37:4b03:: with SMTP id y3mr12266242qka.215.1567113988934; Thu, 29 Aug 2019 14:26:28 -0700 (PDT) MIME-Version: 1.0 References: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> In-Reply-To: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> From: Warner Losh Date: Thu, 29 Aug 2019 15:26:17 -0600 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: "Rodney W. Grimes" Cc: Marcelo Araujo , Ian Lepore , FreeBSD Hackers , fcp@freebsd.org, Konstantin Belousov , Li-Wen Hsu , Kristof Provost X-Rspamd-Queue-Id: 46KFzV0hpKz4Ctc X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=JZd6loCg; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::72b) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-4.93 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[fcp@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; URI_COUNT_ODD(1.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; RCVD_IN_DNSWL_NONE(0.00)[b.2.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCPT_COUNT_SEVEN(0.00)[8]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.94)[ip: (-9.47), ipnet: 2607:f8b0::/32(-2.85), asn: 15169(-2.32), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 21:26:31 -0000 On Thu, Aug 29, 2019 at 1:05 PM Rodney W. Grimes < freebsd-rwg@gndrsh.dnsmgr.net> wrote: > (unneeded context removed) > > > > In either scenario we end up reducing test coverage, which means we?re > > > going to push more bugs towards users. > > > > > > > I totally agree. This is an overly-bureaucratic solution in search > of > > > > a problem. > > > > > > > > If this needs to be addressed at all (and I'm not sure it does), then > > > > another sentence or two in bullet item 10 in section 18.1 [*] of the > > > > committer's guide should be enough. And even then it needn't be > > > > overly-formal and should just mention that if a commit does break the > > > > build the committer is expected to be responsive to that problem and > > > > the commit might get reverted if they're unresponsive. I don't think > > > > we need schedules. > > > > > > > I do feel that?s a better argument. We?ve always had a policy of > > > reverting on request (AIUI), so this is more or less trying to be a > > > strong restatement of that, more than a fundamental shift in policy. > > > > > > > We don't have a policy to revert commit, actually revert commit is > > something bad, it is kind of punishment, I have been there, nobody wants > to > > be there. Stop to push this non-sense argument. > > Here in lies one of the fundemental problems, this view by some that > a "revert commit is something bad, it is kind of punishment". That is > not true. Reverts are GREAT things, they allow the tree to be returned > to a known state, usually quicly. The original commit is STILL IN SVN, > and a bad revert can guess what.. be reverted!. > > IMHO the project as a whole needs to overcome its fear of reverts and > start to use them for the great and powerful things that they are. > > This connection of bad and punishment needs to stop, and the sooner > the better. > > -- > Rod Grimes > rgrimes@freebsd.org > _______________________________________________ > freebsd-fcp@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fcp > To unsubscribe, send any mail to "freebsd-fcp-unsubscribe@freebsd.org" > From owner-freebsd-fcp@freebsd.org Thu Aug 29 21:32:30 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 93ED3E07EA for ; Thu, 29 Aug 2019 21:32:30 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46KG6Q2l2Jz4DNP for ; Thu, 29 Aug 2019 21:32:30 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.nyi.freebsd.org (Postfix) id 5C1FDE07E9; Thu, 29 Aug 2019 21:32:30 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5BDFFE07E8 for ; Thu, 29 Aug 2019 21:32:30 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x82b.google.com (mail-qt1-x82b.google.com [IPv6:2607:f8b0:4864:20::82b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KG6P31M1z4DNC for ; Thu, 29 Aug 2019 21:32:29 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x82b.google.com with SMTP id g4so5409500qtq.7 for ; Thu, 29 Aug 2019 14:32:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:from:date:message-id:subject:to:cc; bh=t370hma+L6lmoL29ph5rMXaIN65CYesVtRZQtj6/L3Y=; b=2OeaZ8GYhg7L52BtAAYIKbGd4620qyJXxpLw8+CfVkTuLeVACLc8skdM4MF0SqOvb3 o2L6vDWdvwaVymd5TkKDjGARfZTGWQtThdChCDfzqqDyC663SS69EwIIHB2ht/ia9YHO 7mE1K2dgAspXb4pZKx+FJm7I41fTH4nzRREOuGGlY7nVe3x+tJBWHDYo4pqhNwJNXqej tHJhTEhvDiWejEYynxzHEYxcuVpfKtD8gR5OGz0KeQYf0D8iG/OxeqLGp87zn3pIIzgX nO65JULsffWqCZ5QYtLeCM2Mj7UYqhmhK8o/RAarqJfFw6JKO+ypWqZFtuxlaZrS1Ban oErw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:from:date:message-id :subject:to:cc; bh=t370hma+L6lmoL29ph5rMXaIN65CYesVtRZQtj6/L3Y=; b=C6fpahMVk5AwRfNxzF9RK/WrkN6dQ/x+yJYf1uHCiwwdeoB71ccaAOvqEaFaD9oCcI gDQqOCo9MmBsf8BXj46zTCNsUYJvjWCgOguL3WM089ttkGuy6eioXAcfExS/7jrt7LQG eTWfJ5hDUsA0dM5pCQTE98LSUT8cUjpTVS8VqiEVmqeTruX+XYYrH7YlrzthUpdJl9EE FSjaKXkjc+oU3XjQ5mypQOI3Rd95nu0WDBKdfRa6tWmKRI0+DZoPqgUXlckUxrsfg9LX 2beJwnMYS/KyTdGZCfEFI6eWCu7+veFLMjjnIQyJC20OfGvQ5nqgzZtXUTwlmbu7AWM7 jlRg== X-Gm-Message-State: APjAAAU+qUGCULCbNeaxeOrvF3KQEBW4OnXb+mChwpbLNKWhi0PAPAOX SLqbmlmBuyjJIHmudFLdopfftYFLoaO+SBqdqSvw8g== X-Google-Smtp-Source: APXvYqzu/TEaMKEGJaicoo/kG89YsEyHSSBwdn/EWwXCDFL8TNWn9IHFu4aXk465mkEFyOIo19ZHinj31xCclBp5N7g= X-Received: by 2002:a0c:f6c6:: with SMTP id d6mr8235136qvo.102.1567114348210; Thu, 29 Aug 2019 14:32:28 -0700 (PDT) MIME-Version: 1.0 References: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> From: Warner Losh Date: Thu, 29 Aug 2019 15:32:17 -0600 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: "Rodney W. Grimes" Cc: Marcelo Araujo , Ian Lepore , FreeBSD Hackers , fcp@freebsd.org, Konstantin Belousov , Li-Wen Hsu , Kristof Provost X-Rspamd-Queue-Id: 46KG6P31M1z4DNC X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=2OeaZ8GY; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::82b) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-5.92 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[fcp@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; RCVD_IN_DNSWL_NONE(0.00)[b.2.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCPT_COUNT_SEVEN(0.00)[8]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.92)[ip: (-9.41), ipnet: 2607:f8b0::/32(-2.84), asn: 15169(-2.32), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 21:32:30 -0000 On Thu, Aug 29, 2019 at 3:26 PM Warner Losh wrote: > > > On Thu, Aug 29, 2019 at 1:05 PM Rodney W. Grimes < > freebsd-rwg@gndrsh.dnsmgr.net> wrote: > >> (unneeded context removed) >> >> > > In either scenario we end up reducing test coverage, which means we?re >> > > going to push more bugs towards users. >> > > >> > > > I totally agree. This is an overly-bureaucratic solution in search >> of >> > > > a problem. >> > > > >> > > > If this needs to be addressed at all (and I'm not sure it does), >> then >> > > > another sentence or two in bullet item 10 in section 18.1 [*] of the >> > > > committer's guide should be enough. And even then it needn't be >> > > > overly-formal and should just mention that if a commit does break >> the >> > > > build the committer is expected to be responsive to that problem and >> > > > the commit might get reverted if they're unresponsive. I don't >> think >> > > > we need schedules. >> > > > >> > > I do feel that?s a better argument. We?ve always had a policy of >> > > reverting on request (AIUI), so this is more or less trying to be a >> > > strong restatement of that, more than a fundamental shift in policy. >> > > >> > >> > We don't have a policy to revert commit, actually revert commit is >> > something bad, it is kind of punishment, I have been there, nobody >> wants to >> > be there. Stop to push this non-sense argument. >> >> Here in lies one of the fundemental problems, this view by some that >> a "revert commit is something bad, it is kind of punishment". That is >> not true. Reverts are GREAT things, they allow the tree to be returned >> to a known state, usually quicly. The original commit is STILL IN SVN, >> and a bad revert can guess what.. be reverted!. >> >> IMHO the project as a whole needs to overcome its fear of reverts and >> start to use them for the great and powerful things that they are. >> >> This connection of bad and punishment needs to stop, and the sooner >> the better. >> > In the past, if someone had any follow on work at all in their tree, the reversion would be quite disruptive to that work. Most of the time it's a lot easier for me, with a lot less friction, to just fix issues that come up after the commit than to revert and prepare a new commit. Sure, it's possible, but it can destroy work in extreme cases. *THAT* is why I'm firmly in the camp of giving the original committer a shot at fixing things because it's much less disruptive to them, and generally we can get a fix into the tree faster. It reduces friction and encourages people to fix things quickly, imho, to hesitate a little on the revert. Especailly when the broken thing is the playstation loader on powerpc that can stay broken for the hour or six (or even days) it takes me to figure out why it broke... Often things away from the beaten path don't get discovered for days or weeks or months, and a reversion then can be extremely disruptive if there's other changes layered on top of the offending commit.... So the whole reversion issue is a lot more complicated than 'oh, it's still in svn'. There are real high costs associated with being too quick or liberal on the revert and those must be weighed against the damage the bad commit is doing.. Warner From owner-freebsd-fcp@freebsd.org Thu Aug 29 23:27:21 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 41E1FE260D for ; Thu, 29 Aug 2019 23:27:21 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KJfx04Dtz4JdN for ; Thu, 29 Aug 2019 23:27:21 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 00849E260B; Thu, 29 Aug 2019 23:27:21 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 002E5E260A; Thu, 29 Aug 2019 23:27:21 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f50.google.com (mail-io1-f50.google.com [209.85.166.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KJfw1NcJz4JdM; Thu, 29 Aug 2019 23:27:19 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f50.google.com with SMTP id d25so7881750iob.6; Thu, 29 Aug 2019 16:27:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=dEoLdEVaF4XHeFHQy6IytzlYnAzvIf+XzwOID05rIfA=; b=bjwIB44oeNBMH3yjASjJOu8URI39+yfVaWR1SsIwmWEn1nUw182IkP4ewOzu2HRVB9 BIsq4jK5w1Xw4Vc7bPQacrQMBay6ZrhMmbxK+2ij7roeDEm7laC9pT18eBZRW5fVDvz6 VsGM3LOorT8Qd8jzbVdkJluojUrRpnpseal/Jjk0cNWXWe+1VgFYkUR6exfv6e9HN5By MYW9OMsmYXrC2qJbjHMdMScXhQWtMoKnjuAY4xvOsZgwbfYE+RdN5HLogvo2o0gOWY83 fxgY6UylkJNAeHde/7gOQ5mRcM3QElv/hPVYqvjeoXWa7DaTXGfWFKkO1eQ0iNpVbMHp QJgA== X-Gm-Message-State: APjAAAXsk4190DjJ2e8DebDQqFa61WkqcxT/J3hhEPqLjhrrv4SuSn3p oI4K1CKgr3/EAPv1kImQCy0Tyj4nf3rt8jrPkcIulw== X-Google-Smtp-Source: APXvYqxvZ6DCuJBCeNxR3auFNov3NTAXDIXQSfdLVHpJMJbfWMIlCwjamuyGu/b6KnJ/AMQIG/TxBaSxbf/Noi5dYi4= X-Received: by 2002:a02:37c6:: with SMTP id r189mr13535732jar.118.1567121238698; Thu, 29 Aug 2019 16:27:18 -0700 (PDT) MIME-Version: 1.0 References: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> In-Reply-To: From: Ed Maste Date: Thu, 29 Aug 2019 19:27:01 -0400 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Warner Losh Cc: FreeBSD Hackers , fcp@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46KJfw1NcJz4JdM X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.50 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-5.54 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.993,0]; RCVD_IN_DNSWL_NONE(0.00)[50.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.55)[ip: (-7.04), ipnet: 209.85.128.0/17(-3.34), asn: 15169(-2.32), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 23:27:21 -0000 On Thu, 29 Aug 2019 at 17:32, Warner Losh wrote: > > In the past, if someone had any follow on work at all in their tree, the > reversion would be quite disruptive to that work. This sounds more like a problem with the tooling than an argument against reverting though. I agree that in the case where the fix is straightforward it's sensible, and in line with community norms, to just commit it. But in the case that a regression was introduced by a committed change, modern tools facilitate reverting and replaying changes without a lot of effort. > things quickly, imho, to hesitate a little on the revert. Especailly when > the broken thing is the playstation loader on powerpc that can stay broken > for the hour or six (or even days) it takes me to figure out why it > broke... Often things away from the beaten path don't get discovered for > days or weeks or months, and a reversion then can be extremely disruptive > if there's other changes layered on top of the offending commit.... Note that this isn't at all the issue under discussion in the FCP, which refers to issues that have already been detected by CI. For example, a commit which means amd64 panics on boot. Reverting quickly is a benefit in this sort of case found by CI precisely so that we don't end up with other changes on top that are difficult to unwind. From owner-freebsd-fcp@freebsd.org Thu Aug 29 23:35:44 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4D364E29B4 for ; Thu, 29 Aug 2019 23:35:44 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46KJrc0jdRz4K6d for ; Thu, 29 Aug 2019 23:35:44 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.nyi.freebsd.org (Postfix) id 168A5E29B2; Thu, 29 Aug 2019 23:35:44 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 154A8E29B1 for ; Thu, 29 Aug 2019 23:35:44 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KJrb1st5z4K6b for ; Thu, 29 Aug 2019 23:35:43 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x82d.google.com with SMTP id z4so5737920qtc.3 for ; Thu, 29 Aug 2019 16:35:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=OuE5G/e6FXGMIkJBV8BP3Wk+5bkDJcfztwL1SvRTBzw=; b=wy+1S0BELR6E4b2udOu2XvcXrdWNJlpPtozUulVNWd7KlACMl7CaLkBeItIVxb68II J51mSc1MSKClJswNZUDjrXNWenBhTYRcAQsqWao8hU98dQFaQPeY2O9yActusBOKL0d+ CvtiLdJ2bN1MfX+UvjSxA+0SCoL3lvu016hphgRl7+Qu1H0hqkeVLWdEI8Mt3x7/O0/V dHYxR/ZGHfNjjsyaBDKp34+PWfs/WGTM1NZrAh6JT7CusNGkOlRBsh2AmJI1bIxz5OzZ pcq9LmIR5T9iAdOsT5ykP1QyIBk/qsyBjPwJmkeXod5F9FlKUBpXkssi66vwm9hwrbno onhA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=OuE5G/e6FXGMIkJBV8BP3Wk+5bkDJcfztwL1SvRTBzw=; b=OG9Y0hOIDEJIc17nqxgHWE2ZfxGf6X9vi0YPtHkVRhF321X3pK03n6kDuj6Bm6QJdb +0e/Pu8dSFcEuuLr8IVEFF9oAGEfjA0NseeDlRCMKY58eiXfeJ1Rm+jr9pZ8VYmOzi+c 6p7HVNOZQ8hp1DBCj6BH5c5+d5lVcwa1Of3Jv5DWfPjb+TBiWXMqWY/OxEHTeWFG/olc zIzBMimTg9I4nq7ZxyXG+0Admv7072AB9ZQMEnJRrpQNE4GImC4QtlGQmtIMZpYMAuTf SZmw4AaIMT1XX8sz8n7D6wozrX8ARUU37+KgWs3bM/pSkZ46Pf0nYmVHZI2OqE865sVZ eKLA== X-Gm-Message-State: APjAAAWr685Gi81ew6ydblZ8cBTGnxHEfzJEi7oP0pmFhntL+YGOxdYZ HoJG6ZXqXcnuHnQ/t0EWiRJnj9Ez18qip1X6UwD/+A== X-Google-Smtp-Source: APXvYqzTMPTj69Ol2cCMDvT0Szma2ouUiUxKeSVLtqMCIutdNKm+dZxP09Xg8gGHjgbA8zLtCU++I6+o/7ECcIFRycQ= X-Received: by 2002:ac8:4602:: with SMTP id p2mr12594446qtn.291.1567121741918; Thu, 29 Aug 2019 16:35:41 -0700 (PDT) MIME-Version: 1.0 References: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> In-Reply-To: From: Warner Losh Date: Thu, 29 Aug 2019 17:35:29 -0600 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Ed Maste Cc: FreeBSD Hackers , fcp@freebsd.org X-Rspamd-Queue-Id: 46KJrb1st5z4K6b X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=wy+1S0BE; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::82d) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-5.92 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[fcp@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; RCVD_IN_DNSWL_NONE(0.00)[d.2.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.92)[ip: (-9.40), ipnet: 2607:f8b0::/32(-2.84), asn: 15169(-2.32), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 23:35:44 -0000 On Thu, Aug 29, 2019, 5:27 PM Ed Maste wrote: > On Thu, 29 Aug 2019 at 17:32, Warner Losh wrote: > > > > In the past, if someone had any follow on work at all in their tree, the > > reversion would be quite disruptive to that work. > > This sounds more like a problem with the tooling than an argument > against reverting though. > We live in a subversion universe for the moment, so you have to view it through that lens. I agree that in the case where the fix is straightforward it's > sensible, and in line with community norms, to just commit it. But in > the case that a regression was introduced by a committed change, > modern tools facilitate reverting and replaying changes without a lot > of effort. > Sometimes yes, sometimes no. Even with git svn, there is a cost associated with it. The level of effort is not zero. Especially when one pushes several interrelated changes at once. If the first of these caused an issue on gcc, say, often the cost is too high to revert the whole chain. It's a lot easier to put in a fix and move on. > things quickly, imho, to hesitate a little on the revert. Especailly when > > the broken thing is the playstation loader on powerpc that can stay > broken > > for the hour or six (or even days) it takes me to figure out why it > > broke... Often things away from the beaten path don't get discovered for > > days or weeks or months, and a reversion then can be extremely disruptive > > if there's other changes layered on top of the offending commit.... > > Note that this isn't at all the issue under discussion in the FCP, > which refers to issues that have already been detected by CI. For > example, a commit which means amd64 panics on boot. Reverting quickly > is a benefit in this sort of case found by CI precisely so that we > don't end up with other changes on top that are difficult to unwind. > It's a fair example for why a simpleminded approach will create more friction than the current system. And there is a need for caution in expanding the logic beyond all but the most recent changes... Warner > From owner-freebsd-fcp@freebsd.org Thu Aug 29 23:36:18 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 006CFE2A84 for ; Thu, 29 Aug 2019 23:36:18 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KJsF5Gkpz4KF2 for ; Thu, 29 Aug 2019 23:36:17 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id B4EB8E2A80; Thu, 29 Aug 2019 23:36:17 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B497DE2A7F; Thu, 29 Aug 2019 23:36:17 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f44.google.com (mail-io1-f44.google.com [209.85.166.44]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KJsD1Rflz4KDy; Thu, 29 Aug 2019 23:36:15 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f44.google.com with SMTP id j5so10394996ioj.8; Thu, 29 Aug 2019 16:36:15 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Ua4zHBQZ8l/telU2EihwTZk/176grRwgom62ZiitBoM=; b=QgMYZy0lLoHhNqJmpdUCMCKpoT7UCMpohCHDKkIVvskY21DExp/HhBzOWsGjLS1Hnx vFgqnbLgE4i+Wtf54WibFxtKs1SVYzM8BIQPrZwTc1XhPmc7XQtf2vyM1amANllUmIBR UEhNF+M6irPcNJJEz1HVe7JwoMb0EKD3nmlOB57GpAARlR3w7GqvBzTUb7/mwC4Bk+Xv jM2YOjWPYn1MEZtw/nR3jvJ8q1bxei3VrUAMQiIdbPa6yX3krzzuPzaGKNtYxBfV6bhG jlBWOkIpL0jcE/aDClP1x8/s7sxNuQwqj+NLudzHWxCVeHRZIdtmTQfbbNDXl6rkPAST j9rQ== X-Gm-Message-State: APjAAAWO53v7g+734sk5SnxCy9gm0tnpt5FzYgefnE/vVFNxMzPOYjy9 qif2PlAla5qULtRAxPDZVsb+4HAxwPU7NnHX2hs= X-Google-Smtp-Source: APXvYqyIya+E6staTKdL0E9iswUIHwjX0ue+iYsK6/O1VQFGcLqT7fJ/JyYx9r+rdZYDTMGdj1rwFFymq2XTS/lYRfI= X-Received: by 2002:a5d:9b96:: with SMTP id r22mr1006520iom.17.1567121775027; Thu, 29 Aug 2019 16:36:15 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> In-Reply-To: <20190829114057.GZ71821@kib.kiev.ua> From: Ed Maste Date: Thu, 29 Aug 2019 19:35:57 -0400 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Konstantin Belousov Cc: Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46KJsD1Rflz4KDy X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.44 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-5.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; RCVD_IN_DNSWL_NONE(0.00)[44.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.01)[ip: (-4.31), ipnet: 209.85.128.0/17(-3.34), asn: 15169(-2.32), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 23:36:18 -0000 On Thu, 29 Aug 2019 at 07:41, Konstantin Belousov wrote: > > More, I know that tests are of very low quality, which means that > brokeness of the tests is not an indicator of anything until root cause > is identified. "Low quality" needs clarification here. I can think of many attributes of a test that might lead someone to claim tests are low quality: - The test result is not consistent (e.g., a "flaky test") - The test does not actually test what it claims to - The test does as it claims, but there is no value in the result - Test coverage overall is insufficient (i.e., not an issue with a specific test) - The test has excessive requirements (run time, memory usage, etc.) - The test is difficult to maintain From owner-freebsd-fcp@freebsd.org Thu Aug 29 23:45:19 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0A78AE2DE1 for ; Thu, 29 Aug 2019 23:45:19 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46KK3f5YSSz4Klg for ; Thu, 29 Aug 2019 23:45:18 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id BCD05E2DDE; Thu, 29 Aug 2019 23:45:18 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BC82FE2DDD; Thu, 29 Aug 2019 23:45:18 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f54.google.com (mail-io1-f54.google.com [209.85.166.54]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KK3d4fxyz4Klf; Thu, 29 Aug 2019 23:45:17 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f54.google.com with SMTP id p12so10491877iog.5; Thu, 29 Aug 2019 16:45:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=0FSmYVskYbBYnGN185uLQL22o1T6D9MYbnRnoxDC67o=; b=YquEtH80LyW3R9CDO7QUZqCKeO5zjbWtVE6YWGMkcLMvlGXBfddG7Ce+fS1VC8IpHw JodIJyadRUhHtAFJijSvkHnoWd5sdUanOgKOQXgcLl0OuBq+nV/uj5DBF4W9xRBOkntF DwabNmKLapTfZQjcoTggiVhJ2VVICHst5+u8CLs7N+GzQb3Kt0TpqjYax8PVD9AzDQ6b yfej53kV8rH/C58O/YA8Nh+seaiLQl8wOAhuH6OPyyxsAyjSZyk3qAlxspwq0FHcNQgn kNjZxfwisDlQAnMgwFJXmPWpzYETFCScG1WizxG7TfU6vTeXH3XAYqno5FCkcJl+M9wa Dy6A== X-Gm-Message-State: APjAAAXgI3f8nh7PU5W2gVGtOtzkyf8JMfrfunKRYvRnxXbtheSaXOjs 7xZgqxAEJmS3X9ytFFnZGNIhcxBX3bhOxAx0exM= X-Google-Smtp-Source: APXvYqxgh0nMR4dvDtHiT+C0pqoSVClomevBvA1i9zsdwkPeAyVUjWvvFInhdvwOKiJlGFD0Rk5dDi2llkHn41/2ObY= X-Received: by 2002:a6b:3806:: with SMTP id f6mr1805827ioa.120.1567122316168; Thu, 29 Aug 2019 16:45:16 -0700 (PDT) MIME-Version: 1.0 References: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> In-Reply-To: From: Ed Maste Date: Thu, 29 Aug 2019 19:44:59 -0400 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Warner Losh Cc: FreeBSD Hackers , fcp@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 46KK3d4fxyz4Klf X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.54 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-5.49 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.993,0]; RCVD_IN_DNSWL_NONE(0.00)[54.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.50)[ip: (-6.78), ipnet: 209.85.128.0/17(-3.34), asn: 15169(-2.32), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 23:45:19 -0000 >> This sounds more like a problem with the tooling than an argument >> against reverting though. > > We live in a subversion universe for the moment, so you have to view it t= hrough that lens. Fair enough, right now the policy needs to accommodate the reality of the tools we're using today. Perhaps it's a failure of imagination on my part but I have trouble seeing how a revert would lead to losing work - could you give an example? > Sometimes yes, sometimes no. Even with git svn, there is a cost associate= d with it. The level of effort is not zero. Especially when one pushes sev= eral interrelated changes at once. If the first of these caused an issue on= gcc, say, often the cost is too high to revert the whole chain. It's a lot= easier to put in a fix and move on. The level of effort imposed on other users while the tree is broken is not zero, either. Certainly if it's possible to commit a fix and move forward that's the approach favoured by community norms. > It's a fair example for why a simpleminded approach will create more fric= tion than the current system. And there is a need for caution in expanding = the logic beyond all but the most recent changes... The point of the FCP is to facilitate the revert while the change is (among the) most recent, precisely so that additional changes don't build on it. From owner-freebsd-fcp@freebsd.org Thu Aug 29 23:54:50 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0C693E30DC for ; Thu, 29 Aug 2019 23:54:50 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46KKGd5Fzmz4LDW for ; Thu, 29 Aug 2019 23:54:49 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id B2968E30DA; Thu, 29 Aug 2019 23:54:49 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B248FE30D9; Thu, 29 Aug 2019 23:54:49 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f53.google.com (mail-io1-f53.google.com [209.85.166.53]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KKGd4Hbgz4LDV; Thu, 29 Aug 2019 23:54:49 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f53.google.com with SMTP id u185so6683695iod.10; Thu, 29 Aug 2019 16:54:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=PoIWWx4fSq3VskXld9iN/CzgUfJmJ9GAd0lWF1+bQzY=; b=kSxcHhOQY5xSZmi1v0CfPJolAfXwi3XG7HJUs94d4OLBJIxFd6AjE2Z00/gO+0bx1j 8WbuyskOQwWwmbZquBmvuHSCtAunVrStyFwGXwJYZAGqU6e/sPjoF7Y8huyOmkMAOcDe ob0ab0hCE/s4CrMDYmbcxlRWAqIDph35n7QbwUkGWjJwW1ndvYpvs7UFYLyg/lDlMRbl CF1buuiV/RUPCqiAdjy9zY9ean8t5n/3lH1jF0yUDXdDjmBMwCTYN9aac5p/rCyVWzaI CPU9LnO/DVIowH3Zm9spLAV2wOOyi7XazI1pkjyiRm5WJRuHzSSpyyPD5oyMeabZH/ni xZAA== X-Gm-Message-State: APjAAAX+VeJIuYqqYn5DEosT5L8shYZqK5pfz55x5QdcTDGNFSxVcZ2v DDkuNayOnJBXfJAVt51uBXj5GeGpt/ZODQ5l2uYcFA== X-Google-Smtp-Source: APXvYqxw0zz6Avaf3xljxYeS7M1ZnHfStnwytUNEyLTK25j9Swu5Ho1oyV3LW4zE0PPdjvTz/Ix8+G2XvsLN78LjbIo= X-Received: by 2002:a5d:9b96:: with SMTP id r22mr1084304iom.17.1567122886864; Thu, 29 Aug 2019 16:54:46 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> In-Reply-To: From: Ed Maste Date: Thu, 29 Aug 2019 19:54:29 -0400 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Marcelo Araujo Cc: Kristof Provost , Konstantin Belousov , FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46KKGd4Hbgz4LDV X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.98 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.98)[-0.980,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Aug 2019 23:54:50 -0000 On Thu, 29 Aug 2019 at 10:47, Marcelo Araujo wrote: > > One last thing I'd like to say: We should stop to try to control FreeBSD > contributions, it is not doing good for the project. Let it be, let it go, > there is no owner, just go and fix things. I don't see things that way. Other projects I contribute to (like LLVM) have a comprehensive test suite, and a culture that expects tests to be run before commit. Many projects run the tests automatically as part of the contribution process (e.g. GitHub pull requests). Rather than being some bureaucratic hurdle this actually supports casual contributors - because the contributor has more confidence that their change isn't going to break the build or introduce a regression, and doesn't need to remain attentive for the next days or weeks in case of unexpected failures as a result of the change. From owner-freebsd-fcp@freebsd.org Fri Aug 30 00:01:00 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5F1AAE34D2 for ; Fri, 30 Aug 2019 00:01:00 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KKPl6JXkz4LV5 for ; Fri, 30 Aug 2019 00:00:59 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id D8684E34D0; Fri, 30 Aug 2019 00:00:59 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D809EE34CF; Fri, 30 Aug 2019 00:00:59 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f51.google.com (mail-io1-f51.google.com [209.85.166.51]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KKPk5QPVz4LV1; Fri, 30 Aug 2019 00:00:58 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f51.google.com with SMTP id n197so8538092iod.9; Thu, 29 Aug 2019 17:00:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=TZytS+qLdG33lUssuiKaFq2nPipy4oB5XPa0SwhYjtg=; b=MWyHBS0B8BapQAaXz7ql1Hnt0RBxjCrjIVBHlg3PRA0ZCV/c9lkafsWrsiqAXSkFGs 8AFGmC2+e2ErhHx8/BoZXJYLzjxi+7IA1EbOvA4IOTpyl2d2btdEoh4NZBPhHb28vyuq wScCp+W6K7lFnVr3/rXZvM4rzh1jyqbE3QAfP5L2queRInd6pbUFIP0vVSbg8tsGUx73 jLZg+DC+fDNNnBiArWHYdG9V03wtgeI3Wt6PTgfaiA9rXhHsmkFBcYheAX6UU9sTJETH rJIkjEFc0vnw754vk7qNNCqOsHyxbnwTxs3yMNihHzlOWVAgxvbvO3xRJKN6PMGHCJqY 8JhA== X-Gm-Message-State: APjAAAV9yLeeIUrjGHbxEtpvDg424LNpkBduMeawsB15VswURzG2FYCP nnLpnIkE7y7hkLVkoPC6lKqN2AdYr/4LJw2FwFs= X-Google-Smtp-Source: APXvYqwkSda+4/rES3/SE85Qy3NmavwPXy57WQVVKqnVZBKioRGX5foRbFpzBY2ZhvncpUKh5YMitnbu412iijsP9QQ= X-Received: by 2002:a5d:9b96:: with SMTP id r22mr1111435iom.17.1567123257701; Thu, 29 Aug 2019 17:00:57 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <412537DD-D98F-4B92-85F5-CB93CF33F281@FreeBSD.org> <20190829144228.GA71821@kib.kiev.ua> <20190829153712.GB71821@kib.kiev.ua> In-Reply-To: <20190829153712.GB71821@kib.kiev.ua> From: Ed Maste Date: Thu, 29 Aug 2019 20:00:41 -0400 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Konstantin Belousov Cc: Kristof Provost , FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46KKPk5QPVz4LV1 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.51 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-5.49 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.990,0]; RCVD_IN_DNSWL_NONE(0.00)[51.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.50)[ip: (-6.77), ipnet: 209.85.128.0/17(-3.34), asn: 15169(-2.32), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 00:01:00 -0000 On Thu, 29 Aug 2019 at 11:37, Konstantin Belousov wrote: > > I fully agree with your attitude there, and understand your frustration. > IMO the right action would be to contact the committers who did the > relevant changes, first. Was it done ? What was their response ? If we had tests running consistently at the time an offending change was introduced we might know that :) This is exactly the reason I want us to have a large corpus of tests that are, as a rule, expected to consistently pass. As far as I can tell from a cursory investigation this is a flaw that predates the pf tests, and just happens to be demonstrated by them. From owner-freebsd-fcp@freebsd.org Fri Aug 30 00:05:24 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CCC21E3DBE for ; Fri, 30 Aug 2019 00:05:24 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KKVr21SXz4P1h for ; Fri, 30 Aug 2019 00:05:24 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 10907E3D83; Fri, 30 Aug 2019 00:05:24 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0EF80E3D82; Fri, 30 Aug 2019 00:05:24 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f54.google.com (mail-io1-f54.google.com [209.85.166.54]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KKVq23Pbz4Nyd; Fri, 30 Aug 2019 00:05:23 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f54.google.com with SMTP id t6so10519715ios.7; Thu, 29 Aug 2019 17:05:23 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=URxnojM7DYm+fSQKO6xIxHE6v96bixxiCb42ycf6FKo=; b=lUbR8XSKlXf2nIXW6/6wJaQU1Xl2x5sbTxMLHSjPzCe7Tpn0obN+y8VlB6UsrcVxpJ P2IwsOznnyxPnjeuu3Kxk5ylU6bg9swQriXISx3+mNufaGfqYUTCFgjiA3KvH+137aqj 2aY5dlkicc9fIy5pKBvMVPJyPoZ0zBRmzbbWILPnysim7jhittO8/S049ZEFaOfU8Drz XP/NLZnc41sOcG63I2hucqby6SMcn8tH4TFNGgmXCRTbSxdwvFbnGyL7t5gg9aHOtUv1 OIEK3DobKxfQoKzNyEPsLlGhMBsXHsn1QauoAH04tXXkd8TACGGtsI/Ld+fthwO8Yyqh 0FbQ== X-Gm-Message-State: APjAAAVJobdFZ90xNN/ndidBr8AvkZMrlNIrH/HnQ6JfoaIvXLqlr7wE 7Rq/x2bkN6Y44CEpTLgPjsDi7nUe0x+tBNaDri1buA== X-Google-Smtp-Source: APXvYqx0xvT5UR1XEZJhksJ4+mAwv4OS2uxkMCylQPLys5IW+ibcg9+8XtHj4VFYPV5wH9feBFo/B/RHzSEMMXXPzRU= X-Received: by 2002:a6b:ea16:: with SMTP id m22mr8361491ioc.115.1567123522546; Thu, 29 Aug 2019 17:05:22 -0700 (PDT) MIME-Version: 1.0 References: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> In-Reply-To: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> From: Ed Maste Date: Thu, 29 Aug 2019 20:05:06 -0400 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: "Rodney W. Grimes" Cc: FreeBSD Hackers , fcp@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46KKVq23Pbz4Nyd X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.54 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-5.46 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.993,0]; RCVD_IN_DNSWL_NONE(0.00)[54.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.46)[ip: (-6.60), ipnet: 209.85.128.0/17(-3.34), asn: 15169(-2.32), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 00:05:24 -0000 On Thu, 29 Aug 2019 at 15:05, Rodney W. Grimes wrote: > > Here in lies one of the fundemental problems, this view by some that > a "revert commit is something bad, it is kind of punishment". That is > not true. Reverts are GREAT things, they allow the tree to be returned > to a known state, usually quicly. The original commit is STILL IN SVN, > and a bad revert can guess what.. be reverted!. Let me echo Rod here. I'm also very happy that this statement was made by one of the original FreeBSD committers. Reverting a change is not an insult, not a punishment, not something bad - it's simply an acknowledgement that some aspect of the change didn't meet expectations. From owner-freebsd-fcp@freebsd.org Fri Aug 30 00:19:16 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8AAC7E45C6 for ; Fri, 30 Aug 2019 00:19:16 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KKpr2RJWz4Qd5 for ; Fri, 30 Aug 2019 00:19:16 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.nyi.freebsd.org (Postfix) id 51553E45C5; Fri, 30 Aug 2019 00:19:16 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 510F6E45C4 for ; Fri, 30 Aug 2019 00:19:16 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x72e.google.com (mail-qk1-x72e.google.com [IPv6:2607:f8b0:4864:20::72e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KKpq2nLWz4Qd2 for ; Fri, 30 Aug 2019 00:19:15 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x72e.google.com with SMTP id i78so3215075qke.11 for ; Thu, 29 Aug 2019 17:19:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=j+SRZ5kUCBdK4V6FTFHS6+HnVnHlwC8XmWCMrEVfwHI=; b=OWeFDN3bJkNaDnRPUwTvNCeugNyhD4xZ34GEAduGSdcAywKWcb7gIbmJcHopvj9qt6 W99K1qGIV7nmZirLgfqeUx0lPHmDAUVKMU6XdJy7/QShvY4+UBzM+kb4FhmOegIm7f86 YJwwpgxNmRGQFhPhrnHyB909IMqi3BolNl2TaLLhj9HC/jCzOw/pN2DKVNOkrUiVh2mJ 90/QDLeTJKN2vXdidemzUg6fjyzrj9G/DZN860AeAMNooPxPmbHdyghjf62qhfNcMupe CTnl+dlI6H6sOfE7OWf75PuJLzvf9f665Od+20nkrtwXTv1W0k5sOQFUCwcQC7G3h3rs 0MMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=j+SRZ5kUCBdK4V6FTFHS6+HnVnHlwC8XmWCMrEVfwHI=; b=Al2If/0TrCPoX6wYoKZwvfyjOHPCBFpfLQkZLDNqIQ52iChTQc+y2CnUN9C85wf3JN zqj8YiJojb77m+n+nifil2JpwczSDL0xeDl1qSJamDz7gLIh78lo9R6BkWtcl09nsIld Orys7f5GStHrxr0hUk6d5XF5w/t9BcjDUPkWTupclHaafInNFdbNpp+bblVkH24MdCOV rkRzwlAv0mBg+mSKZFMW9zADGSbBDT1RcwK1uy5ip+obYn9yw59Fu088wFJyZoTsRoAq CHFR75xqdSZ63MYlUVUis+o6I5qxhYC8Fcg4jGQV6Wwe9vKCi/1FGPltkcAHPba6larj jGpQ== X-Gm-Message-State: APjAAAVrtZIQ6ICy4gwDKceee+HUa2XIvKa5HchbaC2r0xNSVAgSxSOy UmRShLat1dHmRfTGcPyJfQ+iabrpc0kYpICB5/u+vQ== X-Google-Smtp-Source: APXvYqyp1HtSIHpIEncgQbk5/qWyYzY5hebGyJRUdq3pgDtVRBbNWjTHqbXWgSN1y2Nh0mhffhqtpWVAY+ot7BFBNn4= X-Received: by 2002:a37:8902:: with SMTP id l2mr12348810qkd.380.1567124354153; Thu, 29 Aug 2019 17:19:14 -0700 (PDT) MIME-Version: 1.0 References: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> In-Reply-To: From: Warner Losh Date: Thu, 29 Aug 2019 18:19:02 -0600 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Ed Maste Cc: FreeBSD Hackers , fcp@freebsd.org X-Rspamd-Queue-Id: 46KKpq2nLWz4Qd2 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=OWeFDN3b; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::72e) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-5.91 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[fcp@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; RCVD_IN_DNSWL_NONE(0.00)[e.2.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.91)[ip: (-9.34), ipnet: 2607:f8b0::/32(-2.84), asn: 15169(-2.32), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 00:19:16 -0000 On Thu, Aug 29, 2019 at 5:45 PM Ed Maste wrote: > >> This sounds more like a problem with the tooling than an argument > >> against reverting though. > > > > We live in a subversion universe for the moment, so you have to view it > through that lens. > > Fair enough, right now the policy needs to accommodate the reality of > the tools we're using today. > > Perhaps it's a failure of imagination on my part but I have trouble > seeing how a revert would lead to losing work - could you give an > example? > I've twice lost work due to a hasty reversion in the past. In my case, when I did the svn update it ended badly due to the conflicts that were generated. Most of the times I've hit conflicts in the past, it's just been the work of a conflict. When the work was larger, svn got a bit confused and I wound up losing several chunks of work. I was able to redo it, but it was annoying. It's why a super-fast automatic revert without contact with the person that committed it is a bad idea. Of course, if that person falls down in fixing it, I support doing something. Tools have improved, and perhaps this is a case of once bitten twice shy... > > Sometimes yes, sometimes no. Even with git svn, there is a cost > associated with it. The level of effort is not zero. Especially when one > pushes several interrelated changes at once. If the first of these caused > an issue on gcc, say, often the cost is too high to revert the whole chain. > It's a lot easier to put in a fix and move on. > > The level of effort imposed on other users while the tree is broken is > not zero, either. Certainly if it's possible to commit a fix and move > forward that's the approach favoured by community norms. > I agree. My pushback here is against the notion there's zero cost to a revert. Of course we need balance the damage to others vs the impact on the contributor. When the impact is in -current on a fringe platform, we need to not over-react to fixing that by back out. > > It's a fair example for why a simpleminded approach will create more > friction than the current system. And there is a need for caution in > expanding the logic beyond all but the most recent changes... > > The point of the FCP is to facilitate the revert while the change is > (among the) most recent, precisely so that additional changes don't > build on it. > Agreed. I just don't want to swing too far to the automatic end of things, and to apply some level of judgement when there's more than one change involved. Warner From owner-freebsd-fcp@freebsd.org Fri Aug 30 01:35:50 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 70028E5D95 for ; Fri, 30 Aug 2019 01:35:50 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KMWB0Qqmz4VHh for ; Fri, 30 Aug 2019 01:35:50 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: by mailman.nyi.freebsd.org (Postfix) id 0E866E5D92; Fri, 30 Aug 2019 01:35:50 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0E25DE5D91; Fri, 30 Aug 2019 01:35:50 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KMW846DWz4VHg; Fri, 30 Aug 2019 01:35:48 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id 3VpHi2s32sAGk3VpIitl9V; Thu, 29 Aug 2019 19:35:46 -0600 X-Authority-Analysis: v=2.3 cv=WeVylHpX c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=8nJEP1OIZ-IA:10 a=FmdZ9Uzk2mMA:10 a=6I5d2MoRAAAA:8 a=YxBL1-UpAAAA:8 a=duxrOwH0z9cosxiZrWsA:9 a=wPNLvfGTeEIA:10 a=IjZwj45LgO3ly-622nXo:22 a=Ia-lj3WSrqcvXOmTRaiG:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTPS id 98DC4725; Thu, 29 Aug 2019 18:35:42 -0700 (PDT) Received: from slippy.cwsent.com (localhost [127.0.0.1]) by slippy.cwsent.com (8.15.2/8.15.2) with ESMTP id x7U1Zg6M067244; Thu, 29 Aug 2019 18:35:42 -0700 (PDT) (envelope-from Cy.Schubert@cschubert.com) Received: from slippy (cy@localhost) by slippy.cwsent.com (8.15.2/8.15.2/Submit) with ESMTP id x7U1Zfh7067241; Thu, 29 Aug 2019 18:35:41 -0700 (PDT) (envelope-from Cy.Schubert@cschubert.com) Message-Id: <201908300135.x7U1Zfh7067241@slippy.cwsent.com> X-Authentication-Warning: slippy.cwsent.com: cy owned process doing -bs X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.7.1 Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: "Kristof Provost" cc: araujo@freebsd.org, Konstantin Belousov , FreeBSD Hackers , Li-Wen Hsu , Ian Lepore , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy In-reply-to: References: <20190829114057.GZ71821@kib.kiev.ua> <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> Comments: In-reply-to "Kristof Provost" message dated "Thu, 29 Aug 2019 17:09:32 +0200." Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Date: Thu, 29 Aug 2019 18:35:41 -0700 X-CMAE-Envelope: MS4wfCbf9EUat0hnmBN/VxddWxCz2VbiMs9RNbe9T2Rpl2FPyFF9yIjEFS3WPs2NbFDiNbEojxO8rUhOpBG8drXgSquMESIPMpGSN5Ofhq2zAo9nYNFmh6Le K6V3XJS434syjkd5snUoXy90o3NGoEfEgGlEtUyDhAZ570GTu82U0t1EwyEp69EDXkM1J0zD6+qaP+772Sxu1KI4OFfyZ+7+A1iQLp4ipWecHEHmuZsJ4B5X MWPI73lCwlqC3H7dKfzeMUa/BFZsW1/QrMlvOXr0RUdlCDGq+ZYkLglYuhoQPVrtPn2HHCgxd25MdU6MHnvfTw== X-Rspamd-Queue-Id: 46KMW846DWz4VHg X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 64.59.134.9) smtp.mailfrom=cy.schubert@cschubert.com X-Spamd-Result: default: False [-4.95 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; REPLYTO_EQ_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.95)[-0.952,0]; RCVD_IN_DNSWL_NONE(0.00)[9.134.59.64.list.dnswl.org : 127.0.5.0]; RCPT_COUNT_SEVEN(0.00)[7]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6327, ipnet:64.59.128.0/20, country:CA]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-2.40)[ip: (-6.35), ipnet: 64.59.128.0/20(-3.13), asn: 6327(-2.43), country: CA(-0.09)]; RECEIVED_SPAMHAUS_PBL(0.00)[17.125.67.70.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 01:35:50 -0000 In message , "Kristof Provost " writes: > On 29 Aug 2019, at 17:01, Marcelo Araujo wrote: > > Em qui, 29 de ago de 2019 às 22:54, Kristof Provost > > escreveu: > > > >> On 29 Aug 2019, at 16:42, Ian Lepore wrote: > >>> (And I don't think breaking a test counts as > >>> breaking the build.) > >>> > >> I fundamentally disagree on this point. A test failure is, just like > >> a > >> compiler warning, a precious gift that should not be ignored. > >> The more distance (both in terms of time, and in terms of the people > >> involved) there is between a bug being introduced and it being > >> detected > >> the harder it is to fix it. Test accelerate detection of bugs. If we > >> do > >> not take test failures seriously (i.e. as an indication something is > >> wrong and should be fixed) the tests will inevitable become useless > >> in > >> one of two ways: we’ll either disable failing tests (which is what > >> we > >> tend to do now) reducing test coverage or we’ll have a test suite > >> with > >> many failures in it, which makes it useless as well. (As with > >> compiler > >> warnings, the best way to keep them under control is to consider them > >> to > >> be fatal errors.) > >> > > > > Could you elaborate where is the "fundamentally" you disagree? Where > > is the > > fundament? You guys are introducing something new, yes everybody knows > > about test, it is year 2019, but nobody can come with new rules tha in > > hours we gonna revert if you "dare to don't fix it". Sorry, this is > > not how > > people test software and fix it. > > > I do think that breaking a test breaks the build. Something used to work > and now it doesn’t. That’s breakage, even if it’s not as total as > it not compiling any more. I agree. A broken test is an indication of a regression. If a test is not a regression, then it is not a valid test. Having said that, 48 hours is a little draconian. Maybe 48 hours if no one steps up to the plate but if it's actively being worked on with end in sight I'd give the person working on it a little space. What about weekends? I notice commits tend to slow down during weekends. I suspect people who work on FreeBSD at $COMPANY for a living treat it as a job, which it is. However this should be considered. OTOH (arguing against myself to point out another consideration), it is foolhardy to commit a new feature just before a weekend and go camping (or be unreachable for whatever reason). A committer should be reachable N hours after a significant commit. Maybe 48 hours is justifiable then. (At $JOB we don't allow significant changes prior to vacation or even a flex day unless there is someone to cover for the person having done the work.) My point is, we pick a number like 48. What's the rationale? What is the desired result? Can the desired result be accomplished without a 48 hour auto-revert rule? Should it be 48 hours during the week and 72 over weekends? -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. From owner-freebsd-fcp@freebsd.org Fri Aug 30 01:58:17 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B4101E6297 for ; Fri, 30 Aug 2019 01:58:17 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46KN151xJ1z4Vy8 for ; Fri, 30 Aug 2019 01:58:17 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: by mailman.nyi.freebsd.org (Postfix) id 3FD33E6295; Fri, 30 Aug 2019 01:58:17 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3E650E6294; Fri, 30 Aug 2019 01:58:17 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (troutmask.apl.washington.edu [128.95.76.21]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "troutmask", Issuer "troutmask" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KN141ZZhz4Vy7; Fri, 30 Aug 2019 01:58:15 +0000 (UTC) (envelope-from sgk@troutmask.apl.washington.edu) Received: from troutmask.apl.washington.edu (localhost [127.0.0.1]) by troutmask.apl.washington.edu (8.15.2/8.15.2) with ESMTPS id x7U1w6nP016918 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 29 Aug 2019 18:58:06 -0700 (PDT) (envelope-from sgk@troutmask.apl.washington.edu) Received: (from sgk@localhost) by troutmask.apl.washington.edu (8.15.2/8.15.2/Submit) id x7U1w6q2016917; Thu, 29 Aug 2019 18:58:06 -0700 (PDT) (envelope-from sgk) Date: Thu, 29 Aug 2019 18:58:05 -0700 From: Steve Kargl To: Cy Schubert Cc: Kristof Provost , Ian Lepore , FreeBSD Hackers , araujo@freebsd.org, fcp@freebsd.org, Konstantin Belousov , Li-Wen Hsu Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190830015805.GA16894@troutmask.apl.washington.edu> Reply-To: sgk@troutmask.apl.washington.edu References: <20190829114057.GZ71821@kib.kiev.ua> <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> <201908300135.x7U1Zfh7067241@slippy.cwsent.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201908300135.x7U1Zfh7067241@slippy.cwsent.com> User-Agent: Mutt/1.12.1 (2019-06-15) X-Rspamd-Queue-Id: 46KN141ZZhz4Vy7 X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of sgk@troutmask.apl.washington.edu has no SPF policy when checking 128.95.76.21) smtp.mailfrom=sgk@troutmask.apl.washington.edu X-Spamd-Result: default: False [1.18 / 15.00]; ARC_NA(0.00)[]; HAS_REPLYTO(0.00)[sgk@troutmask.apl.washington.edu]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.45)[0.445,0]; IP_SCORE(-0.14)[ip: (0.06), ipnet: 128.95.0.0/16(-0.07), asn: 73(-0.62), country: US(-0.05)]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[washington.edu]; REPLYTO_ADDR_EQ_FROM(0.00)[]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.62)[0.623,0]; NEURAL_HAM_LONG(-0.65)[-0.647,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_SEVEN(0.00)[8]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:73, ipnet:128.95.0.0/16, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 01:58:17 -0000 On Thu, Aug 29, 2019 at 06:35:41PM -0700, Cy Schubert wrote: > > What about weekends? I notice commits tend to slow down during weekends. I > suspect people who work on FreeBSD at $COMPANY for a living treat it as a > job, which it is. However this should be considered. > A committer, who works for a $COMPANY and has no intention of looking at the mailing lists or CI reports until Monday, should not commit on Firday. What ever happen to common sense? -- Steve From owner-freebsd-fcp@freebsd.org Fri Aug 30 02:01:12 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DBAE9E63FC for ; Fri, 30 Aug 2019 02:01:12 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KN4S3hmWz4WFw for ; Fri, 30 Aug 2019 02:01:12 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: by mailman.nyi.freebsd.org (Postfix) id 7E9BEE63FA; Fri, 30 Aug 2019 02:01:12 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7E4A3E63F9; Fri, 30 Aug 2019 02:01:12 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.13]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KN4Q2qRvz4WFn; Fri, 30 Aug 2019 02:01:09 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id 3WDqi33ZosAGk3WDritoMk; Thu, 29 Aug 2019 20:01:08 -0600 X-Authority-Analysis: v=2.3 cv=WeVylHpX c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=kj9zAlcOel0A:10 a=FmdZ9Uzk2mMA:10 a=7Qk2ozbKAAAA:8 a=iKhvJSA4AAAA:8 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=GD4Lj2ni4_LfeXk_s_gA:9 a=CjuIK1q_8ugA:10 a=HNhVPpsFFhwA:10 a=1lyxoWkJIXJV6VJUPhuM:22 a=odh9cflL3HIXMm4fY7Wr:22 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTPS id EE3C3754; Thu, 29 Aug 2019 19:01:04 -0700 (PDT) Received: from slippy.cwsent.com (localhost [127.0.0.1]) by slippy.cwsent.com (8.15.2/8.15.2) with ESMTP id x7U214i0086083; Thu, 29 Aug 2019 19:01:04 -0700 (PDT) (envelope-from Cy.Schubert@cschubert.com) Received: from slippy (cy@localhost) by slippy.cwsent.com (8.15.2/8.15.2/Submit) with ESMTP id x7U214qn086080; Thu, 29 Aug 2019 19:01:04 -0700 (PDT) (envelope-from Cy.Schubert@cschubert.com) Message-Id: <201908300201.x7U214qn086080@slippy.cwsent.com> X-Authentication-Warning: slippy.cwsent.com: cy owned process doing -bs X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.7.1 Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: Warner Losh cc: "Rodney W. Grimes" , Ian Lepore , FreeBSD Hackers , Marcelo Araujo , fcp@freebsd.org, Konstantin Belousov , Li-Wen Hsu , Kristof Provost Subject: Re: FCP 20190401-ci_policy: CI policy In-reply-to: References: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> Comments: In-reply-to Warner Losh message dated "Thu, 29 Aug 2019 15:32:17 -0600." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 29 Aug 2019 19:01:04 -0700 X-CMAE-Envelope: MS4wfCI6Bkf8Ut4g4MzXQp1XYy6ido4ceg6QBsesfQUY+aMQgaACX9ydSCxK/0xBNpj1JgJNpgdrqFAlnwpWR7bWWfmMt6xAWe1lYac7jxAcBPVcRzjMjeSL 5y8ItgPdDptRmOG4F6GNUBNBbe+9YykOS4laGW8S7DggEwYagGaoeFS998kXhOCYHH+6OcJnbRS+LPaZLzF1Hd/N/qnAKkIBtGsJraoGjVKz2PhQ1nJJSCNG ncuXy6NRannaA/DQivkgak67BVWVKBnrCu1RuV1nzvMpz/XmIzVsJni6ja5x4dnyfoVuX2uPewpprvjKG5jGOVHQep68RHe9FPzLYZzezDFQC4hiOVOM7r9f yW7AxOPYOweVgafdCJap9it7RXBYgYEP/GvV2qOhpaV0EPhcIOk= X-Rspamd-Queue-Id: 46KN4Q2qRvz4WFn X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 64.59.134.13) smtp.mailfrom=cy.schubert@cschubert.com X-Spamd-Result: default: False [-5.03 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; REPLYTO_EQ_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.95)[-0.952,0]; RCVD_IN_DNSWL_NONE(0.00)[13.134.59.64.list.dnswl.org : 127.0.5.0]; RCPT_COUNT_SEVEN(0.00)[9]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6327, ipnet:64.59.128.0/20, country:CA]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-2.48)[ip: (-6.76), ipnet: 64.59.128.0/20(-3.13), asn: 6327(-2.43), country: CA(-0.09)]; RECEIVED_SPAMHAUS_PBL(0.00)[17.125.67.70.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 02:01:12 -0000 In message , Warner Losh writes: > On Thu, Aug 29, 2019 at 3:26 PM Warner Losh wrote: > > > > > > > On Thu, Aug 29, 2019 at 1:05 PM Rodney W. Grimes < > > freebsd-rwg@gndrsh.dnsmgr.net> wrote: > > > >> (unneeded context removed) > >> > >> > > In either scenario we end up reducing test coverage, which means we?re > >> > > going to push more bugs towards users. > >> > > > >> > > > I totally agree. This is an overly-bureaucratic solution in search > >> of > >> > > > a problem. > >> > > > > >> > > > If this needs to be addressed at all (and I'm not sure it does), > >> then > >> > > > another sentence or two in bullet item 10 in section 18.1 [*] of the > >> > > > committer's guide should be enough. And even then it needn't be > >> > > > overly-formal and should just mention that if a commit does break > >> the > >> > > > build the committer is expected to be responsive to that problem and > >> > > > the commit might get reverted if they're unresponsive. I don't > >> think > >> > > > we need schedules. > >> > > > > >> > > I do feel that?s a better argument. We?ve always had a policy of > >> > > reverting on request (AIUI), so this is more or less trying to be a > >> > > strong restatement of that, more than a fundamental shift in policy. > >> > > > >> > > >> > We don't have a policy to revert commit, actually revert commit is > >> > something bad, it is kind of punishment, I have been there, nobody > >> wants to > >> > be there. Stop to push this non-sense argument. > >> > >> Here in lies one of the fundemental problems, this view by some that > >> a "revert commit is something bad, it is kind of punishment". That is > >> not true. Reverts are GREAT things, they allow the tree to be returned > >> to a known state, usually quicly. The original commit is STILL IN SVN, > >> and a bad revert can guess what.. be reverted!. > >> > >> IMHO the project as a whole needs to overcome its fear of reverts and > >> start to use them for the great and powerful things that they are. > >> > >> This connection of bad and punishment needs to stop, and the sooner > >> the better. > >> > > > In the past, if someone had any follow on work at all in their tree, the > reversion would be quite disruptive to that work. Most of the time it's a > lot easier for me, with a lot less friction, to just fix issues that come > up after the commit than to revert and prepare a new commit. Sure, it's > possible, but it can destroy work in extreme cases. *THAT* is why I'm > firmly in the camp of giving the original committer a shot at fixing things > because it's much less disruptive to them, and generally we can get a fix > into the tree faster. It reduces friction and encourages people to fix > things quickly, imho, to hesitate a little on the revert. Especailly when > the broken thing is the playstation loader on powerpc that can stay broken > for the hour or six (or even days) it takes me to figure out why it > broke... Often things away from the beaten path don't get discovered for > days or weeks or months, and a reversion then can be extremely disruptive > if there's other changes layered on top of the offending commit.... > > So the whole reversion issue is a lot more complicated than 'oh, it's still > in svn'. There are real high costs associated with being too quick or > liberal on the revert and those must be weighed against the damage the bad > commit is doing.. I think that's why we need to define the problem first. The justification of the arbitrary numbers of minutes/hours isn't clear. I see there are possibly two trains of thought here which need to be separated. 1. A general frustration by some. 2. A tool, a solution to a problem, I am unsure if it is related to #1. Why do I see this as such? The problem statement beings by saying that FreeBSD has a CI system that performs compiles and runs automated tests. In the next paragraph it says sometimes changes break compilation... This tells me that A) we have a solution which we discuss in B (the problem). To my thinking we need to approach this from: A) we have a problem, maybe backing it up with some stats some evidence of sorts. Then explore one or preferably two solutions. Not to be hard on anyone and keeping my emotions out of it, they way the problem statement is structured reads to me as a solution looking for a problem. That's not to say we don't have a problem (nor am I saying we do have a problem either). The problem statement is structured to a foregone conclusion. I'd structure this by stating the problem (paragraph 2 and the bullet points, though I think the problem needs to be explored a little more), then discuss some of the timing issues regarding the fixing of the three types of problems (compile, panic, and regressions, of which tests identify regressions). I'm not saying that there is or isn't a problem but the problem statement as written doesn't convince me there is a problem. It's leading me to a conclusion. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. From owner-freebsd-fcp@freebsd.org Fri Aug 30 02:02:32 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 311D9E665D for ; Fri, 30 Aug 2019 02:02:32 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KN5z72GZz4WSb for ; Fri, 30 Aug 2019 02:02:31 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: by mailman.nyi.freebsd.org (Postfix) id F157FE665A; Fri, 30 Aug 2019 02:02:31 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id F1076E6658; Fri, 30 Aug 2019 02:02:31 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from smtp-out-no.shaw.ca (smtp-out-no.shaw.ca [64.59.134.12]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KN5y54r1z4WSX; Fri, 30 Aug 2019 02:02:30 +0000 (UTC) (envelope-from cy.schubert@cschubert.com) Received: from spqr.komquats.com ([70.67.125.17]) by shaw.ca with ESMTPA id 3WF9i34D3sAGk3WFBitobe; Thu, 29 Aug 2019 20:02:29 -0600 X-Authority-Analysis: v=2.3 cv=WeVylHpX c=1 sm=1 tr=0 a=VFtTW3WuZNDh6VkGe7fA3g==:117 a=VFtTW3WuZNDh6VkGe7fA3g==:17 a=kj9zAlcOel0A:10 a=FmdZ9Uzk2mMA:10 a=YxBL1-UpAAAA:8 a=6I5d2MoRAAAA:8 a=kjVYAFG_BOeK_iIailIA:9 a=CjuIK1q_8ugA:10 a=Ia-lj3WSrqcvXOmTRaiG:22 a=IjZwj45LgO3ly-622nXo:22 Received: from slippy.cwsent.com (slippy [10.1.1.91]) by spqr.komquats.com (Postfix) with ESMTPS id 88E38761; Thu, 29 Aug 2019 19:02:27 -0700 (PDT) Received: from slippy.cwsent.com (localhost [127.0.0.1]) by slippy.cwsent.com (8.15.2/8.15.2) with ESMTP id x7U22RlP087313; Thu, 29 Aug 2019 19:02:27 -0700 (PDT) (envelope-from Cy.Schubert@cschubert.com) Received: from slippy (cy@localhost) by slippy.cwsent.com (8.15.2/8.15.2/Submit) with ESMTP id x7U22RaV087308; Thu, 29 Aug 2019 19:02:27 -0700 (PDT) (envelope-from Cy.Schubert@cschubert.com) Message-Id: <201908300202.x7U22RaV087308@slippy.cwsent.com> X-Authentication-Warning: slippy.cwsent.com: cy owned process doing -bs X-Mailer: exmh version 2.9.0 11/07/2018 with nmh-1.7.1 Reply-to: Cy Schubert From: Cy Schubert X-os: FreeBSD X-Sender: cy@cwsent.com X-URL: http://www.cschubert.com/ To: sgk@troutmask.apl.washington.edu cc: Cy Schubert , Kristof Provost , Ian Lepore , FreeBSD Hackers , araujo@freebsd.org, fcp@freebsd.org, Konstantin Belousov , Li-Wen Hsu Subject: Re: FCP 20190401-ci_policy: CI policy In-reply-to: <20190830015805.GA16894@troutmask.apl.washington.edu> References: <20190829114057.GZ71821@kib.kiev.ua> <28934eb780342605090bf365ac3a2e0d522256f5.camel@freebsd.org> <201908300135.x7U1Zfh7067241@slippy.cwsent.com> <20190830015805.GA16894@troutmask.apl.washington.edu> Comments: In-reply-to Steve Kargl message dated "Thu, 29 Aug 2019 18:58:05 -0700." Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Date: Thu, 29 Aug 2019 19:02:27 -0700 X-CMAE-Envelope: MS4wfGKDwjGnXx+/Juz+HJ9qJ3XckpyZciCBnS1eUmv9SKKHNiBizNQMDwzoNALcWHDxEiNiGaZx65ZpdluDq3U0HsW2kqRXBLsAV+eHWHhZ2VCp5EU/IBje NI9bzyxBh9wiCCS/vvMwbJLAH0CloE/H55WxphH2eixWcY2MGkf9weutAhnvbNlA0RhtW6r5N0KxUhc4apdPcg+GN5d/jKbOkV6UfhNmgc/JLP/XFGK/MoxN 3TKIUjeRP/GwQA5fAh9y+lHVGqyT1eLZ8yGvcmgCmo7EfsVsnXbetDHKtfFCLWvPUthMLGi79hkS1SfkmKiIEqWNK/2oe/8sy5E+mIScxTrysQTqaFzFrGFB 77FrHttO/oWPpxTUtJCd/HpWNuG53Q== X-Rspamd-Queue-Id: 46KN5y54r1z4WSX X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; spf=none (mx1.freebsd.org: domain of cy.schubert@cschubert.com has no SPF policy when checking 64.59.134.12) smtp.mailfrom=cy.schubert@cschubert.com X-Spamd-Result: default: False [-4.99 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; HAS_REPLYTO(0.00)[Cy.Schubert@cschubert.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; REPLYTO_EQ_FROM(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.95)[-0.951,0]; RCVD_IN_DNSWL_NONE(0.00)[12.134.59.64.list.dnswl.org : 127.0.5.0]; RCPT_COUNT_SEVEN(0.00)[9]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6327, ipnet:64.59.128.0/20, country:CA]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-2.44)[ip: (-6.55), ipnet: 64.59.128.0/20(-3.13), asn: 6327(-2.43), country: CA(-0.09)]; RECEIVED_SPAMHAUS_PBL(0.00)[17.125.67.70.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.11] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 02:02:32 -0000 In message <20190830015805.GA16894@troutmask.apl.washington.edu>, Steve Kargl w rites: > On Thu, Aug 29, 2019 at 06:35:41PM -0700, Cy Schubert wrote: > > > > What about weekends? I notice commits tend to slow down during weekends. I > > suspect people who work on FreeBSD at $COMPANY for a living treat it as a > > job, which it is. However this should be considered. > > > > A committer, who works for a $COMPANY and has no intention of > looking at the mailing lists or CI reports until Monday, should > not commit on Firday. What ever happen to common sense? I've worked with and do work with people like that. -- Cheers, Cy Schubert FreeBSD UNIX: Web: http://www.FreeBSD.org The need of the many outweighs the greed of the few. From owner-freebsd-fcp@freebsd.org Fri Aug 30 06:55:48 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AA918C4349 for ; Fri, 30 Aug 2019 06:55:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46KVcN1rdHz3G0t for ; Fri, 30 Aug 2019 06:55:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 3F1B9C4344; Fri, 30 Aug 2019 06:55:48 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3ECFDC4342; Fri, 30 Aug 2019 06:55:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46KVcM5ZFpz3G0q; Fri, 30 Aug 2019 06:55:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x7U6tYGl041508 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 30 Aug 2019 09:55:37 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x7U6tYGl041508 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x7U6tY3t041507; Fri, 30 Aug 2019 09:55:34 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 30 Aug 2019 09:55:34 +0300 From: Konstantin Belousov To: Ed Maste Cc: Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190830065534.GC71821@kib.kiev.ua> References: <20190829114057.GZ71821@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46KVcM5ZFpz3G0q X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.90 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; NEURAL_HAM_SHORT(-0.91)[-0.906,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 06:55:48 -0000 On Thu, Aug 29, 2019 at 07:35:57PM -0400, Ed Maste wrote: > On Thu, 29 Aug 2019 at 07:41, Konstantin Belousov wrote: > > > > More, I know that tests are of very low quality, which means that > > brokeness of the tests is not an indicator of anything until root cause > > is identified. > > "Low quality" needs clarification here. I can think of many attributes > of a test that might lead someone to claim tests are low quality: > > - The test result is not consistent (e.g., a "flaky test") > - The test does not actually test what it claims to This. In fact, for code base like OS which provides an API/ABI to the large corpus of user code, it does not matter what exactly test claims to test. We are interested in any behaviour change. But very unfortunate and deeply engraved wart in many tests I saw is that they depend on unspecified or non-guaranteed API details to claim success. So detected behaviour changes often appears to be irrelevant. When I was (forced to) look into test failures, it was 50 vs. 50 % of test bugs vs. some legitimately catched issues. > - The test does as it claims, but there is no value in the result > - Test coverage overall is insufficient (i.e., not an issue with a > specific test) > - The test has excessive requirements (run time, memory usage, etc.) > - The test is difficult to maintain This is too. My main complain is that to debug a test case, I must strip all atf* to be able to examine it under a debugger. From owner-freebsd-fcp@freebsd.org Fri Aug 30 07:25:05 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 63702C50BF for ; Fri, 30 Aug 2019 07:25:05 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46KWG91142z3HPL for ; Fri, 30 Aug 2019 07:25:05 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 208BEC50BD; Fri, 30 Aug 2019 07:25:05 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1EEFAC50BC; Fri, 30 Aug 2019 07:25:05 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46KWG8602Sz3HPK; Fri, 30 Aug 2019 07:25:04 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x7U7Ou0a047434 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 30 Aug 2019 10:24:59 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x7U7Ou0a047434 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x7U7OtPu047433; Fri, 30 Aug 2019 10:24:55 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 30 Aug 2019 10:24:55 +0300 From: Konstantin Belousov To: Ed Maste Cc: Warner Losh , FreeBSD Hackers , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190830072455.GD71821@kib.kiev.ua> References: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46KWG8602Sz3HPK X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.92 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.92)[-0.920,0] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 07:25:05 -0000 On Thu, Aug 29, 2019 at 07:44:59PM -0400, Ed Maste wrote: > >> This sounds more like a problem with the tooling than an argument > >> against reverting though. > > > > We live in a subversion universe for the moment, so you have to view it through that lens. > > Fair enough, right now the policy needs to accommodate the reality of > the tools we're using today. > > Perhaps it's a failure of imagination on my part but I have trouble > seeing how a revert would lead to losing work - could you give an > example? If you have a committed change that is the precursor for later series, and this series is not yet committed. Then rebase or merge of the master with the base change reverted causes much trouble. > > > Sometimes yes, sometimes no. Even with git svn, there is a cost associated with it. The level of effort is not zero. Especially when one pushes several interrelated changes at once. If the first of these caused an issue on gcc, say, often the cost is too high to revert the whole chain. It's a lot easier to put in a fix and move on. > > The level of effort imposed on other users while the tree is broken is > not zero, either. Certainly if it's possible to commit a fix and move > forward that's the approach favoured by community norms. > > > It's a fair example for why a simpleminded approach will create more friction than the current system. And there is a need for caution in expanding the logic beyond all but the most recent changes... > > The point of the FCP is to facilitate the revert while the change is > (among the) most recent, precisely so that additional changes don't > build on it. Also, I have to admit publically, that my changes were blamed by CI so many times, when they had nothing to do with the breakage. I forwarded such mails to ci@ regularly, then stopped doing that. Also, I believe that imposing requirements on committers to not offend CI is not fair if committers cannot _easily_ verify that the change does not. Give us a way to e.g. mail pgp-signed message to ci-check@ and get the reply 'ok/fail' in reasonable time, or a button on phab to get the same response. Even if I bother to recreate CI env locally (I do not), races are different on the CI machines. From owner-freebsd-fcp@freebsd.org Fri Aug 30 12:44:23 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 34B36CDF4E for ; Fri, 30 Aug 2019 12:44:23 +0000 (UTC) (envelope-from brooks@spindle.one-eyed-alien.net) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KfLZ6J5wz45XY for ; Fri, 30 Aug 2019 12:44:22 +0000 (UTC) (envelope-from brooks@spindle.one-eyed-alien.net) Received: by mailman.nyi.freebsd.org (Postfix) id D5EB5CDF4A; Fri, 30 Aug 2019 12:44:22 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D479ACDF49; Fri, 30 Aug 2019 12:44:22 +0000 (UTC) (envelope-from brooks@spindle.one-eyed-alien.net) Received: from spindle.one-eyed-alien.net (spindle.one-eyed-alien.net [199.48.129.229]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46KfLZ51Cpz45XX; Fri, 30 Aug 2019 12:44:22 +0000 (UTC) (envelope-from brooks@spindle.one-eyed-alien.net) Received: by spindle.one-eyed-alien.net (Postfix, from userid 3001) id 5F47E3C0199; Fri, 30 Aug 2019 12:44:16 +0000 (UTC) Date: Fri, 30 Aug 2019 12:44:16 +0000 From: Brooks Davis To: Ed Maste Cc: "Rodney W. Grimes" , FreeBSD Hackers , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190830124416.GC19377@spindle.one-eyed-alien.net> References: <201908291905.x7TJ5Bw8091371@gndrsh.dnsmgr.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="vOmOzSkFvhd7u8Ms" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.4 (2018-02-28) X-Rspamd-Queue-Id: 46KfLZ51Cpz45XX X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.96 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.96)[-0.959,0] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 12:44:23 -0000 --vOmOzSkFvhd7u8Ms Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Aug 29, 2019 at 08:05:06PM -0400, Ed Maste wrote: > On Thu, 29 Aug 2019 at 15:05, Rodney W. Grimes > wrote: > > > > Here in lies one of the fundemental problems, this view by some that > > a "revert commit is something bad, it is kind of punishment". That is > > not true. Reverts are GREAT things, they allow the tree to be returned > > to a known state, usually quicly. The original commit is STILL IN SVN, > > and a bad revert can guess what.. be reverted!. >=20 > Let me echo Rod here. I'm also very happy that this statement was made > by one of the original FreeBSD committers. >=20 > Reverting a change is not an insult, not a punishment, not something > bad - it's simply an acknowledgement that some aspect of the change > didn't meet expectations. We should be considerably more willing to revert changes that break things. I agree with Warner that it's worth pinging the developer in question, but if they don't respond in in a couple hours, I see no value in waiting further. At that point any loss of productivity is on them and they have already wasted considerable productivity for other committers. All that being said, the LLVM project's aggressive use of reverts is sometimes enormously costly to downstreams with large changes[0] and we should avoid swinging too far toward reverts. The issue comes about when a non-trivial patch conflicts with a local change. When such a patch is reverted that's another conflict and sometimes you end up making/unmaking the same fixes multiple times when a change causes problems in an environment that developer can't test directly. Some balance is required, but I think we should be considerably more willing to revert changes where a fix isn't immediately available. -- Brooks [0] On the CHERI project we're almost certainly the largest public downstream for LLVM to the point that they validated the git monorepo migration tools on our tree(s). --vOmOzSkFvhd7u8Ms Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQEcBAEBAgAGBQJdaRofAAoJEKzQXbSebgfA7xwH/im4SHdyaGqJtJLmCRu2/HF4 3NyTBv91b3pPBYaukl61xFmMwjyZBeQS9OnxozVhyw5fpGjKCHPNEQzoaUzjEMHm 2iL3ag57Dj0hyNuVgJWyPSauSjozLBOMSrOLJLzs/VWq1R/+325YMUJ6nRxnNewF YhnOQiErnlFalWUB2QQ3Gg73sSaXdagLTE+EOlCNuFSBQ+ead3A6mOURspGKSj/u 5LHi6Jr4mOVv9hg2Qgt+sN25WbgGW0o/2UDGY7iNDkEJk0CkzYXN3o53V6pabQHJ 9VfTT0V3sf4xf61Bq/Bp0jM0cjBt0FoduoiJab3e9oM+dRRnUf910rpHf3w5S1s= =9/Vl -----END PGP SIGNATURE----- --vOmOzSkFvhd7u8Ms-- From owner-freebsd-fcp@freebsd.org Fri Aug 30 16:19:55 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D8111D243F for ; Fri, 30 Aug 2019 16:19:55 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46Kl7H447yz4GVB for ; Fri, 30 Aug 2019 16:19:55 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: by mailman.nyi.freebsd.org (Postfix) id 89E7AD243C; Fri, 30 Aug 2019 16:19:55 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 89942D243B; Fri, 30 Aug 2019 16:19:55 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46Kl7G1Hmrz4GV8; Fri, 30 Aug 2019 16:19:53 +0000 (UTC) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: from gndrsh.dnsmgr.net (localhost [127.0.0.1]) by gndrsh.dnsmgr.net (8.13.3/8.13.3) with ESMTP id x7UGJnOn095329; Fri, 30 Aug 2019 09:19:49 -0700 (PDT) (envelope-from freebsd-rwg@gndrsh.dnsmgr.net) Received: (from freebsd-rwg@localhost) by gndrsh.dnsmgr.net (8.13.3/8.13.3/Submit) id x7UGJnUt095328; Fri, 30 Aug 2019 09:19:49 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201908301619.x7UGJnUt095328@gndrsh.dnsmgr.net> Subject: Re: FCP 20190401-ci_policy: CI policy In-Reply-To: <20190830015805.GA16894@troutmask.apl.washington.edu> To: sgk@troutmask.apl.washington.edu Date: Fri, 30 Aug 2019 09:19:49 -0700 (PDT) CC: Cy Schubert , Ian Lepore , FreeBSD Hackers , araujo@freebsd.org, fcp@freebsd.org, Konstantin Belousov , Li-Wen Hsu , Kristof Provost X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-Rspamd-Queue-Id: 46Kl7G1Hmrz4GV8 X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of freebsd-rwg@gndrsh.dnsmgr.net has no SPF policy when checking 69.59.192.140) smtp.mailfrom=freebsd-rwg@gndrsh.dnsmgr.net X-Spamd-Result: default: False [2.43 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.49)[0.485,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[dnsmgr.net]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.77)[0.767,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.23)[0.233,0]; RCPT_COUNT_SEVEN(0.00)[9]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:13868, ipnet:69.59.192.0/19, country:US]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.04)[ip: (0.15), ipnet: 69.59.192.0/19(0.07), asn: 13868(0.05), country: US(-0.05)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 16:19:55 -0000 > On Thu, Aug 29, 2019 at 06:35:41PM -0700, Cy Schubert wrote: > > > > What about weekends? I notice commits tend to slow down during weekends. I > > suspect people who work on FreeBSD at $COMPANY for a living treat it as a > > job, which it is. However this should be considered. > > > > A committer, who works for a $COMPANY and has no intention of > looking at the mailing lists or CI reports until Monday, should > not commit on Firday. What ever happen to common sense? Perhaps this speaks to another issue then, I know the community is a whole against rules, procedures, requirements, etc all, but I'll again stick my head across the block and say: Should it be added to the committers guide that a committer is expected to be "responsive" to commit especially, and project in general emails for 48 hours following any commit? Would that not solve some of this? I do know from first hand interactions that several project members foo@freebsd.org email is basically /dev/nulled into a mail box that gets cleaned out on some random schedule. That technically goes against the requirement that if your a committer your suppose to be reading commit mail but I can agree that the volume of commit mail has become an overwhelming problem. One I do wish we had a better answer to than just ignore it. Regards, -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-fcp@freebsd.org Fri Aug 30 16:25:27 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E6130D2722 for ; Fri, 30 Aug 2019 16:25:27 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46KlFg4jmdz4GtY for ; Fri, 30 Aug 2019 16:25:27 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 9FFDAD2720; Fri, 30 Aug 2019 16:25:27 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9FA6ED271F; Fri, 30 Aug 2019 16:25:27 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pl1-x629.google.com (mail-pl1-x629.google.com [IPv6:2607:f8b0:4864:20::629]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46KlFf4Rxvz4GtX; Fri, 30 Aug 2019 16:25:26 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by mail-pl1-x629.google.com with SMTP id o3so3563526plb.13; Fri, 30 Aug 2019 09:25:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=qYmEuKmi7qTb6I3uIiGtbUztIC0f+a/EYr6bJdfUz1w=; b=YwcqkjhFPMFasBJ74QpHadWVIBvCy0GMMmLfVHu/jUlCq7zdwso4js3ge4ePv6iSmi k3WD/08DgzQp+X7FvNxQBmIOyQwxgSrPwuNjC8UEtxjvOLn1kCCvuB7x9+HFnqzODtj9 mVdCVRL9WdTsWu2ipFgOGpJDE6SWOPpZQKHTkFsJeEoNQosiMH+Ju8ZjsAp5OtTkYBSd 3Dd4VWsF+W/LTpaE1dkU8/830DACrT30fY7MSqEujqVMUiVJRA4kE7+XXawc7dLO5awB n/IvcobtmlSUY0GyCseJacjJxg86/z+aInqGm/fCUYU+gxk41EIQI4iivdwVwunEg7ah BucA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=qYmEuKmi7qTb6I3uIiGtbUztIC0f+a/EYr6bJdfUz1w=; b=tVIClHc8A2zNmCCExNsMdYeUGREtw6XbW8CWBITA4nQsL9YxQEc75CueK49JHu8hfA 1yB+jCc39sysPubTLtRzBZVIjBHmU7QO9I7ZElvTRj0Tp1aKpLoqYqQGIpukBQlOQGcZ BIDpi2JZASKI2xIe36jpn5jeXxwUb70L/dyMMuMCdELjNbO7GCf/VvTlQ1JU65CrRw3N JaXZL7sLd0VTFK1JjVhcNUnYf3hoOZ+okykqIlQKAf29GuVToKjzFDgwRN4leUDJM0IN 5pnWqnlTZOcBajliQajeW8tVAixjRT7OTXAXMMVhNxSozrEzpTJ4WQpFDdaXIgGCWcgd SOZA== X-Gm-Message-State: APjAAAWH8dXRvgNxkAfGfueBI9X7a+NUc3UFMI7LDV24nTM+gdsJUfeR uiWIN/o6Yi2NOv9G49f3/rTUb7SX X-Google-Smtp-Source: APXvYqz6tZcn1sJ82mD87PDMxRpzEjXbQnmWf7e2Mp2b8Jx6KcTB4cCnFshavJiFz92bhh6Ca1I8Eg== X-Received: by 2002:a17:902:fe0f:: with SMTP id g15mr16295103plj.2.1567182324414; Fri, 30 Aug 2019 09:25:24 -0700 (PDT) Received: from ?IPv6:2607:fb90:b23a:cc00:d046:1756:4288:91f6? ([2607:fb90:b23a:cc00:d046:1756:4288:91f6]) by smtp.gmail.com with ESMTPSA id a5sm5274066pjs.31.2019.08.30.09.25.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 30 Aug 2019 09:25:23 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (1.0) Subject: Re: FCP 20190401-ci_policy: CI policy From: Enji Cooper X-Mailer: iPhone Mail (16G77) In-Reply-To: Date: Fri, 30 Aug 2019 09:25:22 -0700 Cc: fcp@freebsd.org, FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <339B7A20-F88D-4F60-B133-612189663272@gmail.com> References: To: Li-Wen Hsu X-Rspamd-Queue-Id: 46KlFf4Rxvz4GtX X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=YwcqkjhF; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of yaneurabeya@gmail.com designates 2607:f8b0:4864:20::629 as permitted sender) smtp.mailfrom=yaneurabeya@gmail.com X-Spamd-Result: default: False [-3.49 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.99)[-0.989,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-9.17), ipnet: 2607:f8b0::/32(-2.84), asn: 15169(-2.32), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE_FREEMAIL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[9.2.6.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 16:25:28 -0000 > On Aug 27, 2019, at 21:29, Li-Wen Hsu wrote: >=20 > It seems I was doing wrong that just changed the content of this FCP > to "feedback", but did not send to the right mailing lists. >=20 > So I would like to make an announcement that the FCP > 20190401-ci_policy "CI policy": >=20 > https://github.com/freebsd/fcp/blob/master/fcp-20190401-ci_policy.md >=20 > is officially in "feedback" state to hopefully receive more comments > and suggestions, then we can move on for the next FCP state. First off, thank you Li-Wen and Kristof for spearheading this proposal; it=E2= =80=99s a very contentious topic with a lot of strong emotions associated wi= th it. As the person who has integrated a number of tests and helped manage them fo= r a few years (along with some of the care and feeding associated with them)= , this task is non-trivial. In particular when issues that I filed in bugzil= la are fixed quickly and linger in the tree for some time, impacting a lot o= f folks who might rely on build and test suite stability. The issue, as I see it, from a CI/release perspective that the new policy at= tempts to define a notion of =E2=80=9Cstable=E2=80=9D, in terms of both test= s and other code; right now, stability is sort of defined on a honor system b= asis with the FreeBSD test suite as a litmus test of sorts to convey a sense= of stability. =3D=3D=3D=3D=3D=3D One thing that I don=E2=80=99t see in the proposal is the health of the =E2=80= =9Cmake tinderbox=E2=80=9D target in a CI world (this is a gap in our curren= t CI process). Another thing that I don=E2=80=99t see in the proposal is about the health o= f head vs stable and how it relates to MFCs. I see a lot more issues occur o= n stable branches go unfixed for some time, in part because some fixes or en= hancements haven=E2=80=99t been MFCed. Part of the problem I see these days i= s a bit of a human/resource problem: if developers can=E2=80=99t test their c= hanges easily, they don=E2=80=99t MFC them. This issue has caused me to do a fair amount of triage in the past when back= porting changes, in order to discover potentially missing puzzle pieces in o= rder to make my tests and code work. =3D=3D=3D=3D=3D=3D The big issues, as I see it based on the discussions that has taken place in= the thread, is around revert timing and etiquette, and dealing with unrelia= ble tests. First off, revert timing and etiquette: while I see the FCP as an initial fr= amework, I am a bit concerned with the heavy handed ness of =E2=80=9Cwhat co= nstitutes needing reversion=E2=80=9D: should this be done after N consistent= failures in a certain period (be they build or test)? Furthermore, why is a= human involved in making this decision (apart from maybe a technical soluti= on via automation not being available yet)? Second off, unreliable tests: * Unreliable tests need to be qualified not based on a single run, but a pat= tern of runs. The way that this worked at Facebook is, if a test failed, it would attempt t= o rerun it multiple times (10 in total IIRC). If the test was consistently f= ailing on a build, the test would be automatically disabled, and all committ= ers in a revision range would be nagged as part of disabling those tests. Th= is generally works because of siloization of Facebook components, but is a m= uch harder problem to solve with FreeBSD because it is a complete OS distribution and sometimes small seemingly disconnected changes can cause= a lot of grief. So what to do? I suggest expanding the executors and running individuals suites instead of t= he whole batch of tests. While it wouldn=E2=80=99t fix everything and would b= e an expensive thing to do with our current test infrastructure, it would al= low folks to better pinpoint issues and be able to get some level of coverag= e, as opposed to throwing all of test execution out, like a baby with the ba= th water. How do we get there? - Expand the CI executor pool. - Provide a tool or process with which we can define test suites. - Make spinning up executors faster: with virtual machines this is typically= done by using Big Iron infrastructure clusters (e.g., ESXi clusters) and so= mething like thin provisioning where one could start from a common image/sna= pshot, instead of taking a hit copying around images. Linux can do this with= btrfs; we can do this with ZFS with per VM datasets, snapshotting, etc. While this only gets part of the way to a potential solution, it is a good w= ay to begin solving the isolation/execution problem. * A number of tests that existed in the tree have varying quality/reliabilit= y; I agree that system level tests (of which the pf tests are one of many) a= re less reliable than unit/API functional tests. This is the nature of the b= east of testing. The core issue I see with the test suite as it stands, is that it mixes inte= gration/system level tests (less deterministic) with functional/unit tests (= generally more deterministic). Using test mock frameworks would be a good technical solution to making syst= em tests into functional/unit tests (googlemock and unittest.mock are two of= many good tools I know of in this area), but we need a way to run both case= s. I can see now where some of the concern over labeling test types was a conce= rn when I first started this work (des@/phk@ aired this concern). Part of the technical/procedural solution to allowing commingling of tests i= s to go back and label the tests appropriately. I=E2=80=99ll send out an FCP= for this sometime in the next week or two. =3D=3D=3D=3D=3D=3D Taking a step back, as others have brought up, we=E2=80=99re currently hinde= red by tooling: we are applying a DVCS (git, hg) based technique (CI) to sub= version and testing changes after they=E2=80=99ve hit head, instead of befor= e they hit head. While phabricator can partially solve this by testing upfront (we don=E2=80=99= t enforce this; I=E2=80=99ve made my concerns with this not being a requirem= ent well-known in the past), the solution is limited by bandwidth for testin= g, i.e., testing is an all or nothing exercise right now and building multip= le toolchains/architectures takes a considerable amount of time. We could le= verage cloud/distributed solutions for this (Cirrus CI, Travis if the integr= ation existed), but this would require using github or teaching a tool how t= o make the appropriate REST api calls to run the tests and query the status (= in progress, pass, fail, etc). Applying labels and filtering on test suites will get us partway to a final s= olution from a test perspective, but a lot of work needs to be done with pha= bricator, etc. We also need to have build failures with tier 1 architectures with GENERIC b= e a commit blocking operation. Full stop. =3D=3D=3D=3D=3D=3D While some of the thoughts I put down aren=E2=80=99t complete solutions, I h= ave subproposals that should be done/things that could be worked on before i= mplementing the proposed CI policy. Some of the things I brought up above=20= While I can=E2=80=99t work on it now, December break is coming up, and with i= t I=E2=80=99ll have more time to work on projects like this. I=E2=80=99ll pu= t down some TODO items so I can look at tackling them during the break. Thank you, -Enji= From owner-freebsd-fcp@freebsd.org Fri Aug 30 17:42:33 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 06782D4678 for ; Fri, 30 Aug 2019 17:42:33 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46Kmyc5Kwrz4MLJ for ; Fri, 30 Aug 2019 17:42:32 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.nyi.freebsd.org (Postfix) id B6CF6D4677; Fri, 30 Aug 2019 17:42:32 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B6535D4676 for ; Fri, 30 Aug 2019 17:42:32 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46Kmyb4Ks3z4MKn for ; Fri, 30 Aug 2019 17:42:31 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x731.google.com with SMTP id g17so6849742qkk.8 for ; Fri, 30 Aug 2019 10:42:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=mEDmYyGD+aYCJIMJYKGfdtezYOt6DpNDj2A2wQiDr08=; b=VUzHJ6gChxj7MjPGyoI5/keny6o1rjNfuJHVGRmcrGI0hYpSo4b23PWanSGJHUDYhR /IpvuKffI5ofRjssqynombCtRpQanlYw5kT3tWUny5XnlQySBBy7rI+4CisHmVPnouN0 J51YyROOu5TuulfIBZ350CJGWTROspnalzGsKnluCavCkWS17UQDX1dnzv3V8I4I3Tm2 VVm3UYx7KnhEEcc8e+l9aqOax3G9xNWdyEd4yUVxqfEBPukCY1DnQ4GYLTUo3AiPpauG EjAMoXooXgiiANRSb4wLoY6/IiEt7sCx/7IEP4y2mnuYEgleDkBObIg0NiNdC7qRQZjy Hq5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=mEDmYyGD+aYCJIMJYKGfdtezYOt6DpNDj2A2wQiDr08=; b=A2KH18y7VsA8WuXE48y4B715idaHMcKAsyvImc7ag9/r4B+zeLp/TXgPVZCi48ybmZ wqjmbAf3v5L53/l+lJsOA/YhvVShkP/TS2j9WQEQYHBF9hXETaDISuk3GCG11HxE3K0H JK2kptSRzzIB5DazYQ42hTzuwlE865kYLlad0NfXUKWdk2txG25M1628wAH/CnkWywRe rK2J/OdfCQ7buwJ8GClrLxnR7+AOMkFgu/B+s1ltRKdwsZ5s9udBDHTjfPKcfvjUzVec +ub9mIdbMAjRUWJncECC+BEtcQMr9CV/02NiYYXeTvGPKXkHzPWLMYx0e4ayKzK4Z7zA brZw== X-Gm-Message-State: APjAAAVOlnX3fwOUixAZj+ziCWeeMIwcHTIr7DqfuwdXo/1D+tJ+GutA jQ96A6H/DQFzSy4FFnAGJL9vTZB0DnfVbggEnu/Opg== X-Google-Smtp-Source: APXvYqx2MNSmSeQi3YG5eOV6OO7gtnz8xJKpvOafPnEie0JxjyLF+yTqI3YbtFyr3vqifFE5iFvwr+tVqTyprh1YGOY= X-Received: by 2002:a37:4804:: with SMTP id v4mr17147516qka.60.1567186950326; Fri, 30 Aug 2019 10:42:30 -0700 (PDT) MIME-Version: 1.0 References: <339B7A20-F88D-4F60-B133-612189663272@gmail.com> In-Reply-To: <339B7A20-F88D-4F60-B133-612189663272@gmail.com> From: Warner Losh Date: Fri, 30 Aug 2019 11:42:19 -0600 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Enji Cooper Cc: Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org X-Rspamd-Queue-Id: 46Kmyb4Ks3z4MKn X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=VUzHJ6gC; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::731) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-5.91 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[fcp@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; RCVD_IN_DNSWL_NONE(0.00)[1.3.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.91)[ip: (-9.34), ipnet: 2607:f8b0::/32(-2.84), asn: 15169(-2.32), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 17:42:33 -0000 On Fri, Aug 30, 2019 at 10:25 AM Enji Cooper wrote: > > Taking a step back, as others have brought up, we=E2=80=99re currently hi= ndered by > tooling: we are applying a DVCS (git, hg) based technique (CI) to > subversion and testing changes after they=E2=80=99ve hit head, instead of= before > they hit head. > > While phabricator can partially solve this by testing upfront (we don=E2= =80=99t > enforce this; I=E2=80=99ve made my concerns with this not being a require= ment > well-known in the past), the solution is limited by bandwidth for testing= , > i.e., testing is an all or nothing exercise right now and building multip= le > toolchains/architectures takes a considerable amount of time. We could > leverage cloud/distributed solutions for this (Cirrus CI, Travis if the > integration existed), but this would require using github or teaching a > tool how to make the appropriate REST api calls to run the tests and quer= y > the status (in progress, pass, fail, etc). > > Applying labels and filtering on test suites will get us partway to a > final solution from a test perspective, but a lot of work needs to be don= e > with phabricator, etc. > > We also need to have build failures with tier 1 architectures with GENERI= C > be a commit blocking operation. Full stop. > Until we have a DCVS, this is a non-starter. It's too hard with svn. Let's table it until we have the migration to git complete. Also FreeBSD is huge, having to wait for a full build on all Tier-1 platforms also is a non-starter. It takes too long, and there's too many ways to build FreeBSD. Having a sanity check incremental build may be OK, but there's a number of changes that break the incremental -DNO_CLEAN build that are none-the-less fine for a complete rebuild. There's a ton of details to get right here to make it not an absolute nightmare for developers to get patches in and slow the velocity of changes to a crawl. Since we know nothing of our future git overlords, it's premature to even start this discussion because so many things dovetail with that effort we won't get beyond the basics (which people generally agree in principle on, but have concerns about the details that will be filibustered to death in the absence of a concrete git system it will add-on to). tl;dr: Love the concept, the devil is in the details to ensure we don't stifle momentum. Warner From owner-freebsd-fcp@freebsd.org Sat Aug 31 01:55:01 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AB201DF9B1 for ; Sat, 31 Aug 2019 01:55:01 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46Kzts33N6z3LR8 for ; Sat, 31 Aug 2019 01:55:01 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 633BBDF9AF; Sat, 31 Aug 2019 01:55:01 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 62E5BDF9AE; Sat, 31 Aug 2019 01:55:01 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46Kztr2MW6z3LR7; Sat, 31 Aug 2019 01:55:00 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f45.google.com with SMTP id x4so17745216iog.13; Fri, 30 Aug 2019 18:55:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZIfZggj9toIo5ZvUWoDtnDPGgIY3F/IW8VfjLo7w8Gk=; b=GCDU0An6agiz9uw1S1atc+Wli4Ct/X0pfjieenZUD/eenyjSDhsYi1qu9m12caXDmJ AcAZtMLu6xApae4iP0Kt4f3tbpws5erzrHLlmiafY5IN9DDR9n+swqkS7IRRzEc0lv0S pZBJt0TmePhTSrWuNRcGU5aJgnJKI8Dd1iBD9xGsOgxnjOUIGpE8bte8ZpMXuu+j+Dk/ 4WXhcnQxPvgU4F5WaFc11rVHKWLKHQ8IVqX6qzaX8PlvBrkj822iomrUA5YkfelfNyDg GWTM4IR5Ckaq+GgU+eC34ZWQFzOPq6cKhq/4vjUjxKfmH99oOpw2Y5Zf4BxkQp7+fpqq akLA== X-Gm-Message-State: APjAAAVDvd6oXC0kN/jbpEt+C9meEzHantPIWlv8mZr37EqNBwfZEuLl 5Vj20drrVpgYbENMJvibFa8w2H+9Qcx6cJWjmWMcfw== X-Google-Smtp-Source: APXvYqw/OIRsCk3ey0qml76Ht6ZFPpqKJcRrzpsDIHUUj32U3B+etIvX/eJpRxDfFnOFSZD5UC+zwS4tOoGKcmmdvrE= X-Received: by 2002:a5e:881a:: with SMTP id l26mr1460058ioj.185.1567216498988; Fri, 30 Aug 2019 18:54:58 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <20190830065534.GC71821@kib.kiev.ua> In-Reply-To: <20190830065534.GC71821@kib.kiev.ua> From: Ed Maste Date: Fri, 30 Aug 2019 21:54:41 -0400 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Konstantin Belousov Cc: Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46Kztr2MW6z3LR7 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.45 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-5.27 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; RCVD_IN_DNSWL_NONE(0.00)[45.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.28)[ip: (-5.67), ipnet: 209.85.128.0/17(-3.34), asn: 15169(-2.31), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 01:55:01 -0000 On Fri, 30 Aug 2019 at 02:56, Konstantin Belousov wrote: > > When I was (forced to) look into test failures, it was 50 vs. 50 % > of test bugs vs. some legitimately catched issues. Certainly if 50% of reported failures are actually test problems that's much too high. But independent of that, this still suggests the tests were responsible for reporting a good number of issues in advance of developers or end users. > > - The test is difficult to maintain > This is too. My main complain is that to debug a test case, I must strip > all atf* to be able to examine it under a debugger. Yes, this is my biggest complaint about our current test setup. But this impacts the tests' friendliness to developers, but not their efficacy in reporting regressions. From owner-freebsd-fcp@freebsd.org Sat Aug 31 08:55:16 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4C6A8CB22D for ; Sat, 31 Aug 2019 08:55:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46L9Cm01H2z4GbS for ; Sat, 31 Aug 2019 08:55:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 00623CB22B; Sat, 31 Aug 2019 08:55:16 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 00158CB22A; Sat, 31 Aug 2019 08:55:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46L9Cl5Jyvz4GbR; Sat, 31 Aug 2019 08:55:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x7V8t02i005839 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 31 Aug 2019 11:55:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x7V8t02i005839 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x7V8t0rf005838; Sat, 31 Aug 2019 11:55:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 31 Aug 2019 11:55:00 +0300 From: Konstantin Belousov To: Ed Maste Cc: Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190831085500.GF71821@kib.kiev.ua> References: <20190829114057.GZ71821@kib.kiev.ua> <20190830065534.GC71821@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46L9Cl5Jyvz4GbR X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.90 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.91)[-0.906,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 08:55:16 -0000 On Fri, Aug 30, 2019 at 09:54:41PM -0400, Ed Maste wrote: > On Fri, 30 Aug 2019 at 02:56, Konstantin Belousov wrote: > > > > When I was (forced to) look into test failures, it was 50 vs. 50 % > > of test bugs vs. some legitimately catched issues. > > Certainly if 50% of reported failures are actually test problems > that's much too high. 50% of what I looked at. The sample size was around 10. > But independent of that, this still suggests the > tests were responsible for reporting a good number of issues in > advance of developers or end users. > > > > - The test is difficult to maintain > > This is too. My main complain is that to debug a test case, I must strip > > all atf* to be able to examine it under a debugger. > > Yes, this is my biggest complaint about our current test setup. But > this impacts the tests' friendliness to developers, but not their > efficacy in reporting regressions. From owner-freebsd-fcp@freebsd.org Sat Aug 31 14:54:27 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 57B3ED3D50 for ; Sat, 31 Aug 2019 14:54:27 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46LKBC0HWMz4YlW for ; Sat, 31 Aug 2019 14:54:27 +0000 (UTC) (envelope-from sjg@juniper.net) Received: by mailman.nyi.freebsd.org (Postfix) id 07DBED3D4E; Sat, 31 Aug 2019 14:54:27 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0788ED3D4D; Sat, 31 Aug 2019 14:54:27 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from mx0a-00273201.pphosted.com (mx0a-00273201.pphosted.com [208.84.65.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.pphosted.com", Issuer "Thawte RSA CA 2018" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46LKBB5rXTz4YlV; Sat, 31 Aug 2019 14:54:26 +0000 (UTC) (envelope-from sjg@juniper.net) Received: from pps.filterd (m0108158.ppops.net [127.0.0.1]) by mx0a-00273201.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id x7VEsO0V018357; Sat, 31 Aug 2019 07:54:24 -0700 Received: from nam02-bl2-obe.outbound.protection.outlook.com (mail-bl2nam02lp2051.outbound.protection.outlook.com [104.47.38.51]) by mx0a-00273201.pphosted.com with ESMTP id 2uqmkq8dvg-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sat, 31 Aug 2019 07:54:24 -0700 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Do0TjvybFu5WsE+DMB8AHNsYtFYiASK+sToCaFvFTGr1snY9WqgpmKOT1jX0L2DFW7L+7tkjZxR8k2F83VrG1W+Kp24nnV128oqzYeW81K8H6LSawLCdf8CbzCP8sy19mwyPH3RBAojJVrSUrOcFMnC4YcdPY1IwkXxgnVGvshTEFi5IvOqHgSEhq1u4dMoTG333rONOuJaZWJbSxLHyyarVvXxKL6lP409w/OYEPyk/8M16Eb96Nn+eS7tacYPn6FLVil5tMYnFFiKFYk9ylfBVO2726qsRVtl1NE+XdQjHERCmLKZsv+QBk52cY3tt0UZ//BBQhJW+/hX/EW3v9A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JOI47K+2zg0zrn1pD9EAZcRd+qqbszg8o1IKtPLoWXY=; b=SxE4qQzGgZnRvKc1TWYGBylfLTmq4NeGPty38pfDF07BC+VP+eDOXiInBucnqFF/2bODIPDQ9G/uRSSsqDDC+e8Sn/asfchErGUiJ10WiC2YPjaNTxqpivz5/ZsJVd8We+LoYnrh0W4q30DILETDhuOndktHpW8otxlUvfuNOeb6GWHSMj/GZQiyKddbsyN2GilZaUb43Qd5urfe4iQ8lEJXMWT4TeykHOwTz39n4VxQa453IBYxrh7fvoiPoW8+wEGWmU0dkCcedU1j61mPrf4bG5W9Ua9ZYOgztQMQDVn1LiGRI12MVfk1hqij/xdX5MMg2d/7iBoXQK1RPAuM/w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=softfail (sender ip is 66.129.239.13) smtp.rcpttodomain=gmail.com smtp.mailfrom=juniper.net; dmarc=fail (p=reject sp=reject pct=100) action=oreject header.from=juniper.net; dkim=none (message not signed); arc=none Received: from BN3PR05CA0028.namprd05.prod.outlook.com (2603:10b6:400::38) by BL0PR05MB4692.namprd05.prod.outlook.com (2603:10b6:208:29::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2241.4; Sat, 31 Aug 2019 14:54:21 +0000 Received: from DM3NAM05FT013.eop-nam05.prod.protection.outlook.com (2a01:111:f400:7e51::203) by BN3PR05CA0028.outlook.office365.com (2603:10b6:400::38) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2241.7 via Frontend Transport; Sat, 31 Aug 2019 14:54:21 +0000 Received-SPF: SoftFail (protection.outlook.com: domain of transitioning juniper.net discourages use of 66.129.239.13 as permitted sender) Received: from P-EXFEND-EQX-02.jnpr.net (66.129.239.13) by DM3NAM05FT013.mail.protection.outlook.com (10.152.98.122) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.2241.7 via Frontend Transport; Sat, 31 Aug 2019 14:54:19 +0000 Received: from P-EXBEND-EQX-01.jnpr.net (10.104.8.52) by P-EXFEND-EQX-02.jnpr.net (10.104.8.55) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Sat, 31 Aug 2019 07:54:19 -0700 Received: from P-EXBEND-EQX-02.jnpr.net (10.104.8.53) by P-EXBEND-EQX-01.jnpr.net (10.104.8.52) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Sat, 31 Aug 2019 07:54:18 -0700 Received: from p-mailhub01.juniper.net (10.104.20.6) by P-EXBEND-EQX-02.jnpr.net (10.104.8.53) with Microsoft SMTP Server (TLS) id 15.0.1367.3 via Frontend Transport; Sat, 31 Aug 2019 07:54:18 -0700 Received: from kaos.jnpr.net (kaos.jnpr.net [172.23.50.162]) by p-mailhub01.juniper.net (8.14.4/8.11.3) with ESMTP id x7VEsIiH018044; Sat, 31 Aug 2019 07:54:18 -0700 (envelope-from sjg@juniper.net) Received: by kaos.jnpr.net (Postfix, from userid 1377) id 5CA113F79D; Sat, 31 Aug 2019 07:54:18 -0700 (PDT) Received: from kaos.jnpr.net (localhost [127.0.0.1]) by kaos.jnpr.net (Postfix) with ESMTP id 5AE593F79C; Sat, 31 Aug 2019 07:54:18 -0700 (PDT) To: Ed Maste CC: Konstantin Belousov , FreeBSD Hackers , Li-Wen Hsu , , Subject: Re: FCP 20190401-ci_policy: CI policy In-Reply-To: References: <20190829114057.GZ71821@kib.kiev.ua> <20190830065534.GC71821@kib.kiev.ua> Comments: In-reply-to: Ed Maste message dated "Fri, 30 Aug 2019 21:54:41 -0400." From: "Simon J. Gerraty" X-Mailer: MH-E 8.6+git; nmh 1.7.1; GNU Emacs 26.1 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <70179.1567263258.1@kaos.jnpr.net> Date: Sat, 31 Aug 2019 07:54:18 -0700 Message-ID: <73111.1567263258@kaos.jnpr.net> X-EXCLAIMER-MD-CONFIG: e3cb0ff2-54e7-4646-8a04-0dae4ac7b136 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:66.129.239.13; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10019020)(4636009)(376002)(136003)(396003)(346002)(39860400002)(2980300002)(189003)(199004)(7696005)(81156014)(8676002)(336012)(117636001)(23726003)(50226002)(53936002)(50466002)(97756001)(76506006)(70586007)(55016002)(9686003)(229853002)(446003)(11346002)(126002)(6246003)(7126003)(8936002)(476003)(6916009)(2906002)(305945005)(4326008)(6266002)(478600001)(486006)(186003)(356004)(26005)(86362001)(81166006)(70206006)(76176011)(4744005)(54906003)(47776003)(5660300002)(97876018)(46406003)(16586007)(316002)(53416004)(107886003); DIR:OUT; SFP:1102; SCL:1; SRVR:BL0PR05MB4692; H:P-EXFEND-EQX-02.jnpr.net; FPR:; SPF:SoftFail; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: f3fe614f-ff4a-4e92-3156-08d72e231aa8 X-Microsoft-Antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600166)(711020)(4605104)(4710121)(4711137)(1401327)(4618075)(2017052603328); SRVR:BL0PR05MB4692; X-MS-TrafficTypeDiagnostic: BL0PR05MB4692: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-Forefront-PRVS: 014617085B X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam-Message-Info: fIj1RjtuLpmbCmXCFtzuC/d+H8Zlr2oFxNLQ6aNHHwL8KWdDbfh2ew13713fQ2wox71TuqnBbSZPFAmGVbKM7RmkDakbge6ShXTSNwXWgRwF+OQ6XprbT8gwXrLAXRJz3S4tAj93Nls+3y4RHn20Vtgxzt0uK6Nq9cWx7wGdpj27i8HsAPRAR0aIfchYyazhlb4+w70y4ILoUgEfVP3+L3IJdUFpE35uxrWxmQgv6Fk8N8/aB7AMMtmjG9/IBhCSzz/Pql/LVQ6+UHYp7GXqLw4QWJklq2RpIQ4uo6beMiCLc0+niCs6TTdoJrr+UAFUZe9RFuMWkVpw/1fjNIUvxkisFwnM3j648oA9/UaVcU+LhKmpNRkwkc22AK2QnFpvUtP1NgMCQP1tff7GOslRcoe9rjoB3zPSI1mkQz2fqbE= X-OriginatorOrg: juniper.net X-MS-Exchange-CrossTenant-OriginalArrivalTime: 31 Aug 2019 14:54:19.9765 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: f3fe614f-ff4a-4e92-3156-08d72e231aa8 X-MS-Exchange-CrossTenant-Id: bea78b3c-4cdb-4130-854a-1d193232e5f4 X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=bea78b3c-4cdb-4130-854a-1d193232e5f4; Ip=[66.129.239.13]; Helo=[P-EXFEND-EQX-02.jnpr.net] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR05MB4692 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.70,1.0.8 definitions=2019-08-31_05:2019-08-29,2019-08-31 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_spam_notspam policy=outbound_spam score=0 mlxlogscore=650 spamscore=0 mlxscore=0 phishscore=0 lowpriorityscore=0 priorityscore=1501 suspectscore=0 bulkscore=0 malwarescore=0 impostorscore=0 clxscore=1011 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-1906280000 definitions=main-1908310173 X-Rspamd-Queue-Id: 46LKBB5rXTz4YlV X-Spamd-Bar: ------ X-Spamd-Result: default: False [-6.95 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[]; NEURAL_HAM_SHORT(-0.95)[-0.953,0] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 14:54:27 -0000 Ed Maste wrote: > > > - The test is difficult to maintain > > This is too. My main complain is that to debug a test case, I must strip > > all atf* to be able to examine it under a debugger. IMO this is a bug in the state of ATF in FreeBSD. The ability to run atf tests on the host in-line with the build is one of the reasons I picked ATF for Junos - but has been discarded in FreeBSD. With atf tests runnable on build host in situ they are not hard to debug there too. I recently added a test suite for my package system (for Junos) and had to use the bmake framework, because FreeBSD made it impossible to use ATF/Kyua > > Yes, this is my biggest complaint about our current test setup. But > this impacts the tests' friendliness to developers, but not their > efficacy in reporting regressions. From owner-freebsd-fcp@freebsd.org Sat Aug 31 17:01:15 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2A0F2D6B3A for ; Sat, 31 Aug 2019 17:01:15 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46LN0V3kKnz3C6y for ; Sat, 31 Aug 2019 17:01:14 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.nyi.freebsd.org (Postfix) id 71518D6B35; Sat, 31 Aug 2019 17:01:14 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 70726D6B34 for ; Sat, 31 Aug 2019 17:01:14 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x741.google.com (mail-qk1-x741.google.com [IPv6:2607:f8b0:4864:20::741]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46LN0T4955z3C6p for ; Sat, 31 Aug 2019 17:01:13 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x741.google.com with SMTP id q7so941389qkn.13 for ; Sat, 31 Aug 2019 10:01:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:cc; bh=rKnj8r2kfeoLJG8aZSTuCjvBewda1V1tBZTE9fnPvic=; b=xG5FOXzFuaLy5p+SQCqCwaCIt4LCcYJStvaV5iDBfjEVImnBdscxa6EdVQpHrpdU29 HVAIiRTKRGbKzhVhYQxs+23DEuxS65dSXsFf3Wrz38bnPk7aZaEb5PggIgw8j9JAngb5 +ZaMAAwlPgRfbSV/SP0lQL75uXD9mTbnZsALKiMUIvG8oY2d/ZDMu859ieem935ZUmeI CKhTiJMA91j95+e5KNbe5yg+zIz9ERmqpr8/O16YlsW1qAczEjl9GM1RyW5H6+GaJcvv 84QjDTZkOec2VcuBpFWGQ2hZRHdMCiHT7T+JciU37Bm3g0/Q+pjAhHXcsXd5kII9am6y 4UWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:cc; bh=rKnj8r2kfeoLJG8aZSTuCjvBewda1V1tBZTE9fnPvic=; b=ZvJm3eH1gjor0DrkfL+MCf2tQk1b68JTttgOrBibxDQDrUK4vqRNaWjZ5rpqnibm2C gdd2VHuinhnM3Gef+ZBydckoj9CrAfZwyTY6Qvq4/krztSV2+XuLXOj2pXtsX4XRoyYB qvfrU1dCoVXGS7Yf7Y+GpKHEx897u0U57R2IK+z1HzqZQTLi5WTPf/IeggWMIbCeslIO a6zkSwm9gow8nLd2/4UQIRccYfcnWFpNVBJB1fGjjFNBrKBeXbxkMX/uU6Ya6jQWhNTd F9dNKRKylJ4j150/t8WLdPu9WraY3RWRBcrPNCQxgNOF3hvEQQYO+gdsOBGMwn74ihwX Argg== X-Gm-Message-State: APjAAAXKp5bDn9o+niWlt1VV3ekxcaMGIidmrB+D1GjaHhP7NGZGw71e 0L/G/niZE0Di2MfohbAHT0C0V5jU1ByVst8JZw542g== X-Received: by 2002:a37:30f:: with SMTP id 15mt11586979qkd.240.1567270871089; Sat, 31 Aug 2019 10:01:11 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <20190830065534.GC71821@kib.kiev.ua> <73111.1567263258@kaos.jnpr.net> In-Reply-To: <73111.1567263258@kaos.jnpr.net> From: Warner Losh Date: Sat, 31 Aug 2019 11:00:59 -0600 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy Cc: FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org X-Rspamd-Queue-Id: 46LN0T4955z3C6p X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=xG5FOXzF; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::741) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-1.45 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.977,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.977,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[fcp@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-0.95)[-0.949,0]; RCVD_IN_DNSWL_NONE(0.00)[1.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MISSING_TO(2.00)[]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-0.55)[ip: (2.47), ipnet: 2607:f8b0::/32(-2.83), asn: 15169(-2.31), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 17:01:15 -0000 After this weeks discussions, I got to thinking about where we were on this topic. I thought I'd spend a few minutes summarizing my impressions so that we don't get bogged down in the details of disagreement, but rather start from where we agree. Here's my take on what that is, but I write this down as the basis for discussion, not to lay down the law. I think there's consensus on the following points: (1) We want CI (2) We want developers to be responsive to breakage in CI (3) At the moment, we have some sub-optimal tools, and when we change those we should evolve the process. (4) Build breakage is something that should be fixed very quickly. (5) People should reach out to the original developer who committed the change when there's breakage ASAP (6) If the original developer can't timely fix the problem, it's OK to either back out the change, or perhaps commit a tiny fix if the original was huge and the fix needed to fix the build is tiny. (7) Breaking tests is a problem, but our tests need to evolve because we have too high a rate of false positives. (8) There's a sliding scale of urgency (Tier 1 build breakage needs to be fixed in a couple of hours, Tier 2 can go a day, Tier 3 can go longer but shouldn't linger). (9) The urgency for a Test regression also is a sliding scale, but given the current state of the tests we need to apply judgement on revert vs fix test vs disable test. (10) Reverts shouldn't be feared, but there's a cost to reverting automatically and there's some desire for developers to have a chance to be in the loop (11) We need to work on the social aspect of reverts to destigmatize them. We might quibble a bit over timelines for the different pieces, but here's what I've noticed the approximate timelines are today (there are exceptions, and I don't have hard data, just my sense from watching the tree, but I think they form a reasonable basis absent better data): * Build system breakage usually is fixed within an hour (eg, I screwed up a Makefile or bsd.foo.mk file somehow). * x86 build breakages are usually fixed in an hour or two (longer over the weekend) * arm and arm64 build breakages are usually fixed within 4-8 hours * Other build breakages are usually fixed within a day or two. * out of tree compiler breakages are fixed on the order of a week. * I have little data on test breakage, but it's my sense most issues are resolved in less than a week. We've been quite reluctant to do reverts to date. They happen, but have usually been initiated by the committer. Li-Wen and others would like to change that to setting firm timelines; start to reset the social aspect of reverts and document the social norms with an eye towards improving things, either within the SVN framework, or the coming git framework. Finally, there's a number of ways we can do this, not limited to the FCP in question. However, nobody has stepped up to drive that initiative. It would be great to have someone sign up to drive revisions of the various internal developer guides to modernize their content to reflect how norms have evolved since the were written (including this topic and the related topic of responsiveness to @freebsd.org email and others...). Warner From owner-freebsd-fcp@freebsd.org Sat Aug 31 18:55:47 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D0B42D90B1 for ; Sat, 31 Aug 2019 18:55:47 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 46LQXg4DwJz3HfK for ; Sat, 31 Aug 2019 18:55:47 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 91652D90AF; Sat, 31 Aug 2019 18:55:47 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 91142D90AE; Sat, 31 Aug 2019 18:55:47 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-io1-f45.google.com (mail-io1-f45.google.com [209.85.166.45]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46LQXf4txGz3HfJ; Sat, 31 Aug 2019 18:55:46 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by mail-io1-f45.google.com with SMTP id b10so21038519ioj.2; Sat, 31 Aug 2019 11:55:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=3CYs+2ssJzGmNRUbeoLJuS8bh7nRG2o7LCTK6RFQZyM=; b=bOhwMQMEFGsPaEfzGRvMKNtLktHheveFQ76N792lDMksoBV9IpWLooakEBFOUrgQb7 aYjYEdRGzqSI4X0b67kdF+gcFONUNZTB6qfXUzYhJXySakjVJLmSZlqqzRrV6Uzv0cL0 jx+J2QmGEPhkW2PfevSkcTKzNfTGvCJ1hO48VuA6LJ+YGRU1ddM7er0kzvXBjtBx9A8B SAaEiuZ+4ZyGPTgwQ/D8UVzJQjhoJtZzcL4F6iyM+BvW1HvoYDYvRz3hNpD40TmQg8Gx CcnGGsyUZtzkpOM0ouPG6s7IRxc5V/qH1VnA+KIszA9XoP+SGMRO+4Wx80apgIYxMnH7 dJWg== X-Gm-Message-State: APjAAAUDWK5lvMHveCO6PmjH+w8LVZ1bGwXcA4ikSJzHIM0ZgvbLVHGY LwZmEgRVaY8vuRCNHchqu30FoVT3wxoHiPmvZHm/fw== X-Google-Smtp-Source: APXvYqyymnvBEk/A2JM1h+b1SaNbL9HrtJWnBgeyXWCQNI8ov1Xngnd7lCV2WQYBBhB1nuDWwbnFbyU1qrb2yXVLcqM= X-Received: by 2002:a05:6638:73d:: with SMTP id j29mr19655895jad.21.1567277744950; Sat, 31 Aug 2019 11:55:44 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <20190830065534.GC71821@kib.kiev.ua> <73111.1567263258@kaos.jnpr.net> In-Reply-To: From: Ed Maste Date: Sat, 31 Aug 2019 14:55:25 -0400 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Warner Losh Cc: FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 46LQXf4txGz3HfJ X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of carpeddiem@gmail.com designates 209.85.166.45 as permitted sender) smtp.mailfrom=carpeddiem@gmail.com X-Spamd-Result: default: False [-5.25 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.99)[-0.991,0]; RCVD_IN_DNSWL_NONE(0.00)[45.166.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.26)[ip: (-5.59), ipnet: 209.85.128.0/17(-3.34), asn: 15169(-2.31), country: US(-0.05)]; FORGED_SENDER(0.30)[emaste@freebsd.org,carpeddiem@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[emaste@freebsd.org,carpeddiem@gmail.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 18:55:47 -0000 On Sat, 31 Aug 2019 at 13:01, Warner Losh wrote: > > (7) Breaking tests is a problem, but our tests need to evolve because we > have too high a rate of false positives. I'm not sure there's consensus on false positives. In particular, Li-Wen has spent an incredible amount of time over more than six months tracking down test failures and submitting PRs. Flaky tests and false positives have generally been addressed by now (by submitting a PR and disabling or xfailing the test). From owner-freebsd-fcp@freebsd.org Sat Aug 31 19:09:53 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 34B1CD95F0 for ; Sat, 31 Aug 2019 19:09:53 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46LQrw6h0Lz3JJb for ; Sat, 31 Aug 2019 19:09:52 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.nyi.freebsd.org (Postfix) id E54ABD95ED; Sat, 31 Aug 2019 19:09:52 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E50DAD95EC for ; Sat, 31 Aug 2019 19:09:52 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x734.google.com (mail-qk1-x734.google.com [IPv6:2607:f8b0:4864:20::734]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46LQrv6m0vz3JJX for ; Sat, 31 Aug 2019 19:09:51 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x734.google.com with SMTP id d23so9160090qko.3 for ; Sat, 31 Aug 2019 12:09:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=wuqIQ6VjJLqAmwFu3UeQjakHZH7aJZq7NTKf+mOEtTM=; b=l61dqkLO8zVFZ10+f7CDeWliXqe0iQLbGAnw6OnVhZOaMnHG47YBV1UnQ2Rw5fRPbB OeJFbH0FUR3ePvwjQd3tka9K3rO1r3c1iyzaEmyjhnYRErBT6wv65koPii8UdG9lLQyx 5Z+ZunLunX+lkH1MXVIUET25dDqEUhgcNrn+dPUldyr2EEVzfM4fYuTCB0jqeZc8oifu m28so9IIWE3okj0Rvj+/x6QhrLIS4X3RpVEfRyI6fB7i4QS3GNmK0Ub4REnnUGDE3uRC FGCggOz1c/DpD0+E8aGu+Ohu0+trHP3SPmhr/nfQIjexTf9oS+krBPYWCktekPMNUbR1 a3yA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=wuqIQ6VjJLqAmwFu3UeQjakHZH7aJZq7NTKf+mOEtTM=; b=CZDoeUA+EuvwftWjf7Zo9oERkj+TZ73nIKCA+xh8qnEC3KE8mEPLo8f/gYK2607xYb zRphHMxs32VT67Jcyjwol9CSr0kOIQosLHsaKmdqJEmb+tCbWgA1L88Ee5aAxBjzESA9 iCRadG33PVUZB8mRkl1JKHFIqbO9WcTXWRrJzneEYTxXSrGUrpIHNTiEu2vr+XdYiYaF irdaZZFpVVmsq5a07dWFxduwE1e99WRMJqhpkkhfVqZzjg9/QzG2pDz7u6DDuXwZTonL U65VC2U/qCcsi1eRBHSR1QebqYBQQ/ISToXqLih2Qr4GcFvsuP/L5GFyD2zYZGXYd6Xd MEdg== X-Gm-Message-State: APjAAAUFTpInpiYziqTssfTFWz1gZmRhi8GuwzzzAC/ZSb8Dk7DvzAyC Qaob4izZOMfK8rcOgZ9VzhBWG2W16M60bOs69V+3UA== X-Google-Smtp-Source: APXvYqy0i3b2EuSlq92DA4ISuvDNoUY1nkICcebV1jzGWzapPJyw3Sm6pVFG67D/oSb/ECBBCLiJRNBkDEDBfRVGtvI= X-Received: by 2002:a37:30f:: with SMTP id 15mr12747587qkd.240.1567278590149; Sat, 31 Aug 2019 12:09:50 -0700 (PDT) MIME-Version: 1.0 References: <20190829114057.GZ71821@kib.kiev.ua> <20190830065534.GC71821@kib.kiev.ua> <73111.1567263258@kaos.jnpr.net> In-Reply-To: From: Warner Losh Date: Sat, 31 Aug 2019 13:09:38 -0600 Message-ID: Subject: Re: FCP 20190401-ci_policy: CI policy To: Ed Maste Cc: FreeBSD Hackers , Li-Wen Hsu , fcp@freebsd.org X-Rspamd-Queue-Id: 46LQrv6m0vz3JJX X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=l61dqkLO; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::734) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-5.92 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[fcp@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-1.00)[-0.996,0]; RCVD_IN_DNSWL_NONE(0.00)[4.3.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-2.92)[ip: (-9.41), ipnet: 2607:f8b0::/32(-2.83), asn: 15169(-2.31), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 19:09:53 -0000 On Sat, Aug 31, 2019, 12:55 PM Ed Maste wrote: > On Sat, 31 Aug 2019 at 13:01, Warner Losh wrote: > > > > (7) Breaking tests is a problem, but our tests need to evolve because we > > have too high a rate of false positives. > > I'm not sure there's consensus on false positives. In particular, > Li-Wen has spent an incredible amount of time over more than six > months tracking down test failures and submitting PRs. Flaky tests and > false positives have generally been addressed by now (by submitting a > PR and disabling or xfailing the test). > Fair enough. Do we have recent data that can distinguish between false positives or bad perception? Warner >