From owner-freebsd-fcp@freebsd.org Fri Aug 30 06:55:48 2019 Return-Path: Delivered-To: freebsd-fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AA918C4349 for ; Fri, 30 Aug 2019 06:55:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 46KVcN1rdHz3G0t for ; Fri, 30 Aug 2019 06:55:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.nyi.freebsd.org (Postfix) id 3F1B9C4344; Fri, 30 Aug 2019 06:55:48 +0000 (UTC) Delivered-To: fcp@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3ECFDC4342; Fri, 30 Aug 2019 06:55:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46KVcM5ZFpz3G0q; Fri, 30 Aug 2019 06:55:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x7U6tYGl041508 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 30 Aug 2019 09:55:37 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x7U6tYGl041508 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x7U6tY3t041507; Fri, 30 Aug 2019 09:55:34 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 30 Aug 2019 09:55:34 +0300 From: Konstantin Belousov To: Ed Maste Cc: Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190830065534.GC71821@kib.kiev.ua> References: <20190829114057.GZ71821@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46KVcM5ZFpz3G0q X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.90 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; NEURAL_HAM_SHORT(-0.91)[-0.906,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-fcp@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: FreeBSD Community Proposals List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Aug 2019 06:55:48 -0000 On Thu, Aug 29, 2019 at 07:35:57PM -0400, Ed Maste wrote: > On Thu, 29 Aug 2019 at 07:41, Konstantin Belousov wrote: > > > > More, I know that tests are of very low quality, which means that > > brokeness of the tests is not an indicator of anything until root cause > > is identified. > > "Low quality" needs clarification here. I can think of many attributes > of a test that might lead someone to claim tests are low quality: > > - The test result is not consistent (e.g., a "flaky test") > - The test does not actually test what it claims to This. In fact, for code base like OS which provides an API/ABI to the large corpus of user code, it does not matter what exactly test claims to test. We are interested in any behaviour change. But very unfortunate and deeply engraved wart in many tests I saw is that they depend on unspecified or non-guaranteed API details to claim success. So detected behaviour changes often appears to be irrelevant. When I was (forced to) look into test failures, it was 50 vs. 50 % of test bugs vs. some legitimately catched issues. > - The test does as it claims, but there is no value in the result > - Test coverage overall is insufficient (i.e., not an issue with a > specific test) > - The test has excessive requirements (run time, memory usage, etc.) > - The test is difficult to maintain This is too. My main complain is that to debug a test case, I must strip all atf* to be able to examine it under a debugger.