From owner-freebsd-hackers@freebsd.org Sat Aug 31 08:55:16 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 00158CB22A; Sat, 31 Aug 2019 08:55:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46L9Cl5Jyvz4GbR; Sat, 31 Aug 2019 08:55:15 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x7V8t02i005839 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 31 Aug 2019 11:55:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x7V8t02i005839 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x7V8t0rf005838; Sat, 31 Aug 2019 11:55:00 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 31 Aug 2019 11:55:00 +0300 From: Konstantin Belousov To: Ed Maste Cc: Li-Wen Hsu , FreeBSD Hackers , fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190831085500.GF71821@kib.kiev.ua> References: <20190829114057.GZ71821@kib.kiev.ua> <20190830065534.GC71821@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46L9Cl5Jyvz4GbR X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.90 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.91)[-0.906,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 31 Aug 2019 08:55:16 -0000 On Fri, Aug 30, 2019 at 09:54:41PM -0400, Ed Maste wrote: > On Fri, 30 Aug 2019 at 02:56, Konstantin Belousov wrote: > > > > When I was (forced to) look into test failures, it was 50 vs. 50 % > > of test bugs vs. some legitimately catched issues. > > Certainly if 50% of reported failures are actually test problems > that's much too high. 50% of what I looked at. The sample size was around 10. > But independent of that, this still suggests the > tests were responsible for reporting a good number of issues in > advance of developers or end users. > > > > - The test is difficult to maintain > > This is too. My main complain is that to debug a test case, I must strip > > all atf* to be able to examine it under a debugger. > > Yes, this is my biggest complaint about our current test setup. But > this impacts the tests' friendliness to developers, but not their > efficacy in reporting regressions.