Date: Sat, 31 Aug 2019 11:55:00 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Ed Maste <emaste@freebsd.org> Cc: Li-Wen Hsu <lwhsu@freebsd.org>, FreeBSD Hackers <freebsd-hackers@freebsd.org>, fcp@freebsd.org Subject: Re: FCP 20190401-ci_policy: CI policy Message-ID: <20190831085500.GF71821@kib.kiev.ua> In-Reply-To: <CAPyFy2CXVMEXZx0u9b0p_CVaD_gu8Gyzvud3KwEDciq83a2_8A@mail.gmail.com> References: <CAKBkRUwKKPKwRvUs00ja0%2BG9vCBB1pKhv6zBS-F-hb=pqMzSxQ@mail.gmail.com> <20190829114057.GZ71821@kib.kiev.ua> <CAPyFy2BNrFfZ1PHaLesW%2Bu7YmbhC7mtiZ%2BBsYrqYE0J7KdagaA@mail.gmail.com> <20190830065534.GC71821@kib.kiev.ua> <CAPyFy2CXVMEXZx0u9b0p_CVaD_gu8Gyzvud3KwEDciq83a2_8A@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Aug 30, 2019 at 09:54:41PM -0400, Ed Maste wrote: > On Fri, 30 Aug 2019 at 02:56, Konstantin Belousov <kostikbel@gmail.com> wrote: > > > > When I was (forced to) look into test failures, it was 50 vs. 50 % > > of test bugs vs. some legitimately catched issues. > > Certainly if 50% of reported failures are actually test problems > that's much too high. 50% of what I looked at. The sample size was around 10. > But independent of that, this still suggests the > tests were responsible for reporting a good number of issues in > advance of developers or end users. > > > > - The test is difficult to maintain > > This is too. My main complain is that to debug a test case, I must strip > > all atf* to be able to examine it under a debugger. > > Yes, this is my biggest complaint about our current test setup. But > this impacts the tests' friendliness to developers, but not their > efficacy in reporting regressions.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190831085500.GF71821>