Date: Wed, 1 May 2019 18:40:46 +0200 From: =?UTF-8?Q?Olivier_Cochard=2DLabb=C3=A9?= <olivier@freebsd.org> To: freebsd-testing@freebsd.org Subject: Non-stable tests Message-ID: <CA%2Bq%2BTcqJaYWkKeHBjKwWFcsF_Qp60MFWdT%2B4n-QBpU7njaremA@mail.gmail.com>
next in thread | raw e-mail | index | archive | help
Hi, On our CI (Jenkins running the full kyua test suite on a bhyve VM), some tests are sometimes failing without a valid cause. So I had to run a second time the kyua test suite to confirm it's a "real" regression, and the second time I didn't meet any problem. The last 2 tests that meet this problem are: - sbin.devd.client_test.seqpacket, failed with "Test case body timed out" - lib.libc.sys.getitimer_test.setitimer_old, failed with "setitimer(2) did not return remaining time, failed with" I've tried to rerun these tests a thousand times, but I didn't reach to reproduce the problem: they've succeeded each time after. So I wonder if there is a problem about "non stable" tests ? I've done a rapid search on ci.freebsd.org and I've found an example. In this series: https://ci.freebsd.org/job/FreeBSD-head-amd64-test/10391/ (r344839) https://ci.freebsd.org/job/FreeBSD-head-amd64-test/10392/ (r344840) https://ci.freebsd.org/job/FreeBSD-head-amd64-test/10393/ (r344843) => with job 10392, the test "lib.libarchive.functional_test.test_fuzz_zip" failed, but there is no change in the code regarding libarchive during these 3 commits revision period ( https://svnweb.freebsd.org/base/head/?view=log&pathrev=344843). So why it failed once here ? This is the same symptom I meet on my platform too. Regards, Olivier
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2Bq%2BTcqJaYWkKeHBjKwWFcsF_Qp60MFWdT%2B4n-QBpU7njaremA>