Date: Sat, 30 May 2015 17:52:22 -0700 From: Garrett Cooper <yaneurabeya@gmail.com> To: "freebsd-testing@freebsd.org" <freebsd-testing@FreeBSD.org> Cc: jenkins-admin@freebsd.org Subject: jenkins.FreeBSD.org test characterization/analysis tools Message-ID: <B2F5C350-E142-4F4F-9CF0-D7E5DD1CEACB@gmail.com>
next in thread | raw e-mail | index | archive | help
--Apple-Mail=_6B727744-058B-444D-9A42-FDFE184221F0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hi all, Jenkins provides some limited test failure analysis tools (the = history graph [1], the test trend point/bar chart [2], some limited = support for determining whether or not a testcase has failed by looking = at the last X failures [3] are the only items that it seems we have = enabled in our Jenkins install), but characterizing non-consecutive test = failures is a bit difficult. I took the time to write up a Makefile and some python scripts = [4] which pull down all of the test results XML files from Jenkins, do = some basic analysis, then summarize the test failures. I=92ve included = an example below (which I=92m going to use in filing bugs for testcases = that have proven problematic with Jenkins). If this seems interesting, I=92ll look at formalizing the API, = adding in =93pass=94 results, and doing a few other things (it=92s doing = similar things to what an internal database at $work is doing tracking = test results). Thanks! -NGie PS This information only goes back ~2 months today. Unfortunately all = test results prior to that date seemed to have gone missing :(. $ pwd /Users/ngie/scratch/tools/analyze-jenkins-failures $ bmake all Test =3D bin.expr/expr_test/overflow, Failed builds =3D 869 Test =3D bin.pax/legacy_test/main, Failed builds =3D 943 Test =3D bin.pkill/pgrep-j_test/main, Failed builds =3D 1066 Test =3D lib.libc.gen/sleep_test/kevent, Failed builds =3D 1054, 1058 Test =3D lib.libc.sys/poll_test/poll_3way, Failed builds =3D 948 Test =3D lib.libc.sys/sigtimedwait_test/sigtimedwait_NULL_timeout, = Failed builds =3D 1040, 1054-1056 Test =3D libexec.atf.atf-sh/atf_check_test/flush_stdout_on_timeout, = Failed builds =3D 937, 964, 1010-1041 Test =3D local.atf.atf-sh/atf_check_test/flush_stdout_on_timeout, Failed = builds =3D 990, 1023-1026, 1037 Test =3D local.atf.test-programs/expect_test/expect_death, Failed builds = =3D 1032 Test =3D local.kyua.testers/atf_inttest/test__cleanup__crash, Failed = builds =3D 1053 Test =3D sbin.devd/client_test/stream, Failed builds =3D 982, 1064 Test =3D sys.aio/aio_kqueue_test/main, Failed builds =3D 983-986 Test =3D sys.aio/aio_test/aio_fifo_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_file_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_md_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_pipe_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_pty_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_unix_socketpair_test, Failed builds =3D = 987-989 Test =3D sys.aio/lio_kqueue_test/main, Failed builds =3D 983-986 Test =3D = sys.kern/ptrace_test/ptrace__parent_sees_exit_after_unrelated_debugger, = Failed builds =3D 1061-1071 Test =3D sys.mqueue/mqueue_test/mqtest1, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest2, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest3, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest4, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest5, Failed builds =3D 983-989 Test =3D usr.bin.grep/grep_test/recurse, Failed builds =3D 943 Test =3D usr.sbin.pw/pw_userdel/rmuser_seperate_group, Failed builds =3D = 1045 1. = https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1071/testReport/history= 2. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/ 3. = https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1071/testReport/junit/s= ys.kern/ptrace_test/ptrace__parent_sees_exit_after_unrelated_debugger/ , = look for =93Failing for the last 11 builds=94. 4. = https://github.com/yaneurabeya/scratch/tree/master/tools/analyze-jenkins-f= ailures --Apple-Mail=_6B727744-058B-444D-9A42-FDFE184221F0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJValtGAAoJEMZr5QU6S73e+6MH/1WP40JMoXpSu1gvgXY0BiVg 8mmBFuqFeNNe/QmtAJhkvlJejhRsrNOs8OVAhyx7TbIumUiglQSjwKWvu0U9/Sg5 oENfuLbjJ+9xVUnyfgK/VkjpFSERGz4kAT2gzd4uW2NEPmUCiHlT6hD9e0k0EMQZ T3AyQjqObrv1KdKfNvpRpZEuH74YW2q7tuxVphHPwq8VpAKYL3SgpxBx3Kb/jZTv wuWY/qIiAanCoxp+iMd26w8cW8roQIm7gLwLo4qC4MBEwdlt6pzrHqDt9ljBQ4oJ T0q7AsQicA64m9hJfWzgLLGhit726uMB/h7HQ6U3KF9Kaf6AIt9l5rLUtV2biJY= =+RVm -----END PGP SIGNATURE----- --Apple-Mail=_6B727744-058B-444D-9A42-FDFE184221F0--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B2F5C350-E142-4F4F-9CF0-D7E5DD1CEACB>