Date: Sat, 30 May 2015 17:52:22 -0700 From: Garrett Cooper <yaneurabeya@gmail.com> To: "freebsd-testing@freebsd.org" <freebsd-testing@FreeBSD.org> Cc: jenkins-admin@freebsd.org Subject: jenkins.FreeBSD.org test characterization/analysis tools Message-ID: <B2F5C350-E142-4F4F-9CF0-D7E5DD1CEACB@gmail.com>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Hi all, Jenkins provides some limited test failure analysis tools (the history graph [1], the test trend point/bar chart [2], some limited support for determining whether or not a testcase has failed by looking at the last X failures [3] are the only items that it seems we have enabled in our Jenkins install), but characterizing non-consecutive test failures is a bit difficult. I took the time to write up a Makefile and some python scripts [4] which pull down all of the test results XML files from Jenkins, do some basic analysis, then summarize the test failures. I’ve included an example below (which I’m going to use in filing bugs for testcases that have proven problematic with Jenkins). If this seems interesting, I’ll look at formalizing the API, adding in “pass” results, and doing a few other things (it’s doing similar things to what an internal database at $work is doing tracking test results). Thanks! -NGie PS This information only goes back ~2 months today. Unfortunately all test results prior to that date seemed to have gone missing :(. $ pwd /Users/ngie/scratch/tools/analyze-jenkins-failures $ bmake all Test = bin.expr/expr_test/overflow, Failed builds = 869 Test = bin.pax/legacy_test/main, Failed builds = 943 Test = bin.pkill/pgrep-j_test/main, Failed builds = 1066 Test = lib.libc.gen/sleep_test/kevent, Failed builds = 1054, 1058 Test = lib.libc.sys/poll_test/poll_3way, Failed builds = 948 Test = lib.libc.sys/sigtimedwait_test/sigtimedwait_NULL_timeout, Failed builds = 1040, 1054-1056 Test = libexec.atf.atf-sh/atf_check_test/flush_stdout_on_timeout, Failed builds = 937, 964, 1010-1041 Test = local.atf.atf-sh/atf_check_test/flush_stdout_on_timeout, Failed builds = 990, 1023-1026, 1037 Test = local.atf.test-programs/expect_test/expect_death, Failed builds = 1032 Test = local.kyua.testers/atf_inttest/test__cleanup__crash, Failed builds = 1053 Test = sbin.devd/client_test/stream, Failed builds = 982, 1064 Test = sys.aio/aio_kqueue_test/main, Failed builds = 983-986 Test = sys.aio/aio_test/aio_fifo_test, Failed builds = 987-989 Test = sys.aio/aio_test/aio_file_test, Failed builds = 987-989 Test = sys.aio/aio_test/aio_md_test, Failed builds = 987-989 Test = sys.aio/aio_test/aio_pipe_test, Failed builds = 987-989 Test = sys.aio/aio_test/aio_pty_test, Failed builds = 987-989 Test = sys.aio/aio_test/aio_unix_socketpair_test, Failed builds = 987-989 Test = sys.aio/lio_kqueue_test/main, Failed builds = 983-986 Test = sys.kern/ptrace_test/ptrace__parent_sees_exit_after_unrelated_debugger, Failed builds = 1061-1071 Test = sys.mqueue/mqueue_test/mqtest1, Failed builds = 983-989 Test = sys.mqueue/mqueue_test/mqtest2, Failed builds = 983-989 Test = sys.mqueue/mqueue_test/mqtest3, Failed builds = 983-989 Test = sys.mqueue/mqueue_test/mqtest4, Failed builds = 983-989 Test = sys.mqueue/mqueue_test/mqtest5, Failed builds = 983-989 Test = usr.bin.grep/grep_test/recurse, Failed builds = 943 Test = usr.sbin.pw/pw_userdel/rmuser_seperate_group, Failed builds = 1045 1. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1071/testReport/history 2. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/ 3. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1071/testReport/junit/sys.kern/ptrace_test/ptrace__parent_sees_exit_after_unrelated_debugger/ , look for “Failing for the last 11 builds”. 4. https://github.com/yaneurabeya/scratch/tree/master/tools/analyze-jenkins-failures [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJValtGAAoJEMZr5QU6S73e+6MH/1WP40JMoXpSu1gvgXY0BiVg 8mmBFuqFeNNe/QmtAJhkvlJejhRsrNOs8OVAhyx7TbIumUiglQSjwKWvu0U9/Sg5 oENfuLbjJ+9xVUnyfgK/VkjpFSERGz4kAT2gzd4uW2NEPmUCiHlT6hD9e0k0EMQZ T3AyQjqObrv1KdKfNvpRpZEuH74YW2q7tuxVphHPwq8VpAKYL3SgpxBx3Kb/jZTv wuWY/qIiAanCoxp+iMd26w8cW8roQIm7gLwLo4qC4MBEwdlt6pzrHqDt9ljBQ4oJ T0q7AsQicA64m9hJfWzgLLGhit726uMB/h7HQ6U3KF9Kaf6AIt9l5rLUtV2biJY= =+RVm -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B2F5C350-E142-4F4F-9CF0-D7E5DD1CEACB>