From owner-freebsd-testing@FreeBSD.ORG Sun May 31 00:52:23 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8FEE1462; Sun, 31 May 2015 00:52:23 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5488F18A7; Sun, 31 May 2015 00:52:23 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by iebgx4 with SMTP id gx4so86897537ieb.0; Sat, 30 May 2015 17:52:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:subject:date:message-id:cc:to:mime-version; bh=F+P2xACkj4+mBFWJqoRX39QpFHaThEJwL5tB4BnynCI=; b=vjaRXZNSevxPN6atcZMxo83gCmcg0AgEWiykQjM1ojm1/Qy6i9Cok9qjy2O+IBiA8N Mohe24M0YmgoRHkS4BMdu6I+slL5SdPdinTlrx4uNatz71X3i4ypwXupK1ZoZ6NxnzBn kdl+2thEJc1Q/Va326rEPz7vObjp7n2WRZo31yo/n9AiOVidWHwUX4EVkothp5FXVipp X7Y1sUE0uKUrxalmoH8b4C/n2AhdPj8o7D7DXUShdr4wAgWRcmapQ0Xai5FB3Q1IYuPT GQ6fS6sh6imUUiU2xgtMx+8gjc+mfG+UqSwgwDdNQHNx2DS75E0yGb0Ya0z/SFAU7QAX MPrQ== X-Received: by 10.50.143.38 with SMTP id sb6mr5265683igb.44.1433033542583; Sat, 30 May 2015 17:52:22 -0700 (PDT) Received: from ?IPv6:2601:8:ab80:7d6:f56e:8e67:9b06:a? ([2601:8:ab80:7d6:f56e:8e67:9b06:a]) by mx.google.com with ESMTPSA id f126sm7359887ioe.21.2015.05.30.17.52.21 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 30 May 2015 17:52:21 -0700 (PDT) From: Garrett Cooper X-Pgp-Agent: GPGMail 2.5b6 Content-Type: multipart/signed; boundary="Apple-Mail=_6B727744-058B-444D-9A42-FDFE184221F0"; protocol="application/pgp-signature"; micalg=pgp-sha512 Subject: jenkins.FreeBSD.org test characterization/analysis tools Date: Sat, 30 May 2015 17:52:22 -0700 Message-Id: Cc: jenkins-admin@freebsd.org To: "freebsd-testing@freebsd.org" Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 May 2015 00:52:23 -0000 --Apple-Mail=_6B727744-058B-444D-9A42-FDFE184221F0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hi all, Jenkins provides some limited test failure analysis tools (the = history graph [1], the test trend point/bar chart [2], some limited = support for determining whether or not a testcase has failed by looking = at the last X failures [3] are the only items that it seems we have = enabled in our Jenkins install), but characterizing non-consecutive test = failures is a bit difficult. I took the time to write up a Makefile and some python scripts = [4] which pull down all of the test results XML files from Jenkins, do = some basic analysis, then summarize the test failures. I=92ve included = an example below (which I=92m going to use in filing bugs for testcases = that have proven problematic with Jenkins). If this seems interesting, I=92ll look at formalizing the API, = adding in =93pass=94 results, and doing a few other things (it=92s doing = similar things to what an internal database at $work is doing tracking = test results). Thanks! -NGie PS This information only goes back ~2 months today. Unfortunately all = test results prior to that date seemed to have gone missing :(. $ pwd /Users/ngie/scratch/tools/analyze-jenkins-failures $ bmake all Test =3D bin.expr/expr_test/overflow, Failed builds =3D 869 Test =3D bin.pax/legacy_test/main, Failed builds =3D 943 Test =3D bin.pkill/pgrep-j_test/main, Failed builds =3D 1066 Test =3D lib.libc.gen/sleep_test/kevent, Failed builds =3D 1054, 1058 Test =3D lib.libc.sys/poll_test/poll_3way, Failed builds =3D 948 Test =3D lib.libc.sys/sigtimedwait_test/sigtimedwait_NULL_timeout, = Failed builds =3D 1040, 1054-1056 Test =3D libexec.atf.atf-sh/atf_check_test/flush_stdout_on_timeout, = Failed builds =3D 937, 964, 1010-1041 Test =3D local.atf.atf-sh/atf_check_test/flush_stdout_on_timeout, Failed = builds =3D 990, 1023-1026, 1037 Test =3D local.atf.test-programs/expect_test/expect_death, Failed builds = =3D 1032 Test =3D local.kyua.testers/atf_inttest/test__cleanup__crash, Failed = builds =3D 1053 Test =3D sbin.devd/client_test/stream, Failed builds =3D 982, 1064 Test =3D sys.aio/aio_kqueue_test/main, Failed builds =3D 983-986 Test =3D sys.aio/aio_test/aio_fifo_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_file_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_md_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_pipe_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_pty_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_unix_socketpair_test, Failed builds =3D = 987-989 Test =3D sys.aio/lio_kqueue_test/main, Failed builds =3D 983-986 Test =3D = sys.kern/ptrace_test/ptrace__parent_sees_exit_after_unrelated_debugger, = Failed builds =3D 1061-1071 Test =3D sys.mqueue/mqueue_test/mqtest1, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest2, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest3, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest4, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest5, Failed builds =3D 983-989 Test =3D usr.bin.grep/grep_test/recurse, Failed builds =3D 943 Test =3D usr.sbin.pw/pw_userdel/rmuser_seperate_group, Failed builds =3D = 1045 1. = https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1071/testReport/history= 2. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/ 3. = https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1071/testReport/junit/s= ys.kern/ptrace_test/ptrace__parent_sees_exit_after_unrelated_debugger/ , = look for =93Failing for the last 11 builds=94. 4. = https://github.com/yaneurabeya/scratch/tree/master/tools/analyze-jenkins-f= ailures --Apple-Mail=_6B727744-058B-444D-9A42-FDFE184221F0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJValtGAAoJEMZr5QU6S73e+6MH/1WP40JMoXpSu1gvgXY0BiVg 8mmBFuqFeNNe/QmtAJhkvlJejhRsrNOs8OVAhyx7TbIumUiglQSjwKWvu0U9/Sg5 oENfuLbjJ+9xVUnyfgK/VkjpFSERGz4kAT2gzd4uW2NEPmUCiHlT6hD9e0k0EMQZ T3AyQjqObrv1KdKfNvpRpZEuH74YW2q7tuxVphHPwq8VpAKYL3SgpxBx3Kb/jZTv wuWY/qIiAanCoxp+iMd26w8cW8roQIm7gLwLo4qC4MBEwdlt6pzrHqDt9ljBQ4oJ T0q7AsQicA64m9hJfWzgLLGhit726uMB/h7HQ6U3KF9Kaf6AIt9l5rLUtV2biJY= =+RVm -----END PGP SIGNATURE----- --Apple-Mail=_6B727744-058B-444D-9A42-FDFE184221F0--