From owner-freebsd-testing@FreeBSD.ORG Sun May 31 00:52:23 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8FEE1462; Sun, 31 May 2015 00:52:23 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-ie0-x235.google.com (mail-ie0-x235.google.com [IPv6:2607:f8b0:4001:c03::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5488F18A7; Sun, 31 May 2015 00:52:23 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by iebgx4 with SMTP id gx4so86897537ieb.0; Sat, 30 May 2015 17:52:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:subject:date:message-id:cc:to:mime-version; bh=F+P2xACkj4+mBFWJqoRX39QpFHaThEJwL5tB4BnynCI=; b=vjaRXZNSevxPN6atcZMxo83gCmcg0AgEWiykQjM1ojm1/Qy6i9Cok9qjy2O+IBiA8N Mohe24M0YmgoRHkS4BMdu6I+slL5SdPdinTlrx4uNatz71X3i4ypwXupK1ZoZ6NxnzBn kdl+2thEJc1Q/Va326rEPz7vObjp7n2WRZo31yo/n9AiOVidWHwUX4EVkothp5FXVipp X7Y1sUE0uKUrxalmoH8b4C/n2AhdPj8o7D7DXUShdr4wAgWRcmapQ0Xai5FB3Q1IYuPT GQ6fS6sh6imUUiU2xgtMx+8gjc+mfG+UqSwgwDdNQHNx2DS75E0yGb0Ya0z/SFAU7QAX MPrQ== X-Received: by 10.50.143.38 with SMTP id sb6mr5265683igb.44.1433033542583; Sat, 30 May 2015 17:52:22 -0700 (PDT) Received: from ?IPv6:2601:8:ab80:7d6:f56e:8e67:9b06:a? ([2601:8:ab80:7d6:f56e:8e67:9b06:a]) by mx.google.com with ESMTPSA id f126sm7359887ioe.21.2015.05.30.17.52.21 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 30 May 2015 17:52:21 -0700 (PDT) From: Garrett Cooper X-Pgp-Agent: GPGMail 2.5b6 Content-Type: multipart/signed; boundary="Apple-Mail=_6B727744-058B-444D-9A42-FDFE184221F0"; protocol="application/pgp-signature"; micalg=pgp-sha512 Subject: jenkins.FreeBSD.org test characterization/analysis tools Date: Sat, 30 May 2015 17:52:22 -0700 Message-Id: Cc: jenkins-admin@freebsd.org To: "freebsd-testing@freebsd.org" Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 May 2015 00:52:23 -0000 --Apple-Mail=_6B727744-058B-444D-9A42-FDFE184221F0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hi all, Jenkins provides some limited test failure analysis tools (the = history graph [1], the test trend point/bar chart [2], some limited = support for determining whether or not a testcase has failed by looking = at the last X failures [3] are the only items that it seems we have = enabled in our Jenkins install), but characterizing non-consecutive test = failures is a bit difficult. I took the time to write up a Makefile and some python scripts = [4] which pull down all of the test results XML files from Jenkins, do = some basic analysis, then summarize the test failures. I=92ve included = an example below (which I=92m going to use in filing bugs for testcases = that have proven problematic with Jenkins). If this seems interesting, I=92ll look at formalizing the API, = adding in =93pass=94 results, and doing a few other things (it=92s doing = similar things to what an internal database at $work is doing tracking = test results). Thanks! -NGie PS This information only goes back ~2 months today. Unfortunately all = test results prior to that date seemed to have gone missing :(. $ pwd /Users/ngie/scratch/tools/analyze-jenkins-failures $ bmake all Test =3D bin.expr/expr_test/overflow, Failed builds =3D 869 Test =3D bin.pax/legacy_test/main, Failed builds =3D 943 Test =3D bin.pkill/pgrep-j_test/main, Failed builds =3D 1066 Test =3D lib.libc.gen/sleep_test/kevent, Failed builds =3D 1054, 1058 Test =3D lib.libc.sys/poll_test/poll_3way, Failed builds =3D 948 Test =3D lib.libc.sys/sigtimedwait_test/sigtimedwait_NULL_timeout, = Failed builds =3D 1040, 1054-1056 Test =3D libexec.atf.atf-sh/atf_check_test/flush_stdout_on_timeout, = Failed builds =3D 937, 964, 1010-1041 Test =3D local.atf.atf-sh/atf_check_test/flush_stdout_on_timeout, Failed = builds =3D 990, 1023-1026, 1037 Test =3D local.atf.test-programs/expect_test/expect_death, Failed builds = =3D 1032 Test =3D local.kyua.testers/atf_inttest/test__cleanup__crash, Failed = builds =3D 1053 Test =3D sbin.devd/client_test/stream, Failed builds =3D 982, 1064 Test =3D sys.aio/aio_kqueue_test/main, Failed builds =3D 983-986 Test =3D sys.aio/aio_test/aio_fifo_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_file_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_md_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_pipe_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_pty_test, Failed builds =3D 987-989 Test =3D sys.aio/aio_test/aio_unix_socketpair_test, Failed builds =3D = 987-989 Test =3D sys.aio/lio_kqueue_test/main, Failed builds =3D 983-986 Test =3D = sys.kern/ptrace_test/ptrace__parent_sees_exit_after_unrelated_debugger, = Failed builds =3D 1061-1071 Test =3D sys.mqueue/mqueue_test/mqtest1, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest2, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest3, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest4, Failed builds =3D 983-989 Test =3D sys.mqueue/mqueue_test/mqtest5, Failed builds =3D 983-989 Test =3D usr.bin.grep/grep_test/recurse, Failed builds =3D 943 Test =3D usr.sbin.pw/pw_userdel/rmuser_seperate_group, Failed builds =3D = 1045 1. = https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1071/testReport/history= 2. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/ 3. = https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests/1071/testReport/junit/s= ys.kern/ptrace_test/ptrace__parent_sees_exit_after_unrelated_debugger/ , = look for =93Failing for the last 11 builds=94. 4. = https://github.com/yaneurabeya/scratch/tree/master/tools/analyze-jenkins-f= ailures --Apple-Mail=_6B727744-058B-444D-9A42-FDFE184221F0 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJValtGAAoJEMZr5QU6S73e+6MH/1WP40JMoXpSu1gvgXY0BiVg 8mmBFuqFeNNe/QmtAJhkvlJejhRsrNOs8OVAhyx7TbIumUiglQSjwKWvu0U9/Sg5 oENfuLbjJ+9xVUnyfgK/VkjpFSERGz4kAT2gzd4uW2NEPmUCiHlT6hD9e0k0EMQZ T3AyQjqObrv1KdKfNvpRpZEuH74YW2q7tuxVphHPwq8VpAKYL3SgpxBx3Kb/jZTv wuWY/qIiAanCoxp+iMd26w8cW8roQIm7gLwLo4qC4MBEwdlt6pzrHqDt9ljBQ4oJ T0q7AsQicA64m9hJfWzgLLGhit726uMB/h7HQ6U3KF9Kaf6AIt9l5rLUtV2biJY= =+RVm -----END PGP SIGNATURE----- --Apple-Mail=_6B727744-058B-444D-9A42-FDFE184221F0-- From owner-freebsd-testing@FreeBSD.ORG Sun May 31 02:15:44 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 34FAAEE3; Sun, 31 May 2015 02:15:44 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mail-yk0-x22c.google.com (mail-yk0-x22c.google.com [IPv6:2607:f8b0:4002:c07::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E5EA819C4; Sun, 31 May 2015 02:15:43 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: by ykfl8 with SMTP id l8so33961354ykf.1; Sat, 30 May 2015 19:15:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=TOeZgbqXlJ+oueF0TDxEWfoDm4DelKpyEUu018e3AzU=; b=BC72Mg38PWMoqH2uxVbg+OWPOR7/kXzvZ+fkbLZp5SE0OdMCUCeG59V3fWziO/TjEI 55w/jYTi0CvtQetMyRJnf1zwRGLRHdKLtnjvwgs4P9gp+mH7L4KHTkAVZZcroitsdxqy gM9xh9HoG2ZD2YIgjrWzi058Y8LSkOfIHWSGmVIrCsWV5By2HDjOw56sDO2WUDWBo8Qs 6k6kcLuMEgpQwjeTZZLiTlRAWuxTv40LVc+2kAYBqksg61rP1+ZNVr3j4OuAPqGcZNpD T1Y0igIsjuuQCopVN/rTMzidk4DkhT7rAnshn0j/+RpKfFvxi6FlDbkcG4NA/zM9s0lm Zpwg== MIME-Version: 1.0 X-Received: by 10.236.2.137 with SMTP id 9mr16075316yhf.61.1433038543057; Sat, 30 May 2015 19:15:43 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.13.233.70 with HTTP; Sat, 30 May 2015 19:15:42 -0700 (PDT) In-Reply-To: <20150528164940.GC2499@kib.kiev.ua> References: <20150528164940.GC2499@kib.kiev.ua> Date: Sat, 30 May 2015 19:15:42 -0700 X-Google-Sender-Auth: U1jrnU4ehGMEJfvB3ir-t2CCGVg Message-ID: Subject: Re: Need help reducing compilation warnings in CURRENT From: Craig Rodrigues To: Konstantin Belousov Cc: freebsd-current Current , "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 May 2015 02:15:44 -0000 On Thu, May 28, 2015 at 9:49 AM, Konstantin Belousov wrote: > On Wed, May 27, 2015 at 11:30:45PM -0700, Craig Rodrigues wrote: > > I take some time to do a pass over mine code or code I am somewhat > knowledgable, to correct some 'assigned but not used variable' warnings. > There might be even one real bug in SU code uncovered, but I am not > sure yet. > > Please review at https://reviews.freebsd.org/D2665 . > I do not have an intention of splitting this into individual changes. > Thanks for committing these fixes. Jenkins detected this as fixing 42 warnings: https://jenkins.freebsd.org/job/FreeBSD_HEAD_amd64_gcc4.9/34/warnings17Result/ Every little bit helps! -- Craig From owner-freebsd-testing@FreeBSD.ORG Sun May 31 02:18:04 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F0D6614; Sun, 31 May 2015 02:18:03 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pa0-x235.google.com (mail-pa0-x235.google.com [IPv6:2607:f8b0:400e:c03::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BA20F19DA; Sun, 31 May 2015 02:18:03 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by padj3 with SMTP id j3so18767706pad.0; Sat, 30 May 2015 19:18:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to; bh=mKlpSQC6Zl2H2CEovHwB5Pn8KoP0r7WipoqrHrSYwCY=; b=LlF+a5l72kTNzG8oqWtP/p47pHSgQ9wruKZRwdxwHf/MlHsUNHft7STh+xYgfnAfRP BuomXX3FLkf0s++L9NzDIktsYOHUgXX/WBcnwJDAWE+bHRnv6pVOt+Pm7Ef0YfnZoVXI qzS9BSniGKNpbS/8gN8YppYD5ffMtKxFQGSaqnx4TBJokUt0vn6cPfKqpRG7CqwU/z/t HJc7TtXGgdiNijhq7XPSBg/R8MfPB/U1eqcyFEdSzaXmbJzIr4mMnhiH4m4uYkSTTTgD ie9+/mcppgEscHPiOsQbUmxoLyCw+ntxTgEpqDlDHgSVuXvwb2y2062tz1m5/OOH6Xg1 DOPw== X-Received: by 10.68.65.43 with SMTP id u11mr27822360pbs.41.1433038683267; Sat, 30 May 2015 19:18:03 -0700 (PDT) Received: from ?IPv6:2601:8:ab80:7d6:f56e:8e67:9b06:a? ([2601:8:ab80:7d6:f56e:8e67:9b06:a]) by mx.google.com with ESMTPSA id j9sm9984805pdm.53.2015.05.30.19.18.01 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 30 May 2015 19:18:02 -0700 (PDT) Subject: Re: Need help reducing compilation warnings in CURRENT Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Content-Type: multipart/signed; boundary="Apple-Mail=_BC2C76BD-34C9-41C7-93DC-73BD19D87A9F"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.5b6 From: Garrett Cooper In-Reply-To: Date: Sat, 30 May 2015 19:18:01 -0700 Cc: Konstantin Belousov , freebsd-current Current , "freebsd-testing@freebsd.org" Message-Id: <5A5C2989-7901-4241-8557-8F411E9DEFC3@gmail.com> References: <20150528164940.GC2499@kib.kiev.ua> To: Craig Rodrigues X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 May 2015 02:18:04 -0000 --Apple-Mail=_BC2C76BD-34C9-41C7-93DC-73BD19D87A9F Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On May 30, 2015, at 19:15, Craig Rodrigues wrote: > On Thu, May 28, 2015 at 9:49 AM, Konstantin Belousov = > wrote: >=20 >> On Wed, May 27, 2015 at 11:30:45PM -0700, Craig Rodrigues wrote: >>=20 >> I take some time to do a pass over mine code or code I am somewhat >> knowledgable, to correct some 'assigned but not used variable' = warnings. >> There might be even one real bug in SU code uncovered, but I am not >> sure yet. >>=20 >> Please review at https://reviews.freebsd.org/D2665 . >> I do not have an intention of splitting this into individual changes. >>=20 >=20 > Thanks for committing these fixes. > Jenkins detected this as fixing 42 warnings: >=20 > = https://jenkins.freebsd.org/job/FreeBSD_HEAD_amd64_gcc4.9/34/warnings17Res= ult/ >=20 > Every little bit helps! The -Wstrict-aliasing warnings are particularly useful, considering = clang still doesn=92t support this detection=85 Thanks :)! -NGie --Apple-Mail=_BC2C76BD-34C9-41C7-93DC-73BD19D87A9F Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJVam9bAAoJEMZr5QU6S73ego4IAJhzQgmoNh0ETqxpn5EgiGGF S1A3GyAYGWtIauSRQPFzJiDuCVfit5B8hCntDfIO6q42zz/+bEou/ezIL2Q9lbLa JMHNV5L8+VVSWQ4mN+tLPvZOB5TCSTwBc235kyLYMq8CaGbtiiFB5HrtNvcpFUOz 6tPUr00oenYPd0np5Su048l1sU3buQ5VnYFP3SrRhHpNen8kz9DcTwFoNK8OHbid sLcjBYLCPW5UKklTyRy5L5Q5p6fJ69BWEoj9KbXc180L9nEQBjYDGVGdRR0zpoZZ 5Yh5bFX68lDixvj2XOtr3tX4dF3bx81o/gRFvKHPe5q6Cq7LJOd1zLgdoLR15aQ= =Bywr -----END PGP SIGNATURE----- --Apple-Mail=_BC2C76BD-34C9-41C7-93DC-73BD19D87A9F-- From owner-freebsd-testing@FreeBSD.ORG Tue Jun 2 20:40:55 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 522629FA for ; Tue, 2 Jun 2015 20:40:55 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-ie0-x231.google.com (mail-ie0-x231.google.com [IPv6:2607:f8b0:4001:c03::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 25B4C1836 for ; Tue, 2 Jun 2015 20:40:55 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by ieczm2 with SMTP id zm2so142510909iec.1 for ; Tue, 02 Jun 2015 13:40:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to:content-type; bh=FALgWuMoBQHqxo0MQDO7x3e3UQdqNnKr8T/KbKJHBpU=; b=fP+WQ+CxXkfV/Inn9ORXAdqX1+7E65mXxZu2mmt/ZvXeQprK/XhDvK8s/+e6Uo3Jft L9tYYWgyyHgwriWnq2z3ycYm/6Wyzn/LJsTYPFba8bnyu/gzHLxx8Bmeae7S3/Y1Z034 nSDuJ0Kgs4IPbkqZCn/s3wgaddlDoc5Bsu0108PNCtSRgkLWlWaerreRJ1WJ5ZlDnaD8 SDB6Y6gGiZBwmCGDGB/RW9ErtzpObs5GM/ExDR1DEzM+YMow4mWz1irwqoSnuk4MtJmV CzSzW9+bKMuzQIqqZN0mXW6ydgDZnBh3RjdBKip/JMK702a0ZjIIFLTwHvbyVA1OraYd WHng== X-Received: by 10.42.176.8 with SMTP id bc8mr25831711icb.22.1433277654353; Tue, 02 Jun 2015 13:40:54 -0700 (PDT) MIME-Version: 1.0 Sender: carpeddiem@gmail.com Received: by 10.107.48.3 with HTTP; Tue, 2 Jun 2015 13:40:33 -0700 (PDT) From: Ed Maste Date: Tue, 2 Jun 2015 16:40:33 -0400 X-Google-Sender-Auth: PWjeTj7oE9hwBCHzLgGlRriJ30g Message-ID: Subject: Incorporating the Capsicum test suite To: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 20:40:55 -0000 I would like to incorporate the Capsicum test suite[1] into our test infrastructure and Jenkins. There are a small handful of minor issues to resolve, but it's very close to a state where it "just builds and passes." It uses googletest, which produces output of the form: [----------] 10 tests from Capmode [ RUN ] Capmode.AllowedIdentifierSyscallsForked [ OK ] Capmode.AllowedIdentifierSyscallsForked (11 ms) [ RUN ] Capmode.AllowedSchedSyscallsForked capmode.cc:289: Failure Expected: (0) <= (sched_rr_get_interval(0, &ts)), actual: 0 vs -1 errno 94 Not permitted in capability mode capmode.cc:276: Failure Value of: rc Actual: 1 Expected: 0 [ FAILED ] Capmode.AllowedSchedSyscallsForked (11 ms) and reports success/failure via the result code. I have a couple of questions. First, do we have an existing pattern for connecting googletest tests to Kyua? We could incorporate it fairly easily by just encapsulating the whole test, but integrating at the level of individual tests would be much better. Second, I wonder if we should import these tests into contrib/ and build them during buildworld, or put them into a port and install them into /usr/local/tests/capsicum. I'd lean towards putting them in contrib since Capsicum is an integrated part of the base system, but it might be a bit easier to maintain if we keep it as a port/package. Presumably the Jenkins run will just execute all tests in /usr/tests/ and /usr/local/tests/. [1] https://github.com/google/capsicum-test From owner-freebsd-testing@FreeBSD.ORG Wed Jun 3 15:30:13 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D2D08385; Wed, 3 Jun 2015 15:30:13 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mail-yk0-x22e.google.com (mail-yk0-x22e.google.com [IPv6:2607:f8b0:4002:c07::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 92D8A1493; Wed, 3 Jun 2015 15:30:13 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: by ykfl8 with SMTP id l8so4758310ykf.1; Wed, 03 Jun 2015 08:30:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=sBqtdjNupHM/QqVaIF81R57sI8aELwDjhGv04GFiW+g=; b=OdU5/3G1Bxw3tNwjR4w7VxvjdkFEBxcGKZoJHWsErnHTbQT73uf3RqqYIahW7EME6V KZmCVVvzmmL/oiQb99Rx7hMHzXu1oLfq2ND3/H5sfDfVGG/7UpUy5BXPAonWqNgIVVlD K7PRwMCw7Fr+BlEOxAP9TgI9t0reX1CgE8W2ymZHKN+SNOItFSKmTvoJmW8wJ7nYzmBX UfnvhQiUXlW4uwd4DEZTdsKf+0jnnz/j7PRoViNmUIYZnMMGZFVvjCjJdl3KBO3TimH/ iheAu/YnU5qWyUqEz+I0sc9CpiVg7Hj0raM1J2dsRqVJvqYnF58I0Ful4ttBI0nMC+rz 7gdw== MIME-Version: 1.0 X-Received: by 10.170.125.208 with SMTP id r199mr13580554ykb.15.1433345412668; Wed, 03 Jun 2015 08:30:12 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.13.233.70 with HTTP; Wed, 3 Jun 2015 08:30:12 -0700 (PDT) In-Reply-To: References: Date: Wed, 3 Jun 2015 08:30:12 -0700 X-Google-Sender-Auth: giv6x2CWat6G4c9UzedNnqMrM1Q Message-ID: Subject: Re: Incorporating the Capsicum test suite From: Craig Rodrigues To: Ed Maste Cc: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 15:30:14 -0000 On Tue, Jun 2, 2015 at 1:40 PM, Ed Maste wrote: > > I have a couple of questions. First, do we have an existing pattern > for connecting googletest tests to Kyua? > No we do not. Googletest looks like it is basically trying to do the same thing as ATF or CUnit. Googletest has a way to generate JUnit XML test output: https://code.google.com/p/googletest/wiki/AdvancedGuide#Generating_an_XML_Report The easiest thing to do would be to configure the Capsicum tests to generate this XML test output. This can be imported natively into Jenkins, which can then parse the test report. This would require no changes to the Capsicum tests. If you want to integrate googletest with kyua, then things get tricky. On the googletest page: https://code.google.com/p/googletest/ they list a third party plugin which can take googletest and generate TAP output. I don't know how well that works. It would be nice if that functionality was integrated in googletest natively. Another option would be to modify kyua, so that it can parse the output of googletest natively. This would be quite nice. I haven't found extending kyua to be very easy....it requires good knowledge of C++ and Lua. Also, whoever contributes code to kyua must sign a Contributor License Agreement (CLA) with Google. That's something to keep in mind. Another option would be to rewrite the Capsicum tests in ATF. I don't know how many tests there are, but this might be doable. ATF and Googletest look similar. If you wanted to import the Capsicum tests into FreeBSD as they are right now, then it looks like you would need to import googletest into FreeBSD as well. googletest is BSD licensed, so that is not a problem, but we already have ATF in the tree, so I don't know how useful it would be to have multiple test frameworks in the base system. -- Craig From owner-freebsd-testing@FreeBSD.ORG Wed Jun 3 19:19:05 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0E0E1551; Wed, 3 Jun 2015 19:19:05 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pa0-x22d.google.com (mail-pa0-x22d.google.com [IPv6:2607:f8b0:400e:c03::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D0AAF1EDA; Wed, 3 Jun 2015 19:19:04 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by payr10 with SMTP id r10so13284589pay.1; Wed, 03 Jun 2015 12:19:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=jwvci3Wn25Dg1gclzjwOj+PpKTaocPogM3xDrBqyHyA=; b=Td/lHUp788RmpkSenRYlsRZ5LxgIbSmemUpoBeNT/ab0y0BzG/nbDy24YNeYfEpIuy l43ip+vPwP2gquYm5dWLZuhIl5+sw7Q67E2r19XJMqUeZgC1xIvoFHrohlyNbrdkPoBu XaBbGmOhssvZ6FiRkKBDtPKPe+MYQUcEmG25xWuYqQdKBM2ouEWV5+rRM9Y72J3xTuNB sD3MZkBZObK8HE3lNOicT+yqBf9MnB+hUfQ4+ETtsfyqgpsIN5DQE0QP3+dTsC2OUvnn sgGqPAgVbWEvK9hCIHizx7ZrZjFz+KQRnvv71Zs3/qgIERx1sNB9UoC23Wn0uIp5e7Am Sviw== X-Received: by 10.70.129.17 with SMTP id ns17mr62779579pdb.12.1433359144344; Wed, 03 Jun 2015 12:19:04 -0700 (PDT) Received: from [33.161.27.125] ([172.56.32.97]) by mx.google.com with ESMTPSA id qy7sm1529742pbb.12.2015.06.03.12.19.03 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 03 Jun 2015 12:19:03 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: Incorporating the Capsicum test suite From: Garrett Cooper X-Mailer: iPhone Mail (12F70) In-Reply-To: Date: Wed, 3 Jun 2015 12:19:02 -0700 Cc: Ed Maste , "freebsd-testing@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <0582DC06-4021-443D-B693-93F848654441@gmail.com> References: To: Craig Rodrigues X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 19:19:05 -0000 > On Jun 3, 2015, at 08:30, Craig Rodrigues wrote: >=20 >> On Tue, Jun 2, 2015 at 1:40 PM, Ed Maste wrote: >>=20 >>=20 >> I have a couple of questions. First, do we have an existing pattern >> for connecting googletest tests to Kyua? >=20 > No we do not. Googletest looks like it is basically trying to do > the same thing as ATF or CUnit. >=20 > Googletest has a way to generate JUnit XML test output: > https://code.google.com/p/googletest/wiki/AdvancedGuide#Generating_an_XML_= Report >=20 > The easiest thing to do would be to configure the Capsicum tests to > generate this XML test output. This can be imported natively into > Jenkins, which can then parse the test report. > This would require no changes to the Capsicum tests. >=20 > If you want to integrate googletest with kyua, then things get tricky. > On the googletest page: https://code.google.com/p/googletest/ > they list a third party plugin which can take googletest and generate TAP > output. > I don't know how well that works. It would be nice if that functionality > was > integrated in googletest natively. >=20 > Another option would be to modify kyua, so that it can parse the output of= > googletest > natively. This would be quite nice. I haven't found extending kyua to be= > very > easy....it requires good knowledge of C++ and Lua. Also, whoever > contributes > code to kyua must sign a Contributor License Agreement (CLA) with Google. > That's something to keep in mind. >=20 > Another option would be to rewrite the Capsicum tests in ATF. > I don't know how many tests there are, but this might be doable. > ATF and Googletest look similar. >=20 > If you wanted to import the Capsicum tests into FreeBSD as they are right > now, then it looks > like you would need to import googletest into FreeBSD as well. googletest= > is BSD licensed, > so that is not a problem, but we already have ATF in the tree, so I don't > know how > useful it would be to have multiple test frameworks in the base system. Hi Ed, I'll take a look at the test infrastructure and see if I can write a sim= ple shim for ATF to interact with it. I agree with a lot of what Craig said. The Google corporate CLA has impe= ded engineers in EMC / Isilon and some other groups from contributing back t= o the ATF / Kyua projects ;( (this is why I've just been filing bugs against= the projects and have been focusing my efforts on integrating tools/regress= ion tests into Kyua), so anything that needs to be done development wise on t= hese projects will need to be done internal to the FreeBSD project for the t= ime being. Thanks! -NGie= From owner-freebsd-testing@FreeBSD.ORG Wed Jun 3 19:51:33 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3F274EB5; Wed, 3 Jun 2015 19:51:33 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-ig0-x233.google.com (mail-ig0-x233.google.com [IPv6:2607:f8b0:4001:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 08E1817B6; Wed, 3 Jun 2015 19:51:33 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by igbpi8 with SMTP id pi8so121200490igb.1; Wed, 03 Jun 2015 12:51:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=6N/Vihz0xUPOSyQiU+QU44nweMOt2pwPVfMCaezdLCg=; b=CpQh70/ArSUXOW04pE7BozkETeSpm/rlL+IGu2xX6W8Tvgu1Z+N74DIKKV+d0Xb/OV CMH8ero60TCHZtvq1Ylkdr7uI7vsIGTGd3KyRcEDfi0PZ5y8JNGUfoJXKWNOfJzDKIpL cau2sLNK3muSmzi89P5uRs1GSbxbZjnPpph6t00iyxu4LzAqyUe7kEvAbBh2CzBDvGTI VY/C45WVfBGrrxoOFNCbv8tEboPx2KfGnnb0LtFHIQMua2H2dlyJ1/sC8l5tgQv50Aw3 NMZs4MNJwg0+vKpyg5peUY2TN4QYcu0sd3SA71AD3fbC6R7VVNNqvyGlkFr203kkVF60 SXtA== X-Received: by 10.50.30.105 with SMTP id r9mr29482742igh.11.1433361092508; Wed, 03 Jun 2015 12:51:32 -0700 (PDT) MIME-Version: 1.0 Sender: carpeddiem@gmail.com Received: by 10.107.48.3 with HTTP; Wed, 3 Jun 2015 12:51:11 -0700 (PDT) In-Reply-To: References: From: Ed Maste Date: Wed, 3 Jun 2015 15:51:11 -0400 X-Google-Sender-Auth: b0KK3IvMy5uXVB0Bw3Rz5xFS56c Message-ID: Subject: Re: Incorporating the Capsicum test suite To: Craig Rodrigues Cc: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 19:51:33 -0000 On 3 June 2015 at 11:30, Craig Rodrigues wrote: > On Tue, Jun 2, 2015 at 1:40 PM, Ed Maste wrote: >> >> I have a couple of questions. First, do we have an existing pattern >> for connecting googletest tests to Kyua? > > No we do not. Googletest looks like it is basically trying to do > the same thing as ATF or CUnit. Yes, the intent is the same. My desire to integrate it with Kyua is so that we can add this test coverage without needing the user to run additional commands, parse different output formats, etc. We could integrate the existing tests as-is into Jenkins fairly easily, but I also want us to move to a model where developers consistently run the test suite prior to committing a change. > Another option would be to modify kyua, so that it can parse the output of > googletest > natively. This would be quite nice. I haven't found extending kyua to be > very > easy....it requires good knowledge of C++ and Lua. Also, whoever > contributes > code to kyua must sign a Contributor License Agreement (CLA) with Google. > That's something to keep in mind. It seems to me that this is the best way to go, with connecting the existing tests as-is into Jenkins as my second choice. FreeBSD developers who wish to make changes to Kyua and the Google CLA is an issue we could fork it and make changes in our own version. It's not ideal, but in practice if we don't expect large, conflicting changes from upstream it should be a workable approach. > Another option would be to rewrite the Capsicum tests in ATF. > I don't know how many tests there are, but this might be doable. > ATF and Googletest look similar. I think this is feasible, and may make sense for the Capsicum tests. However, the idea extends beyond the Capsicum tests. Clang/LLVM (and LLDB) have a large test suite that I'd like incorporate into our process, and other 3rd party software does as well. It's not feasible to try to convert all of these to ATF. > If you wanted to import the Capsicum tests into FreeBSD as they are right > now, then it looks > like you would need to import googletest into FreeBSD as well. googletest > is BSD licensed, > so that is not a problem, but we already have ATF in the tree, so I don't > know how > useful it would be to have multiple test frameworks in the base system. The parts of googletest actually required for the capsicum tests are fairly small and self-contained (even though the entire googletest package is included in the capsicum-test repo). Assuming capsicum-test gets stripped it down to just what's required I might just consider the embedded googletest part of capsicum-test, rather than importing it as a standalone project and having capsicum-test use it.