From owner-freebsd-testing@FreeBSD.ORG Sun Oct 19 12:24:36 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9EE12F9B for ; Sun, 19 Oct 2014 12:24:36 +0000 (UTC) Received: from smtprelay03.ispgateway.de (smtprelay03.ispgateway.de [80.67.29.28]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5D84FFD9 for ; Sun, 19 Oct 2014 12:24:35 +0000 (UTC) Received: from [78.35.153.72] (helo=fabiankeil.de) by smtprelay03.ispgateway.de with esmtpsa (SSLv3:AES128-SHA:128) (Exim 4.84) (envelope-from ) id 1XfpVS-00086F-Aq for freebsd-testing@freebsd.org; Sun, 19 Oct 2014 14:22:42 +0200 Date: Sun, 19 Oct 2014 14:22:44 +0200 From: Fabian Keil To: freebsd-testing@freebsd.org Subject: Re: atf_check() equivalent to test individual shell functions? Message-ID: <1dfb5b7d.6133bc91@fabiankeil.de> In-Reply-To: <1552a16f.63e4cafe@fabiankeil.de> References: <41d8a2fb.75b47028@fabiankeil.de> <1552a16f.63e4cafe@fabiankeil.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/Bb/IFLwJuM1q=7yysZJ_xuE"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Oct 2014 12:24:36 -0000 --Sig_/Bb/IFLwJuM1q=7yysZJ_xuE Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Fabian Keil wrote: > Julio Merino wrote: >=20 > > On Sat, Jul 26, 2014 at 4:30 AM, Fabian Keil > > wrote: > > > Here's another zogftw-related problem. > >=20 > > What's zogftw? >=20 > The program I'm writing tests for: > http://www.fabiankeil.de/gehacktes/zogftw/ FYI, sysutils/zogftw 2014-09-19 comes with a bunch of ATF test cases, the port optionally executes them before packaging: http://www.fabiankeil.de/gehacktes/zogftw/#tests With the help of Redports this has prevented a bug that affected 8.4 and 9.2 (which I don't use myself) from making it into the final release: https://www.redports.org/~fk/20140919190307-42924-246115/zogftw-0.0.2014.09= .19.log Debugging this was a bit painful due to lack of shell access and DTrace, but obviously having a user debug this after the release would have been painful, too. Currently most of the system binaries zogftw relies are only mocked, but I intend to look into optionally using the real ones in the future. Fabian --Sig_/Bb/IFLwJuM1q=7yysZJ_xuE Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlRDrRQACgkQBYqIVf93VJ0i/wCffH4hlefPeQBKZbxKGF9bktLk b0wAoI3sulvXGAzjQpxmExNRYnmbZnVp =qqck -----END PGP SIGNATURE----- --Sig_/Bb/IFLwJuM1q=7yysZJ_xuE-- From owner-freebsd-testing@FreeBSD.ORG Mon Oct 20 16:42:14 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4B2D9AF; Mon, 20 Oct 2014 16:42:14 +0000 (UTC) Received: from mail-la0-x235.google.com (mail-la0-x235.google.com [IPv6:2a00:1450:4010:c03::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9B3BB8E3; Mon, 20 Oct 2014 16:42:13 +0000 (UTC) Received: by mail-la0-f53.google.com with SMTP id gq15so4250272lab.26 for ; Mon, 20 Oct 2014 09:42:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=J9LMJCWL5A+RSgqVdMyk4ewqL2aAmkPbAxkmRUlY1kU=; b=o/WwVZHS/nfaMRFDG06zn98TLqZI7kP9eCunH/HYz95iF+UhPPTHlhOcXJ5yJXKPfr 92FjH25EambLgzcOuSYUyiIxddw0BvjRB45/BvNW/YjBaNvDKDdIPERf1ei4mE73RRSM OXJMTxjbNoft8zibkIR+O+WZ3V3i3d9nhfkFJFbnQL4W+1pP2xhi1awPfIpRkCiOnd4r IP/VwtJY0L/bAJeqThLYn2KxcubKqSWOpozenXjzzspydZPTqB9ABISBtfJcVpIzvAPi 249/JtOL99wWr+JjVgZ1bKZMS7xzOUKfGOjAzB+DEhEdXQbUMKl1T0BqzBuv062TkP+U 9icg== MIME-Version: 1.0 X-Received: by 10.112.12.35 with SMTP id v3mr10546203lbb.80.1413823329572; Mon, 20 Oct 2014 09:42:09 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.131.66 with HTTP; Mon, 20 Oct 2014 09:42:09 -0700 (PDT) Date: Mon, 20 Oct 2014 09:42:09 -0700 X-Google-Sender-Auth: VtkaPBLJ5VtfhZtnYpsoP4wBOlk Message-ID: Subject: Jenkins, Kyua, and Bhyve used for FreeBSD OS testing From: Craig Rodrigues To: freebsd-current Current , "freebsd-testing@freebsd.org" , kyua-discuss@googlegroups.com Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Oct 2014 16:42:14 -0000 Hi, FYI, Kohsuke Kawaguchi, the lead developer of Jenkins, accepted my posting on the Jenkins blog, which describes how the FreeBSD project is using Jenkins, Kyua, and Bhyve for FreeBSD OS testing: http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing -- Craig From owner-freebsd-testing@FreeBSD.ORG Tue Oct 21 05:34:53 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C939F7B5; Tue, 21 Oct 2014 05:34:53 +0000 (UTC) Received: from mail-lb0-x235.google.com (mail-lb0-x235.google.com [IPv6:2a00:1450:4010:c04::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 20AB71FF; Tue, 21 Oct 2014 05:34:52 +0000 (UTC) Received: by mail-lb0-f181.google.com with SMTP id l4so346230lbv.26 for ; Mon, 20 Oct 2014 22:34:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=4dj4iB0+z3FWJ04kCaVxie6z5Xa/8/LwoTgUuG/YRsA=; b=KFAf9Yy/WPB2LeSSTbZoiJ+KUezyLrUMlxd9dKidMScynxnYazVt/0K5P3cM/fnWyj scVHfQGVaX3XcX7RJkcJ6Fmm3przsCmUejuB31NDwndy9GTfRhLhbukLSTPp+MkFpfb/ YP5Ex2yIW/H5Od9FO95++C3rXwYQWRqybv9Aar+1ESZ4gm/8Idi47kAko+xlan8uoe9h plnPkRmbBGJgGPFf7PAHzQr3u+fChyEiCmpBGnbqnqHsmC1vsZz7HuHjU3BqJT3zoK+C D5ZGR3efLNju+hR9StM+vNvGdabE0zQzqSAz+SB92zvlIi1VWVGrLVxVZuT59lNzruqw TTJg== MIME-Version: 1.0 X-Received: by 10.112.12.35 with SMTP id v3mr13772960lbb.80.1413869690994; Mon, 20 Oct 2014 22:34:50 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.131.66 with HTTP; Mon, 20 Oct 2014 22:34:50 -0700 (PDT) In-Reply-To: <17CD4750-33C9-4D03-B26D-9A72E8019E3B@mu.org> References: <17CD4750-33C9-4D03-B26D-9A72E8019E3B@mu.org> Date: Mon, 20 Oct 2014 22:34:50 -0700 X-Google-Sender-Auth: LjxTeq5us_4opD-NsLn0ngmYX5E Message-ID: Subject: Re: Jenkins, Kyua, and Bhyve used for FreeBSD OS testing From: Craig Rodrigues To: freebsd-current Current Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: kyua-discuss@googlegroups.com, "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Oct 2014 05:34:53 -0000 Hi, Thanks for those kind words, Alfred! I would like to point out a few things. (1) Jenkins team are really easy to work with, and I've been pushing fixes upstream to them, such as: https://github.com/jenkinsci/jenkins/pull/1387 which fixes shared library support on FreeBSD, Windows, Linux/ARM, Linux/PPC. Jenkins has something called the Jenkins Pull Request builder, so that when you submit a patch to them via a Github Pull Request, it checks out the Jenkins source, patches it, builds it, and runs automated tests. This helps the Jenkins team to decide whether to pull in a patch or not. It's quick, easy, works well........and honestly, makes working with an open source project fun! Unfortunately, working with FreeBSD can seem to be antiquated and a drudgery in comparison, but things are improving with things like Phabricator and Bugzilla. (2) By fixing up the unit tests and paying attention to failures, I've found real problems that need to be fixed. The yacc unit tests started failing after recent imports of byacc. I traced down the problem, and reported it upstream to Thomas Dickey who maintains byacc. He fixed the problem (see http://invisible-island.net/byacc/CHANGES 2014-10-05 and 2014-10-06) and I confirmed that fixed the unit tests. We have the fixed byacc in HEAD and stable/10 branches. Although I've got the ball rolling, hopefully more people in the FreeBSD community can help out by writing tests, paying attention to test failures, helping out with devops issues related to Jenkins, etc. For folks interested, hop on freebsd-testing@freebsd.org and help pitch in! -- Craig On Mon, Oct 20, 2014 at 9:56 PM, Alfred Perlstein wrote: > Craig this is really great. Thanks for doing this and thanks for the > Jenkins guys on giving a shout out! > > On Oct 20, 2014, at 9:42 AM, Craig Rodrigues wrote: > > > Hi, > > > > FYI, Kohsuke Kawaguchi, the lead developer of Jenkins, > > accepted my posting on the Jenkins blog, which describes > > how the FreeBSD project is using Jenkins, Kyua, and Bhyve > > for FreeBSD OS testing: > > > > http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing > > > > -- > > Craig > > From owner-freebsd-testing@FreeBSD.ORG Wed Oct 22 15:20:58 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8F32F229 for ; Wed, 22 Oct 2014 15:20:58 +0000 (UTC) Received: from mouf.net (mouf.net [IPv6:2607:fc50:0:4400:216:3eff:fe69:33b3]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mouf.net", Issuer "mouf.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 549898D5 for ; Wed, 22 Oct 2014 15:20:58 +0000 (UTC) Received: from mouf.net (swills@mouf [199.48.129.64]) by mouf.net (8.14.5/8.14.5) with ESMTP id s9MFKpCk086284 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 22 Oct 2014 15:20:56 GMT (envelope-from swills@mouf.net) Received: (from swills@localhost) by mouf.net (8.14.5/8.14.5/Submit) id s9MFKpC1086283; Wed, 22 Oct 2014 15:20:51 GMT (envelope-from swills) Date: Wed, 22 Oct 2014 15:20:51 +0000 From: Steve Wills To: kyua-discuss@googlegroups.com, freebsd-testing@freebsd.org Subject: jenkins and kyua junit test output question Message-ID: <20141022152047.GE18058@mouf.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.23 (2014-03-12) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (mouf.net [199.48.129.64]); Wed, 22 Oct 2014 15:20:57 +0000 (UTC) X-Spam-Status: No, score=0.0 required=4.5 tests=HEADER_FROM_DIFFERENT_DOMAINS autolearn=unavailable autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on mouf.net X-Virus-Scanned: clamav-milter 0.98.3 at mouf.net X-Virus-Status: Clean X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Oct 2014 15:20:58 -0000 Hi, Was running into an issue with Jenkins and Kyua and wondered if anyone might have any insight. The issue is in the pkg build test. This build failed: http://jenkins.mouf.net/job/pkg/1387/ due to test issues, but they don't show up in the junit output, I think. On the other hand if you look at the html generated for this, it shows up there: http://kyua.mouf.net/html/1387/ Any ideas what we need to change? I can send the job config or other details if needed. Thanks, Steve PS. I'm not on the list so please copy me on replies. From owner-freebsd-testing@FreeBSD.ORG Thu Oct 23 20:47:20 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EBE81C6B for ; Thu, 23 Oct 2014 20:47:20 +0000 (UTC) Received: from mail-qg0-f41.google.com (mail-qg0-f41.google.com [209.85.192.41]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AC7A6E50 for ; Thu, 23 Oct 2014 20:47:20 +0000 (UTC) Received: by mail-qg0-f41.google.com with SMTP id a108so90172qge.28 for ; Thu, 23 Oct 2014 13:47:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:content-type; bh=XQi5noA4anbVykpAxZxdJSKq85wzg0n8FkQYzKk5j2I=; b=Mu1Wj+inynQSZEHaui3q+y669+02tobKEBxxludlrlNsTjqKrIO0SiGLeiwSix4cCy FNNdScJ5C4SEQgZlEeIdwBNK6GRW14CscwSLWd0kojklZufLYk4zNz9QN3cVbjNOnpet ZOFcx4+zAwB8ghJBNw/7Ym2B3dzQWDz95BGBocRQZPnRq4u8WhqXP1a3IXd8rqyHR/BF 5xpF1T0Y6RSt86dru8yRVkcii6HaRcj48QRpT73OSur8Nr/W0P1ymcjuWmxCybwbcb5o OYQpFlezYjC0NmLMhHW7KRs+Uegvuk5/eboZR4fLhUKlyU3L+UtUyJ0EVxe/e636HyXX X3nw== X-Gm-Message-State: ALoCoQmGDpwRqgs8483SlzAuNZ3h6UZEg0lQVkSpIATz6rbzLxIpO1M6SbwCer+jC43ogM0SCrpT X-Received: by 10.170.139.86 with SMTP id g83mr53698ykc.96.1414097233552; Thu, 23 Oct 2014 13:47:13 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.75.134 with HTTP; Thu, 23 Oct 2014 13:46:53 -0700 (PDT) X-Originating-IP: [2620:0:1040:404:4125:2f63:d571:20d0] In-Reply-To: References: From: Julio Merino Date: Thu, 23 Oct 2014 21:46:53 +0100 X-Google-Sender-Auth: M16mz0Mw4s2SqO_9JqEb80su7Bk Message-ID: Subject: Fwd: ANNOUNCE: ATF 0.21 released To: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Oct 2014 20:47:21 -0000 FYI. Will update the ports shortly. Also working on an import of this release into current, but that's a bit trickier so it will take a bit longer. Lastly, also note the To: line. atf-devel is being phased out in favor of atf-discuss. ---------- Forwarded message ---------- From: Julio Merino Date: Thu, Oct 23, 2014 at 9:29 PM Subject: ANNOUNCE: ATF 0.21 released To: atf-devel@netbsd.org, atf-discuss@googlegroups.com Hello everyone, I'm pleased to announce that ATF 0.21 is available for download. https://github.com/jmmv/atf/releases/tag/atf-0.21 This is a major "cleanup release" in which many outstanding bugs have been addressed and in which the documentation has significantly improved to provide better usability. Please refer to the link above for a link to the release file as well as the detailed list of all changes. Enjoy! From owner-freebsd-testing@FreeBSD.ORG Thu Oct 23 20:47:31 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0F19DC7E for ; Thu, 23 Oct 2014 20:47:31 +0000 (UTC) Received: from mail-qa0-f52.google.com (mail-qa0-f52.google.com [209.85.216.52]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C3E5BE51 for ; Thu, 23 Oct 2014 20:47:30 +0000 (UTC) Received: by mail-qa0-f52.google.com with SMTP id v10so86264qac.25 for ; Thu, 23 Oct 2014 13:47:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:content-type; bh=QfaXYFla2s/aZS7BAX/LX2qlelyO7KXg8tkANlKTFr0=; b=ZhPKarNpSt/LgW9/I8yUE1RZCc0V4Yq0B3/eP0ldL0pE0Db9anrJTk3vD3VNaoktO2 lcs84LUUZOg+fwrgxIy+QzQizflsGu1CkSnTTEoSB68vGvmDkiPYbSfB42ZOG6nGsCO7 QucVQujBZhkdfMyZaF9ajnqk+lL4MHzu2B0vrEcb6/IldqvsltjzUSGTPaYxwCrkoEga Ci+ehr6wfyIjJjOvg3C2pcYcmjEfoa3+mxiwQP79V6FkOuZbmB2p50oj7O5ZdVu61heM ojqWk0CHvPth8LGY2I6VhKBACmO5e7/9FoEu7uqNPqfVhHpArHNYXsrxX1LX8Styyth4 xI1w== X-Gm-Message-State: ALoCoQlJQAGYdSh7o6uiawb7V3TM2KErMEjQji6L5ywfmc+6A3SuSNGRtvDULq4v4Pe4xz2Pcaxt X-Received: by 10.170.162.86 with SMTP id e83mr1075877ykd.113.1414097249305; Thu, 23 Oct 2014 13:47:29 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.75.134 with HTTP; Thu, 23 Oct 2014 13:47:08 -0700 (PDT) X-Originating-IP: [2620:0:1040:404:4125:2f63:d571:20d0] In-Reply-To: References: From: Julio Merino Date: Thu, 23 Oct 2014 21:47:08 +0100 X-Google-Sender-Auth: aejfrOX6kMJntRcVFQyemColrQo Message-ID: Subject: Fwd: ANNOUNCE: Kyua 0.11 released To: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Oct 2014 20:47:31 -0000 FYI. Will update the ports shortly. ---------- Forwarded message ---------- From: 'Julio Merino' via kyua-discuss Date: Thu, Oct 23, 2014 at 9:43 PM Subject: ANNOUNCE: Kyua 0.11 released To: kyua-discuss@googlegroups.com Hello everyone, I'm pleased to announce that Kyua 0.11 is available for download: https://github.com/jmmv/kyua/releases/tag/kyua-0.11 This is a major "cleanup release" in which many outstanding bugs have been addressed and in which the documentation has significantly improved to provide better usability. Please refer to the link above for a link to the release file as well as the detailed list of all changes. Enjoy! From owner-freebsd-testing@FreeBSD.ORG Thu Oct 23 22:30:14 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AADBD400; Thu, 23 Oct 2014 22:30:14 +0000 (UTC) Received: from mail-la0-x22e.google.com (mail-la0-x22e.google.com [IPv6:2a00:1450:4010:c03::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BA0EAB50; Thu, 23 Oct 2014 22:30:13 +0000 (UTC) Received: by mail-la0-f46.google.com with SMTP id gi9so1704324lab.33 for ; Thu, 23 Oct 2014 15:30:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=/5fqvNvE/qdXl+isJp4pGo9u4qWhv3pR10hD+WYUvRE=; b=bNc8/R4fGAYwoTexq4fMoSUj1U/wgQ5WY3kuf4WXaC8t3F5jwPuv+Vx3i7liDMqJr+ DUksZTXY8/Arf3lmuqQEvAwl8FlkQiRMN0RSPfCoeiCptoQndikPuvr39NDHtrs9A5xe jtGnqBa+kJZwAVf8WgKDiNtfttlu6XG+Ob7GQVvxybL9NFwZ1gjT2IXjZ2vu/p6srDM4 6PjiyhblmqUwqbyv2156oJLC+C+n8wE2XL4VPE9psR+ZzdsKK/Q8Ghy/6nAHl2lflbuh Jzjd24oPpLFfagvlQDv5ElIFRXDaoCa0K9+kCfKKfdARlrdE7YJ7csC2XJvoxWumwebq H4Rg== MIME-Version: 1.0 X-Received: by 10.152.42.172 with SMTP id p12mr438220lal.11.1414103411024; Thu, 23 Oct 2014 15:30:11 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.84.197 with HTTP; Thu, 23 Oct 2014 15:30:10 -0700 (PDT) In-Reply-To: References: Date: Thu, 23 Oct 2014 15:30:10 -0700 X-Google-Sender-Auth: Vjq-Kaj1UQORJRV-k9cUp2U4-Mw Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Craig Rodrigues To: Garrett Cooper Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Oct 2014 22:30:14 -0000 Garrett Cooper wrote: > > Hi Craig! > As much as everyone would like to take i386 out to pasture, > there's a large degree of value in running i386 tests on 11-CURRENT and > 10-STABLE (I've caught some interesting build bugs and test bugs by running > on my i386/CURRENT VM). Are there any plans to have i386 executors running > tests anytime soon (does bhyve support i386?)? > We (jenkins-admin@freebsd.org) have been busy the 5 months since BSDCan 2014 as you can see by our status report, which lists what we have done, and what our future plans are: https://www.freebsd.org/news/status/report-2014-07-2014-09.html#Jenkins-Continuous-Integration-for-FreeBSD Integrating Jenkins with Kyua on amd64 was a major milestone which was achieved, and was even mentioned on the Jenkins web site: http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing Now, where to take this further (such as i386) is an interesting question. Personally, I would like to see: -> integration of automated kyua testing with the FreeBSD release engineering process -> more involvement from FreeBSD developers, and even companies such as EMC/Isilon, who can: -> write tests -> suggest and implement new tests (network, storage, VM, etc.) -> help with devops maintenance of the existing jenkins.freebsd.org cluster -> improve bhyve support in libvirt -> set up their own Jenkins build environments outside of FreeBSD, and help test things in their own private environments I've had requests for; -> running tests in bhyve VM's with very small memory footprints (Adrian Chadd) -> running tests in MIPS environment (Adrian Chadd) -> running Java JDK tests (Kip Macy) -> i386 builds (Garrett Cooper) There are a lot of different directions to take this, but without more people (and companies) pitching in and helping out, progress is limited by bodies working on the stuff. I will be at the MeetBSD Vendor summit (not the conference) Nov. 3-4, so hopefully anyone interested in pushing this stuff forward can talk to me there. -- Craig From owner-freebsd-testing@FreeBSD.ORG Fri Oct 24 03:31:28 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7423A981; Fri, 24 Oct 2014 03:31:28 +0000 (UTC) Received: from mx1.scaleengine.net (beauharnois2.bhs1.scaleengine.net [142.4.218.15]) by mx1.freebsd.org (Postfix) with ESMTP id 32A77C42; Fri, 24 Oct 2014 03:31:27 +0000 (UTC) Received: from [192.168.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 0731E63E88; Fri, 24 Oct 2014 03:31:25 +0000 (UTC) Message-ID: <5449C81B.8080008@freebsd.org> Date: Thu, 23 Oct 2014 23:31:39 -0400 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: freebsd-virtualization@freebsd.org, freebsd-testing@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins References: In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="EO7bdR25mcpewkJ1lSwKXBRl5VclR2krx" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Oct 2014 03:31:28 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --EO7bdR25mcpewkJ1lSwKXBRl5VclR2krx Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2014-10-23 18:30, Craig Rodrigues wrote: > Garrett Cooper wrote: >=20 >> >> Hi Craig! >> As much as everyone would like to take i386 out to pasture, >> there's a large degree of value in running i386 tests on 11-CURRENT an= d >> 10-STABLE (I've caught some interesting build bugs and test bugs by ru= nning >> on my i386/CURRENT VM). Are there any plans to have i386 executors run= ning >> tests anytime soon (does bhyve support i386?)? >> >=20 >=20 > We (jenkins-admin@freebsd.org) have been busy the 5 months since BSDCan= > 2014 as > you can see by our status report, which lists what we have done, and wh= at > our future plans are: >=20 > https://www.freebsd.org/news/status/report-2014-07-2014-09.html#Jenkins= -Continuous-Integration-for-FreeBSD >=20 > Integrating Jenkins with Kyua on amd64 was a major milestone which was > achieved, and > was even mentioned on the Jenkins web site: >=20 > http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing >=20 > Now, where to take this further (such as i386) is an interesting questi= on. >=20 > Personally, I would like to see: > -> integration of automated kyua testing with the FreeBSD release > engineering process > -> more involvement from FreeBSD developers, and even companies such= as > EMC/Isilon, > who can: > -> write tests > -> suggest and implement new tests (network, storage, VM, et= c.) > -> help with devops maintenance of the existing > jenkins.freebsd.org cluster > -> improve bhyve support in libvirt > -> set up their own Jenkins build environments outside of > FreeBSD, and help test things in their own private environments >=20 > I've had requests for; > -> running tests in bhyve VM's with very small memory footprints > (Adrian Chadd) > -> running tests in MIPS environment (Adrian Chadd) > -> running Java JDK tests (Kip Macy) > -> i386 builds (Garrett Cooper) >=20 > There are a lot of different directions to take this, but without more > people (and companies) pitching in and helping out, > progress is limited by bodies working on the stuff. >=20 > I will be at the MeetBSD Vendor summit (not the conference) Nov. 3-4, s= o > hopefully > anyone interested in pushing this stuff forward can talk to me there. >=20 > -- > Craig > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fr= eebsd.org" >=20 At the Cambridge Dev Summit, Xinuous specifically mentioned helping with testing and writing tests. Might be good people to reach out to --=20 Allan Jude --EO7bdR25mcpewkJ1lSwKXBRl5VclR2krx Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJUScgeAAoJEJrBFpNRJZKfOSoP/RjL9wJCyl04M0OHCmXb5n0f sUbbElAF+Us4URbVNPVG5mVZooKT5Dl5WFm5uEjXHkzXXRbK24IPere4Tn46H3uw BB4AW9gmOcOxSOEYtghATIzLhBI+UkEZxLJrhKyQIMZFVg9P7hm2TC9wjUc6S/d3 Z6x+bd/ukh0ojRfAXaZ3oLVV2N/uAIgIOGAkDk34WApF8EjFwEO33dt035qO7/n4 OmgP3J2JvprEM2HKvYfgnDUpvYHz/DgJAoRaW0vTD2OK6yDuCF8ROIiBOI7gvl9M NKFUP4Wa8zIUF4u+WtqJ7x1yyDnvRhe/8Z63aSdcbITQ19nSX8YvQ0wCbpXScOgf 5lCkolSBolSNDsNeILVu3f4JV5/N8QwCV/9B1XuzGN98+cZiLLoqvXwkkz3HscZR QXwYlqVTP0VlJ9Z4yZMKjVEKwzYpzTwNR30shpIMhhLyzJVeAOAutaEhkTVuPEci 2H5jcCssQlCGBs24gSsl8yh5sN/pz2RmSa2t8OOFMtYCKpmYtZxs5XEObeNqI+qo Vseu8+zqAMFh59sRBdynp9BYzIY0vlHhJgklSDT8PBGECBvR7X0w8S9Pg4efjGaq YHs+4yoDMMQklap3dLJoqzeHsbrDfKy+30YTbx8UfB+ffrA8er1uLMJnlOzfqGR0 IMSS1NRGiP++uUgKIbFO =U9+b -----END PGP SIGNATURE----- --EO7bdR25mcpewkJ1lSwKXBRl5VclR2krx-- From owner-freebsd-testing@FreeBSD.ORG Fri Oct 24 04:20:11 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4EF5457F; Fri, 24 Oct 2014 04:20:11 +0000 (UTC) Received: from mail-lb0-x229.google.com (mail-lb0-x229.google.com [IPv6:2a00:1450:4010:c04::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 457A9FD6; Fri, 24 Oct 2014 04:20:10 +0000 (UTC) Received: by mail-lb0-f169.google.com with SMTP id 10so1931539lbg.28 for ; Thu, 23 Oct 2014 21:20:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=a4h/yTVcb77Uzncemt6arDYRCSWVMJfN6seay5CYQZY=; b=VbmeUbQHCEuXpBDenR4ge+Lw69ZZKtTspwJV0Z28I/sVoI4mqAejvxNA/f9MtErN3R aOVX//GGg/bTzArUSC9bTtgdl76jnW6WIlkArpmCW0f60/GnwOVGdLssIEcRUFqjgkaU 2uw/Wcr8LBp9RXsVL7/proGlHIAUPcqUt02WfCfHkrB1+aEja2Ly1f9OxOtX9evhWIEU drYgSWVGChpwGzQP/MCYlBCTsDX/2hsAM3CH6veQc6OQSoO6fv6Y5ybAZc0gTmURVB/1 nWMijJM9fWG1nVqK7fg/EGx3qDMP1BoVnZpN136+KIZBXRWHG56zeyAJv1GvFv0H7Z6Z mSGA== MIME-Version: 1.0 X-Received: by 10.152.27.38 with SMTP id q6mr90580lag.92.1414124408058; Thu, 23 Oct 2014 21:20:08 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.84.197 with HTTP; Thu, 23 Oct 2014 21:20:07 -0700 (PDT) In-Reply-To: <5449C81B.8080008@freebsd.org> References: <5449C81B.8080008@freebsd.org> Date: Thu, 23 Oct 2014 21:20:07 -0700 X-Google-Sender-Auth: nnY3939Gop-kkXhmr88BME7waH4 Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Craig Rodrigues To: Allan Jude Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Oct 2014 04:20:11 -0000 On Thu, Oct 23, 2014 at 8:31 PM, Allan Jude wrote: > > At the Cambridge Dev Summit, Xinuous specifically mentioned helping with > testing and writing tests. Might be good people to reach out to > Are you referring to Eric Le Blan from Xinuous who was interviewed on BSDNow ( http://www.bsdnow.tv/episodes/2014_08_20-engineering_nginx , starting at 0:16:38 )? -- Craig From owner-freebsd-testing@FreeBSD.ORG Fri Oct 24 05:36:43 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 564D6DCC; Fri, 24 Oct 2014 05:36:43 +0000 (UTC) Received: from mail-wi0-x22a.google.com (mail-wi0-x22a.google.com [IPv6:2a00:1450:400c:c05::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 70D01932; Fri, 24 Oct 2014 05:36:42 +0000 (UTC) Received: by mail-wi0-f170.google.com with SMTP id n3so332698wiv.1 for ; Thu, 23 Oct 2014 22:36:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=7+pxbGFZibruIx59JPm4doCZlK817jtqLwwip/wT9QM=; b=tcMuKDzTsWFWq8GbT08dE3SKrXwylP37kShX6LyB9CrHeMXbQ/6y6oA3cW2gv81SSu Kr131k/7p5qxekmI9fofwbDdpJtKEiva/fvVAUppqLVf3CDfWYigNvXgph0VTw+ur3mB b3souRl9t78ke8NVcIa/UpCtrBfm7IcALsvP6dlcQPID9pIgKlrd63vulsSTtF+oyAIC O0ZsYsnO/O9RAeP0t0yNhAdXSm87O7XLZw4rvIJnmeeHPQ2jRdCBVXZqSGzJFYD1dSjY gwGxvDF3o0LuFJytvalSVxRJtk5bCoaKZvNwsr07GsyhIAtN70Y3qE3zH5DJQGF3K103 dX8g== X-Received: by 10.180.205.162 with SMTP id lh2mr1688937wic.14.1414129000784; Thu, 23 Oct 2014 22:36:40 -0700 (PDT) Received: from dft-labs.eu (n1x0n-1-pt.tunnel.tserv5.lon1.ipv6.he.net. [2001:470:1f08:1f7::2]) by mx.google.com with ESMTPSA id hg14sm735728wib.24.2014.10.23.22.36.39 for (version=TLSv1.2 cipher=RC4-SHA bits=128/128); Thu, 23 Oct 2014 22:36:39 -0700 (PDT) Date: Fri, 24 Oct 2014 07:36:36 +0200 From: Mateusz Guzik To: Craig Rodrigues Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins Message-ID: <20141024053636.GH11222@dft-labs.eu> References: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Oct 2014 05:36:43 -0000 On Sun, Oct 12, 2014 at 11:14:45PM -0700, Craig Rodrigues wrote: > Hi, > > I have created this Jenkins job, which you can see a graphical > representation of: > > https://jenkins.freebsd.org/jenkins/view/FreeBSD_src_stable/job/FreeBSD_stable_10/848/BuildGraph/ > It is noted below stuff is being done this for current as well and in that light: > (1) does a buildworld/buildkernel on amd64 when someone checks new > code into the stable/10 branch Is not this excessive? I would suggest every N commits or M hours if there were not enough commits to reach given threshold. Of course no work in there were no commits whatsoever, but then do the work after the first which gets in. > (2) Creates a bootable UFS image with makefs any chance zfs will be used as well? > (3) Boots the image under bhyve > (4) Runs these commands inside the bhyve VM: > > cd /usr/tests > kyua test > kyua report-junit --output=test-output.xml would be nice to run some kind of stress testing. buildworld with a high -j is an example of a general purpose test. This could be done with different frequency than regular tests. > > (5) Shuts down the bhyve VM Do you have crashdumps configured in case stuff goes wrong? > (6) imports test-output.xml into Jenkins. > > You can see a full test report here: > > https://jenkins.freebsd.org/jenkins/job/FreeBSD_stable_10-tests/4/testReport/ > > We already do the same thing for CURRENT. > > Hopefully by running the tests regularly, we can help improve the quality > of FreeBSD. > > > -- > Craig > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" -- Mateusz Guzik From owner-freebsd-testing@FreeBSD.ORG Fri Oct 24 06:06:32 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 476A76B7; Fri, 24 Oct 2014 06:06:32 +0000 (UTC) Received: from mail-yh0-x235.google.com (mail-yh0-x235.google.com [IPv6:2607:f8b0:4002:c01::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D239DBF6; Fri, 24 Oct 2014 06:06:31 +0000 (UTC) Received: by mail-yh0-f53.google.com with SMTP id z6so2969534yhz.12 for ; Thu, 23 Oct 2014 23:06:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=/fyOPRnYYCIbhxYMFn8WZMuaLqU4ZjEzzlTJRglH0go=; b=wGMvK3AYzPLenHX6lV4ZTx8TgweK0+XZ/vLRSfQWCUiZlEEMB1CaHYYQV00F5md5iQ 2I9buxIeXEqZYBmYmfIEH4fOJDWXX7aCSk2sbezLEt59Pb1DNR5ImFtiJQgCKS/CZtkY iIr6R0xUOES3DUchUH1HdQLZ3+YkfzuJsSlQC8xxRrDhioyfXHEvw0ykELrTW6lJutoj Ez2cGkUUhqn2Hvn+Cc36SVmMzNVG9AFiM7Tyb1mhHHhGatbjdclZvyTUrF57kubHKs3h dMRoRw9KDjR33XV4hrNWqSU8j8yf8pKgq6wat1PPg4F19X5FKpQiwKw+QJhOtS0a0web ic0A== MIME-Version: 1.0 X-Received: by 10.170.74.85 with SMTP id q82mr3009680ykq.119.1414130790985; Thu, 23 Oct 2014 23:06:30 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Thu, 23 Oct 2014 23:06:30 -0700 (PDT) In-Reply-To: <20141024053636.GH11222@dft-labs.eu> References: <20141024053636.GH11222@dft-labs.eu> Date: Thu, 23 Oct 2014 23:06:30 -0700 X-Google-Sender-Auth: oQccbWzenSL947pHPPmkGKERUgE Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: "K. Macy" To: Mateusz Guzik Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Oct 2014 06:06:32 -0000 >> (2) Creates a bootable UFS image with makefs > > any chance zfs will be used as well? > Seconded. There are residual locking issues issues in ZFS. Particularly in the less exercised areas. >> (5) Shuts down the bhyve VM > > Do you have crashdumps configured in case stuff goes wrong? Along those same lines, do you have any sort of watchdog in case it deadlocks? -K From owner-freebsd-testing@FreeBSD.ORG Fri Oct 24 08:12:23 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5D6B31D0; Fri, 24 Oct 2014 08:12:23 +0000 (UTC) Received: from zhost.uxsglobal.com (zhost.xinuos.com [144.202.236.63]) by mx1.freebsd.org (Postfix) with ESMTP id 2CE259F3; Fri, 24 Oct 2014 08:12:22 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zhost.uxsglobal.com (Postfix) with ESMTP id B32338C035B; Fri, 24 Oct 2014 04:12:16 -0400 (EDT) Received: from zhost.uxsglobal.com ([127.0.0.1]) by localhost (zhost.uxsglobalhost.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id f_JEp-p0yZZa; Fri, 24 Oct 2014 04:12:16 -0400 (EDT) Received: from zhost.uxsglobal.com (localhost [127.0.0.1]) by zhost.uxsglobal.com (Postfix) with ESMTP id 8405C8C035A; Fri, 24 Oct 2014 04:12:16 -0400 (EDT) Received: from 64ZN53J (unknown [212.40.155.50]) by zhost.uxsglobal.com (Postfix) with ESMTPSA id EA7978C035C; Fri, 24 Oct 2014 04:12:06 -0400 (EDT) Reply-To: From: "Eric le Blan" To: References: <5449C81B.8080008@freebsd.org> <5449DB7C.9080107@xinuos.com> In-Reply-To: <5449DB7C.9080107@xinuos.com> Subject: RE: Re: Automatically running /usr/tests on stable/10 branch under Jenkins Date: Fri, 24 Oct 2014 11:11:59 +0300 Organization: Xinuos, Inc. Message-ID: <06ec01cfef62$3776a6b0$a663f410$@Xinuos.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 15.0 Thread-Index: AQHziTEWsZrWVIPIV8vYqc5DIi1UpgGVS+E/m+sFWRA= Content-Language: en-gb Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: 'John Wolfe' , freebsd-virtualization@freebsd.org, freebsd-testing@freebsd.org, 'Susan Stanziano' , 'Cheryl Blain' , 'Andy Nagle' X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Oct 2014 08:12:23 -0000 Allan, Always happy to discuss testing. I believe that Susan Stanziono of Xinuos system test team already spoke to some of you. Please do not hesitate to reach her for any specific needs. Also, Cheryl Blain - Xinuos VP of operations - will be at the MeetBSD vendor summit on Nov 3-4 and can reach to Craig. Keep well. Eric Interim CTO * email: eric@xinuos.com -------- Original Message -------- Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins Date: Thu, 23 Oct 2014 23:31:39 -0400 From: Allan Jude To: freebsd-virtualization@freebsd.org , freebsd-testing@freebsd.org , freebsd-stable@freebsd.org On 2014-10-23 18:30, Craig Rodrigues wrote: > Garrett Cooper wrote: > >> >> Hi Craig! >> As much as everyone would like to take i386 out to pasture, >> there's a large degree of value in running i386 tests on 11-CURRENT and >> 10-STABLE (I've caught some interesting build bugs and test bugs by running >> on my i386/CURRENT VM). Are there any plans to have i386 executors running >> tests anytime soon (does bhyve support i386?)? >> > > > We (jenkins-admin@freebsd.org ) have been busy the 5 months since BSDCan > 2014 as > you can see by our status report, which lists what we have done, and what > our future plans are: > > https://www.freebsd.org/news/status/report-2014-07-2014-09.html#Jenkins-Cont inuous-Integration-for-FreeBSD > > Integrating Jenkins with Kyua on amd64 was a major milestone which was > achieved, and > was even mentioned on the Jenkins web site: > > http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing > > Now, where to take this further (such as i386) is an interesting question. > > Personally, I would like to see: > -> integration of automated kyua testing with the FreeBSD release > engineering process > -> more involvement from FreeBSD developers, and even companies such as > EMC/Isilon, > who can: > -> write tests > -> suggest and implement new tests (network, storage, VM, etc.) > -> help with devops maintenance of the existing > jenkins.freebsd.org cluster > -> improve bhyve support in libvirt > -> set up their own Jenkins build environments outside of > FreeBSD, and help test things in their own private environments > > I've had requests for; > -> running tests in bhyve VM's with very small memory footprints > (Adrian Chadd) > -> running tests in MIPS environment (Adrian Chadd) > -> running Java JDK tests (Kip Macy) > -> i386 builds (Garrett Cooper) > > There are a lot of different directions to take this, but without more > people (and companies) pitching in and helping out, > progress is limited by bodies working on the stuff. > > I will be at the MeetBSD Vendor summit (not the conference) Nov. 3-4, so > hopefully > anyone interested in pushing this stuff forward can talk to me there. > > -- > Craig > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > At the Cambridge Dev Summit, Xinuous specifically mentioned helping with testing and writing tests. Might be good people to reach out to -- Allan Jude From owner-freebsd-testing@FreeBSD.ORG Fri Oct 24 11:09:08 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 47B24833; Fri, 24 Oct 2014 11:09:08 +0000 (UTC) Received: from mail-lb0-x22d.google.com (mail-lb0-x22d.google.com [IPv6:2a00:1450:4010:c04::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 65E2BC1; Fri, 24 Oct 2014 11:09:07 +0000 (UTC) Received: by mail-lb0-f173.google.com with SMTP id 10so2365305lbg.4 for ; Fri, 24 Oct 2014 04:09:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=KmYMihj1PwsSWtnggR8jNhB1+EOPLGi1L5jstxB713g=; b=HdUrhdPcCjK9G8i+/E4wIfI0OX1jepW1ZX94SO37pNWV5SepfWl72RaiUkTML37XrQ 5+3+1MEMz3QUf5oxUsXGbVBts43FNl0B7M2DCEZjRNNN5TYissE6c6ILpY/Dxsim/q6b PaxnZob2DuLy2/aKM/XKsAQSifYswglbXyZ5MImKwNdAVokqpGGWzOTimYtMQPakPfog cmfEFV26BTtl4x+JT+WKNJ0NfcfEtIGG/8y47ustUyMNZa3FEOvEepxV+DANwKVkSx6I u2Jb/WkdddZ5KvRDOkX/DzpD4/jD7S9t04X77kal0LOZAEUZcu8JJEWjp2y+qT8d6GOp yZ6Q== MIME-Version: 1.0 X-Received: by 10.152.5.169 with SMTP id t9mr2258113lat.90.1414148945077; Fri, 24 Oct 2014 04:09:05 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.84.197 with HTTP; Fri, 24 Oct 2014 04:09:04 -0700 (PDT) In-Reply-To: <06ec01cfef62$3776a6b0$a663f410$@Xinuos.com> References: <5449C81B.8080008@freebsd.org> <5449DB7C.9080107@xinuos.com> <06ec01cfef62$3776a6b0$a663f410$@Xinuos.com> Date: Fri, 24 Oct 2014 04:09:04 -0700 X-Google-Sender-Auth: TjofZ6_tWhX3p8SFupBERwHvn9c Message-ID: Subject: Re: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Craig Rodrigues To: Eric@xinuos.com Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: John Wolfe , "freebsd-virtualization@freebsd.org" , "freebsd-testing@freebsd.org" , Susan Stanziano , Allan Jude , Cheryl Blain , Andy Nagle X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Oct 2014 11:09:08 -0000 On Fri, Oct 24, 2014 at 1:11 AM, Eric le Blan wrote: > Allan, > > > > Always happy to discuss testing. I believe that Susan Stanziono of Xinuos > system test team already spoke to some of you. Please do not hesitate to > reach her for any specific needs. Also, Cheryl Blain - Xinuos VP of > operations - will be at the MeetBSD vendor summit on Nov 3-4 and can reach > to Craig. It would be helpful if someone (or some people) from Xinuous could do the following: (1) Subscribe to freebsd-testing@freebsd.org at https://lists.freebsd.org/mailman/listinfo/freebsd-testing to start monitoring the discussions on that list. (2) Review this announcement: http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing (3) Review these slides: http://www.slideshare.net/CraigRodrigues1/kyua-jenkins (4) Talk to me at MeetBSD Vendor Summit I have not been given a long timeslot to give a presentation, so reviewing those links before MeetBSD would be helpful. Long term, any testing of FreeBSD will only succeed if there is an ecosystem of people working together and sharing common scripts and information for testing FreeBSD. Any help that Xinuous (or any other company) can provide would be welcome. -- Craig From owner-freebsd-testing@FreeBSD.ORG Fri Oct 24 20:37:34 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2B15973F; Fri, 24 Oct 2014 20:37:34 +0000 (UTC) Received: from zhost.uxsglobal.com (zhost.xinuos.com [144.202.236.63]) by mx1.freebsd.org (Postfix) with ESMTP id E8E3BAC5; Fri, 24 Oct 2014 20:37:33 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by zhost.uxsglobal.com (Postfix) with ESMTP id 8B5D68C025F; Fri, 24 Oct 2014 16:37:32 -0400 (EDT) Received: from zhost.uxsglobal.com ([127.0.0.1]) by localhost (zhost.uxsglobalhost.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id jLEg7D3c1wW4; Fri, 24 Oct 2014 16:37:32 -0400 (EDT) Received: from zhost.uxsglobal.com (localhost [127.0.0.1]) by zhost.uxsglobal.com (Postfix) with ESMTP id 63F048C0354; Fri, 24 Oct 2014 16:37:32 -0400 (EDT) Received: from zhost.uxsglobalhost.com (localhost [127.0.0.1]) by zhost.uxsglobal.com (Postfix) with ESMTP id 529058C025F; Fri, 24 Oct 2014 16:37:32 -0400 (EDT) Date: Fri, 24 Oct 2014 16:37:32 -0400 (EDT) From: Susan Stanziano To: Craig Rodrigues Message-ID: <886660362.1106176.1414183052149.JavaMail.zimbra@xinuos.com> In-Reply-To: References: <5449C81B.8080008@freebsd.org> <5449DB7C.9080107@xinuos.com> <06ec01cfef62$3776a6b0$a663f410$@Xinuos.com> Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins MIME-Version: 1.0 X-Originating-IP: [96.234.37.238] X-Mailer: Zimbra 8.0.6_GA_5922 (ZimbraWebClient - GC38 (Win)/8.0.6_GA_5922) Thread-Topic: Automatically running /usr/tests on stable/10 branch under Jenkins Thread-Index: xzpKhEOfuv17hwVZmKLHXN9t72ikSQ== Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: John Wolfe , freebsd-virtualization@freebsd.org, freebsd-testing@freebsd.org, Eric@xinuos.com, Allan Jude , Cheryl Blain , Andy Nagle X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Oct 2014 20:37:34 -0000 Hello Craig, I have been reading through your slides and your website announcement, as requested, and have a few questions. (I have also subscribed to the mailing list as suggested). (1) The slides indicate that with a FreeBSD10 and lower OS, I should be able to build tests already included with FreeBSD. I have created a VM on FreeBSD10.0 but do not have the /etc/make.conf file necessary to build and install the tests. This may be that I am missing some set-up steps required but not covered in the slide presentation. (2) Does the test build require the use of the "Build Flow Plugin" or can the tests be run independent of the "continuous build" model? (3) Can the tests only be run under Bhyve as described in the announcement? I would be happy to exchange emails with you or skype as I investigate this work further. Regards, Susan ----- Original Message ----- From: "Craig Rodrigues" To: Eric@xinuos.com Cc: "Allan Jude" , "John Wolfe" , freebsd-virtualization@freebsd.org, freebsd-testing@freebsd.org, "Susan Stanziano" , "Cheryl Blain" , "Andy Nagle" Sent: Friday, October 24, 2014 7:09:04 AM Subject: Re: Re: Automatically running /usr/tests on stable/10 branch under Jenkins On Fri, Oct 24, 2014 at 1:11 AM, Eric le Blan < Eric@xinuos.com > wrote: Allan, Always happy to discuss testing. I believe that Susan Stanziono of Xinuos system test team already spoke to some of you. Please do not hesitate to reach her for any specific needs. Also, Cheryl Blain - Xinuos VP of operations - will be at the MeetBSD vendor summit on Nov 3-4 and can reach to Craig. It would be helpful if someone (or some people) from Xinuous could do the following: (1) Subscribe to freebsd-testing@freebsd.org at https://lists.freebsd.org/mailman/listinfo/freebsd-testing to start monitoring the discussions on that list. (2) Review this announcement: http://jenkins-ci.org/content/freebsd-project-use-jenkins-os-testing (3) Review these slides: http://www.slideshare.net/CraigRodrigues1/kyua-jenkins (4) Talk to me at MeetBSD Vendor Summit I have not been given a long timeslot to give a presentation, so reviewing those links before MeetBSD would be helpful. Long term, any testing of FreeBSD will only succeed if there is an ecosystem of people working together and sharing common scripts and information for testing FreeBSD. Any help that Xinuous (or any other company) can provide would be welcome. -- Craig From owner-freebsd-testing@FreeBSD.ORG Fri Oct 24 23:08:15 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C113A7E4 for ; Fri, 24 Oct 2014 23:08:15 +0000 (UTC) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "funkthat.com", Issuer "funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 9E3DBBBC for ; Fri, 24 Oct 2014 23:08:15 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id s9ON861g019418 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 24 Oct 2014 16:08:07 -0700 (PDT) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id s9ON86rl019417; Fri, 24 Oct 2014 16:08:06 -0700 (PDT) (envelope-from jmg) Date: Fri, 24 Oct 2014 16:08:06 -0700 From: John-Mark Gurney To: NGie Cooper Subject: Re: help on testing for FreeBSD... Message-ID: <20141024230806.GO82214@funkthat.com> References: <20141016001543.GD1852@funkthat.com> <69C71414-AE05-4AC2-B07D-23F5569C9FCA@gmail.com> <20141016003822.GE1852@funkthat.com> <20141016005248.GF1852@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20141016005248.GF1852@funkthat.com> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Fri, 24 Oct 2014 16:08:07 -0700 (PDT) Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Oct 2014 23:08:15 -0000 John-Mark Gurney wrote this message on Wed, Oct 15, 2014 at 17:52 -0700: > NGie Cooper wrote this message on Wed, Oct 15, 2014 at 17:46 -0700: > > On Wed, Oct 15, 2014 at 5:38 PM, John-Mark Gurney wrote: > > > Garrett Cooper wrote this message on Wed, Oct 15, 2014 at 17:25 -0700: > > >> Hey jmg@ (-testing CCed)! > > > > > > Sure, np... > > > > > >> On Oct 15, 2014, at 17:15, John-Mark Gurney wrote: > > >> > > >> > Hello, > > >> > > > >> > I'm working on testing parts of OpenCrypto. I have committed some work > > >> > to p4 at: > > >> > https://p4db.freebsd.org/depotTreeBrowser.cgi?FSPC=//depot/projects/opencrypto/tests/sys/opencrypto&HIDEDEL=NO > > >> > > > >> > But it requires a few files to be installed... I'm not sure where the > > >> > best location to install these are and now best to install them? > > >> > > >> I have a couple questions: > > >> - Where would the source live in the tree? > > > > > > Which source are you talking about? The python source? or the source > > > for the input data files? If you mean the later, it's where they will > > > be committed... They are taken from NIST's CAVP program unchanged... > > > I COULD possibly write a complicated rules to fetch and extract the > > > files, but decided not to... > > > > The bulk majority of the driver source (.c, .h, etc). > > The code that I'm testing is in src/sys/opencrypto (and there are other > drivers that can get tested too, but it still goes through the opencrypto > code)... Hence why I put them in tests/sys/opencrypto... If that's the > wrong location, let me know and I can move them... > > > >> - Why are they written in python? > > > > > > Because I'm too lazy to write code in C... I had already written > > > framework code in Python, and C doesn't have a standard function to > > > decode hex.. :) And error handling it much easier in python... > > > > I ask because we don't have a means for doing unittest level execution > > right now and not being written in something that's in base or > > installed automatically (perl for instance) would mean that your test > > code wouldn't be run by default on jenkins runs. > > Yeh... Though as a kld will have to be loaded, and a sysctl set as > root, there's other issues to having it automaticly run as part of > jenkins... Hello, any help w/ this? I haven't received a response on how to integrate my code.. If I don't get help w/ this soon, I will add it to src/tools/regression, and call it a day... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-testing@FreeBSD.ORG Fri Oct 24 23:18:40 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E3DC99CA for ; Fri, 24 Oct 2014 23:18:40 +0000 (UTC) Received: from mail-qg0-f45.google.com (mail-qg0-f45.google.com [209.85.192.45]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A36DEC8C for ; Fri, 24 Oct 2014 23:18:40 +0000 (UTC) Received: by mail-qg0-f45.google.com with SMTP id q107so1733764qgd.32 for ; Fri, 24 Oct 2014 16:18:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=KOUP62hXSajIW7ejXljLzrEBMgzOGMTXBU1OYjFHc5s=; b=Uq7hisrKTHxkdCHLvAZi9Rt4QF23YP6tAu+fX00MDfFlzwDv5oq8h3OA517nwn4XrY 1eosx3RQqhXhVlU7mjMnMPRMIYsQBEtNRn8YTZybJP4RSEwWsZF9NGg5Oz0IkA5epEhQ oKMw6MHEpWsBhSKe/98hUgq2PhqQt4rVj4r5z/wAJA05B0IrwdhiQ+HM42BrtzmeXk6D LA9NyqpZXq1A2JK2oM21nzHEKv7XUF393Tanfd8DSTaQo9v7Xr3e3GvVswPPyehWQdvi T9LjA4W8mNiZqMydo/YsZRbv5NZLpkLhyQHoiOeLassMmE/n04zlYYDshl1beXOLDQF1 55Fg== X-Gm-Message-State: ALoCoQnBb8p2+F7CipPjYGJz+GLm9prGO2fJYYITWrGqLnY2U+wq5G85rMTRTnZTNikjUeaY9pYE X-Received: by 10.140.28.10 with SMTP id 10mr10185907qgy.15.1414192713565; Fri, 24 Oct 2014 16:18:33 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.75.134 with HTTP; Fri, 24 Oct 2014 16:18:13 -0700 (PDT) X-Originating-IP: [81.184.188.119] In-Reply-To: References: <41d8a2fb.75b47028@fabiankeil.de> From: Julio Merino Date: Sat, 25 Oct 2014 01:18:13 +0200 X-Google-Sender-Auth: p7EUoGNFvf1mXcaCzqUcavCPrkg Message-ID: Subject: Re: atf_check() equivalent to test individual shell functions? To: Fabian Keil Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Oct 2014 23:18:41 -0000 On Mon, Jul 28, 2014 at 2:00 AM, Julio Merino wrote: > On Sat, Jul 26, 2014 at 4:30 AM, Fabian Keil > wrote: >> What I'm looking for is something like this: >> >> | atf_check_shell_function -s return:0 -o empty -e empty zogftw_string_ends_with "abc" "c" > > That could be interesting. > > However, note that atf_check currently is just a thin wrapper over the > atf-check binary and that's the reason why you cannot feed shell > functions into it. Implementing an atf_check_shell_function would > involve rewriting atf-check's functionality in libatf-sh.subr. FYI Take a look at this email I just sent: https://groups.google.com/forum/#!topic/kyua-discuss/yhcDuchVtrs The new code I'm prototyping in that repository has atf-check functionality that can be invoked on shell functions, so it would solve your problem. (Don't start using that code just yet given that it's not even merged into the master branch, but please let me know if you'd be interested in it!) Cheers From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 03:14:43 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 26375DC4; Sat, 25 Oct 2014 03:14:43 +0000 (UTC) Received: from mail-la0-x22d.google.com (mail-la0-x22d.google.com [IPv6:2a00:1450:4010:c03::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 37A19610; Sat, 25 Oct 2014 03:14:42 +0000 (UTC) Received: by mail-la0-f45.google.com with SMTP id gm9so2107293lab.18 for ; Fri, 24 Oct 2014 20:14:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=sd7/KXri8xql4nwYIyXIAniP0SDTHcbziALz3VlKMb4=; b=MJEbW3gbe7U7RME+6KJpoN6iuQBwQVVy0RxTkcUDDIrTjHin/KntxK0t7NY0jIWZmI JkCAUw4TCSNfEnWPPitmhnuJl6DOj0p2EV8YU94HeL2EchaP7g+Dokk0edVwh6lO/j6D B0uE4P1wBZKTMKj/Aaw2seV01NOWa410STin+SAyvuZjIHWdnWB1D+G/HQCzboAgHAS2 i5J+AY18n/GzSeUq05vrDmwFJGsJLrmENTGgvNTSt/aEI/AB57lF2TRFdDkeyQmwH6/B C+OtacAggynUxb2jPwObYwciU4+HyVtVA+jim9BWmfjjhujgX66eVzvpZu2OyMw2etvM x9tg== MIME-Version: 1.0 X-Received: by 10.152.4.132 with SMTP id k4mr8316755lak.1.1414206880089; Fri, 24 Oct 2014 20:14:40 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.84.197 with HTTP; Fri, 24 Oct 2014 20:14:39 -0700 (PDT) In-Reply-To: <886660362.1106176.1414183052149.JavaMail.zimbra@xinuos.com> References: <5449C81B.8080008@freebsd.org> <5449DB7C.9080107@xinuos.com> <06ec01cfef62$3776a6b0$a663f410$@Xinuos.com> <886660362.1106176.1414183052149.JavaMail.zimbra@xinuos.com> Date: Fri, 24 Oct 2014 20:14:39 -0700 X-Google-Sender-Auth: 2x616JZDbzg5ICNmUSa5zEsjztc Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Craig Rodrigues To: Susan Stanziano Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: John Wolfe , "freebsd-virtualization@freebsd.org" , "freebsd-testing@freebsd.org" , Eric@xinuos.com, Allan Jude , Cheryl Blain , Andy Nagle X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 03:14:43 -0000 On Fri, Oct 24, 2014 at 1:37 PM, Susan Stanziano wrote: > Hello Craig, > > I have been reading through your slides and your website announcement, as > requested, and have a few questions. (I have also subscribed to the > mailing list as suggested). > > (1) The slides indicate that with a FreeBSD10 and lower OS, I should be > able to build tests already included with FreeBSD. I have created a VM on > FreeBSD10.0 but do not have the /etc/make.conf file necessary to build and > install the tests. This may be that I am missing some set-up steps > required but not covered in the slide presentation. > /etc/make.conf is not there by default, but you can create it and add a line: WITH_TESTS="yes" After you do that, you should rebuild everything and install everything as per the steps listed here: https://svnweb.freebsd.org/base/head/Makefile?revision=271168&view=markup#l70 and the tests should be visible under /usr/tests . > > (2) Does the test build require the use of the "Build Flow Plugin" or can > the tests be run independent of the "continuous build" model? > No, "continuous build" and 'Build Flow Plugin" are not required to run the tests. You should focus on building the tests and running them manually before using "continuous integration". > (3) Can the tests only be run under Bhyve as described in the announcement? > > No, you do not need Bhyve to run the tests. Feel free to ask any additional questions on freebsd-testing@freebsd.org, so that we can clarify the steps. -- Craig From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 04:09:51 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A74F06A7; Sat, 25 Oct 2014 04:09:51 +0000 (UTC) Received: from mail-la0-x230.google.com (mail-la0-x230.google.com [IPv6:2a00:1450:4010:c03::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CAED0AAC; Sat, 25 Oct 2014 04:09:50 +0000 (UTC) Received: by mail-la0-f48.google.com with SMTP id gi9so3625537lab.35 for ; Fri, 24 Oct 2014 21:09:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=qr/rXrJooVx762BNCGeZLQJlZqU3HKlQPWZ7sfLtm3c=; b=WzXiAEUVBlMrMqT/7YZNZen8mlrPCHCXS9274zlbszRKQdM3fXfRlmOEMp9ul61pge ApDZann6s3YTVVx78AyM4v0v5ezokZ/p8ajDDsaovQOXTwz1x9SPKm4yHK6x8298GsRF Y5XtdV86om+h+185/fVDgyjD9WAkM7r8Tjw2LHGWtFg0k4Yco641NOvsCDHSMg4L/1S/ Cj1ELosJxuXHoDysWwAblW1252aCVRbXxwO1Ub/DCjNnhTH8w92241nY7FzxjniplePY /xti57eDYguRh5icA9QANm50eIn/f5pHhI2amsZJZfLi3GDMb6S2K0KPFp3I3BiDFjGv 6B3A== MIME-Version: 1.0 X-Received: by 10.152.198.166 with SMTP id jd6mr7354418lac.81.1414210188680; Fri, 24 Oct 2014 21:09:48 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.84.197 with HTTP; Fri, 24 Oct 2014 21:09:48 -0700 (PDT) In-Reply-To: <20141024053636.GH11222@dft-labs.eu> References: <20141024053636.GH11222@dft-labs.eu> Date: Fri, 24 Oct 2014 21:09:48 -0700 X-Google-Sender-Auth: yE4uALWkHkllRY-YNfQm8cqtUco Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Craig Rodrigues To: Mateusz Guzik Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 04:09:51 -0000 On Thu, Oct 23, 2014 at 10:36 PM, Mateusz Guzik wrote: > > (1) does a buildworld/buildkernel on amd64 when someone checks new > > code into the stable/10 branch > > Is not this excessive? It has not been a problem. For example, when a build occurs on the HEAD in svn, if further commits come in on HEAD, we have Jenkins configured so that it will not trigger another build on HEAD until the build in progress is done. It has been working fine. If no commits > > (2) Creates a bootable UFS image with makefs > > any chance zfs will be used as well? > Sure, we can look at that as well, but as I said earlier, there need to be more bodies working on setting up builds and configurations for this to happen. > > would be nice to run some kind of stress testing. buildworld with a high > -j is an example of a general purpose test. This could be done with > different frequency than regular tests. > > Are you volunteering to write the scripts that incorporate any stress testing that you think should be done? We would welcome any contributions. > Do you have crashdumps configured in case stuff goes wrong? > > No. You can look at our scripts used to build and boot the various VM's: https://wiki.freebsd.org/Jenkins#Repositories It's all on github, so if you think you have new scripts to add, or fixes to existing scripts, you can feel free to do a github pull request to contribute. -- Craig From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 04:20:22 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 91E15B78; Sat, 25 Oct 2014 04:20:22 +0000 (UTC) Received: from mail-lb0-x236.google.com (mail-lb0-x236.google.com [IPv6:2a00:1450:4010:c04::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 88044C7A; Sat, 25 Oct 2014 04:20:21 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id f15so2308328lbj.13 for ; Fri, 24 Oct 2014 21:20:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=eMt+sDIlpJGnS5JxkXDSZT1TZb/JCTw7KbaGM6giSrs=; b=UeQM8xvCXsqRKV6SmVW5SbTXzz2XCs6ubWveIwV6nz0icQD2gVI4rkXsjecJKFD67i zpnGcQsakvW2BoUc1jfDM+TALGNRpCHbG8Kb3Hx3TtOzIa6vRqYYwn2yWZmq+iaePXZd AyxK7IhTVb3TNeqI0fPouQv8cTmzCD+oSBQK4IflUyosMwV5wBmIKacJhOvuX+MVT+qp kA6NI9sxAUcxZMruv/6hLcH5UeC/fU9XnAf7Nn0IDUarZt+X2VwgMKE3xRrAg/hDq6cO giT1zrdxlqJFEL1+/QSsBaBTzR9RUM+haRfxDBAQQzcEOMt8ZpFgwhBXpvJyqCunzER0 gQsA== MIME-Version: 1.0 X-Received: by 10.152.22.135 with SMTP id d7mr8544867laf.46.1414210819435; Fri, 24 Oct 2014 21:20:19 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.84.197 with HTTP; Fri, 24 Oct 2014 21:20:19 -0700 (PDT) In-Reply-To: References: <20141024053636.GH11222@dft-labs.eu> Date: Fri, 24 Oct 2014 21:20:19 -0700 X-Google-Sender-Auth: T2JiUvJPzpUzUjMpVLkU7vfsiW0 Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Craig Rodrigues To: "K. Macy" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 04:20:22 -0000 On Thu, Oct 23, 2014 at 11:06 PM, K. Macy wrote: > >> (2) Creates a bootable UFS image with makefs > > > > any chance zfs will be used as well? > > > > Seconded. There are residual locking issues issues in ZFS. > Particularly in the less exercised areas. > I think what would be an interesting exercise is to set up a Jenkins job to build and boot a bhyve VM with ZFS, and then run the ZFS Test Suite, ported by Alan Somers to FreeBSD: https://lists.freebsd.org/pipermail/freebsd-testing/2014-August/000503.html Are there any volunteers who would like to help set that up and get it running under Jenkins? > > Along those same lines, do you have any sort of watchdog in case it > deadlocks? > No. You can look at our scripts used to build and boot the various VM's: https://wiki.freebsd.org/Jenkins#Repositories It's all on github, so if you think you have new scripts to add, or fixes to existing scripts, you can feel free to do a github pull request to contribute. -- Craig From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 04:45:24 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 988EF1FD; Sat, 25 Oct 2014 04:45:24 +0000 (UTC) Received: from mail-pd0-x22d.google.com (mail-pd0-x22d.google.com [IPv6:2607:f8b0:400e:c02::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4F233ED4; Sat, 25 Oct 2014 04:45:24 +0000 (UTC) Received: by mail-pd0-f173.google.com with SMTP id v10so2550832pde.18 for ; Fri, 24 Oct 2014 21:45:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=sakTeGSANXio1+THlQlVOipJbwfFO0ckzcxpaeBzyjs=; b=j37WmbuziOXpvv4R8d4Rc9dgyMs3m0ee61g6tF64kqx+EhRAvzw4F35j9ZSZ+jWgsg UjEovt54uSSu+F5mDnmZZVNtBCnmrvWl9K0n2NTyS34W0T0NTR3FyZlFVf70NXSEXr+z OFs6NTLGCkdoUiNN0O5vLobRrmKmo2O2X2Y3vwkpRwAJ0vLSbPiifxCZihG+ClYXeWHC JUKT/C4DXNaXElnXVrQKrGO0NmNL1tLNeJkHixEqKrPRynz0uCUzSvtz3ACOLV3jutnx NOD35Ax+V5IWzoJ7Dp+pxO8zZmfSh4J0OysunnpNHeN6ttf8V8Y5DC/Fg0Hf+oxZYcaA HTpg== X-Received: by 10.70.11.2 with SMTP id m2mr8980923pdb.31.1414212323759; Fri, 24 Oct 2014 21:45:23 -0700 (PDT) Received: from ?IPv6:2601:8:ab80:7d6:4d3d:33a4:5ddc:37e2? ([2601:8:ab80:7d6:4d3d:33a4:5ddc:37e2]) by mx.google.com with ESMTPSA id uj7sm5210916pac.4.2014.10.24.21.45.22 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 24 Oct 2014 21:45:23 -0700 (PDT) Content-Type: multipart/signed; boundary="Apple-Mail=_2A11CEE8-A629-491F-891F-17FC136ECA72"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Garrett Cooper In-Reply-To: Date: Fri, 24 Oct 2014 21:45:21 -0700 Message-Id: <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> References: <20141024053636.GH11222@dft-labs.eu> To: Craig Rodrigues X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "K. Macy" , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 04:45:24 -0000 --Apple-Mail=_2A11CEE8-A629-491F-891F-17FC136ECA72 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Oct 24, 2014, at 21:20, Craig Rodrigues wrote: > On Thu, Oct 23, 2014 at 11:06 PM, K. Macy wrote: >=20 >>>> (2) Creates a bootable UFS image with makefs >>>=20 >>> any chance zfs will be used as well? >>>=20 >>=20 >> Seconded. There are residual locking issues issues in ZFS. >> Particularly in the less exercised areas. >>=20 >=20 > I think what would be an interesting exercise is to set up a Jenkins = job to > build > and boot a bhyve VM with ZFS, and then run > the ZFS Test Suite, ported by Alan Somers to FreeBSD: >=20 > = https://lists.freebsd.org/pipermail/freebsd-testing/2014-August/000503.htm= l >=20 >=20 > Are there any volunteers who would like to help set that up and get it > running > under Jenkins? I think getting tools/regression/zfs working first would be a better = idea (which means that ZFS developers will need to go debug/fix the = issue noted in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D191574 = ). I=92ll go ahead and commit my fixes to head from my github fork so it = runs. Alan also suggested against integrating the test suite as-is, because as = he said, "Remember, don't run these tests on a production system. They = WILL cause panics and deadlocks, and they may cause data loss too.=94 Cheers, -Garrett --Apple-Mail=_2A11CEE8-A629-491F-891F-17FC136ECA72 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUSyrhAAoJEMZr5QU6S73eQq0H/i1iKoeG+vYhyEgN5rLuis5b T7p5Vv5CLpCt0hiWcaAnKw6pRXLHS+NZpp2iaMg6El7C4E9XOAMYPrD6Lf5FSSyO 6jlBZ7NpNuTG7oo8SDe8uC3aHvEf8pw+XMNroAkLliRKrjGGkY7wsoD8IdlMTebT 6jAvMcszrEo6o8q2zi1kNZfT0y64US49htX7mVqxRVsDgvNSJMMsBlTzDigo4aGj rqlkL6WYaWus8XJr0rLXgLzFFNkb/6RsEDA258bA7lR+upordi375fhYCOJ0R6Rh C8l+EQOb18dcsh3d3WQRNF7Blt0O0lht+TL0AcnH4MZtslJLBsl/9fDzW1fLZKM= =psgR -----END PGP SIGNATURE----- --Apple-Mail=_2A11CEE8-A629-491F-891F-17FC136ECA72-- From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 05:11:15 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E3B15608; Sat, 25 Oct 2014 05:11:14 +0000 (UTC) Received: from mail-pd0-x22e.google.com (mail-pd0-x22e.google.com [IPv6:2607:f8b0:400e:c02::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9A2C711F; Sat, 25 Oct 2014 05:11:14 +0000 (UTC) Received: by mail-pd0-f174.google.com with SMTP id p10so2570141pdj.33 for ; Fri, 24 Oct 2014 22:11:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=HQGvuTqDSbBI1h/rjkmmVCseB8iLUxwQ9KzKGEcIAJI=; b=Oayd24Qyh407XMyqb0JfzKScBq7WHCOp/zLZ7rWhSVXedqIWXpW6KqKPGVty/jpPU3 /iGsrICiHGEwI5AM8QawUIZPOfuhUR32bNY3XGNq/GqYFKjb17tLjZ6pxnn0/7BeR90v EN5sp+6fP/KzAvLR00Yxk6JGzFUbzsOEukf9UFrZTUsfD6YyVk5LdM45MWsL/AUb2wgL 2Aod0mSCEjHphPMPY/p70KBhCRBrvyQg/L1LJErYzklkEjACkRWKjJyh2T5bpBJIPeB3 jVny7B/WsN01LgLrd2zKHTcxDdhGWeY7T350BCvoit2KtQwJuFxygeYRV1XKYTlN0wEb HioQ== X-Received: by 10.68.246.229 with SMTP id xz5mr25945pbc.131.1414213874088; Fri, 24 Oct 2014 22:11:14 -0700 (PDT) Received: from ?IPv6:2601:8:ab80:7d6:4d3d:33a4:5ddc:37e2? ([2601:8:ab80:7d6:4d3d:33a4:5ddc:37e2]) by mx.google.com with ESMTPSA id g2sm5218951pdk.46.2014.10.24.22.11.13 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 24 Oct 2014 22:11:13 -0700 (PDT) Content-Type: multipart/signed; boundary="Apple-Mail=_51EBB7D1-6870-4412-A765-22F9D16DF84A"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Garrett Cooper In-Reply-To: <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> Date: Fri, 24 Oct 2014 22:11:11 -0700 Message-Id: <2D7ED585-8A72-462F-9F3E-1C67C620AC72@gmail.com> References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> To: Craig Rodrigues X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "K. Macy" , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 05:11:15 -0000 --Apple-Mail=_51EBB7D1-6870-4412-A765-22F9D16DF84A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Oct 24, 2014, at 21:45, Garrett Cooper wrote: > On Oct 24, 2014, at 21:20, Craig Rodrigues = wrote: >=20 >> On Thu, Oct 23, 2014 at 11:06 PM, K. Macy wrote: >>=20 >>>>> (2) Creates a bootable UFS image with makefs >>>>=20 >>>> any chance zfs will be used as well? >>>>=20 >>>=20 >>> Seconded. There are residual locking issues issues in ZFS. >>> Particularly in the less exercised areas. >>>=20 >>=20 >> I think what would be an interesting exercise is to set up a Jenkins = job to >> build >> and boot a bhyve VM with ZFS, and then run >> the ZFS Test Suite, ported by Alan Somers to FreeBSD: >>=20 >> = https://lists.freebsd.org/pipermail/freebsd-testing/2014-August/000503.htm= l >>=20 >>=20 >> Are there any volunteers who would like to help set that up and get = it >> running >> under Jenkins? >=20 > I think getting tools/regression/zfs working first would be a better = idea (which means that ZFS developers will need to go debug/fix the = issue noted inhttps://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D191574 = ). I=92ll go ahead and commit my fixes to head from my github fork so it = runs. Might have helped if I had referenced the appropriate bug: = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D191573 (I=92m = basically fixing = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D191574 right now). Cheers! --Apple-Mail=_51EBB7D1-6870-4412-A765-22F9D16DF84A Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUSzDvAAoJEMZr5QU6S73eARAH/1P3uHc6LPrjW0rvbl4aTn4D e54PyOxQ/XEoy5MhauP8Wj3rg7HJn6/jcqtPldVQ2w8l0qzIbhiQOzL/Y93zknW8 yzKUAhkvd+5h2KmwxFK/u+fLAvK2UPurYTiJ1RC+UTiaA+8E+RlUH6iUjwl93JFo qp2SVDTxW7xbMq6Tmni8YKl4ZTcVlyPynRoKiVf7JJoy67oQPhOCWeTf8piaXI+x SYhgfetOvPl7ny8EnP3tHHqkg96ewpEqSs4ne0YQ4BejaBJwyQh/+H1rw2Nt6Af0 LTPv35Thnfr0tj2RBkcMaVsvNvkecKcws5SxPCatkNMRoRgFmbwJLhAMsTHi+10= =0kRo -----END PGP SIGNATURE----- --Apple-Mail=_51EBB7D1-6870-4412-A765-22F9D16DF84A-- From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 06:44:10 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C95F77D2; Sat, 25 Oct 2014 06:44:10 +0000 (UTC) Received: from elvis.mu.org (elvis.mu.org [IPv6:2001:470:1f05:b76::196]) by mx1.freebsd.org (Postfix) with ESMTP id B01CCB08; Sat, 25 Oct 2014 06:44:10 +0000 (UTC) Received: from u10-2-32-011.office.norse-data.com (unknown [50.204.88.51]) by elvis.mu.org (Postfix) with ESMTPSA id 9D56F341F86A; Fri, 24 Oct 2014 23:44:10 -0700 (PDT) Message-ID: <544B46BA.4000008@freebsd.org> Date: Fri, 24 Oct 2014 23:44:10 -0700 From: Alfred Perlstein Organization: FreeBSD User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.2.0 MIME-Version: 1.0 To: Garrett Cooper , Craig Rodrigues Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> In-Reply-To: <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 06:44:10 -0000 On 10/24/14 9:45 PM, Garrett Cooper wrote: > On Oct 24, 2014, at 21:20, Craig Rodrigues wrote: > >> On Thu, Oct 23, 2014 at 11:06 PM, K. Macy wrote: >> >>>>> (2) Creates a bootable UFS image with makefs >>>> any chance zfs will be used as well? >>>> >>> Seconded. There are residual locking issues issues in ZFS. >>> Particularly in the less exercised areas. >>> >> I think what would be an interesting exercise is to set up a Jenkins job to >> build >> and boot a bhyve VM with ZFS, and then run >> the ZFS Test Suite, ported by Alan Somers to FreeBSD: >> >> https://lists.freebsd.org/pipermail/freebsd-testing/2014-August/000503.html >> >> >> Are there any volunteers who would like to help set that up and get it >> running >> under Jenkins? > I think getting tools/regression/zfs working first would be a better idea (which means that ZFS developers will need to go debug/fix the issue noted in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191574 ). I’ll go ahead and commit my fixes to head from my github fork so it runs. > > Alan also suggested against integrating the test suite as-is, because as he said, "Remember, don't run these tests on a production system. They WILL cause panics and deadlocks, and they may cause data loss too.” > > Cheers, > -Garrett Wait, we want to sweep those bugs under the rug? What exactly is wrong with making a test harness that can very easily reproduce a known problem? The chances are that anyone will dive into it once the bug is easily reproducible. -Alfred From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 06:48:04 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D6465A25; Sat, 25 Oct 2014 06:48:04 +0000 (UTC) Received: from mail-pd0-x22b.google.com (mail-pd0-x22b.google.com [IPv6:2607:f8b0:400e:c02::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8B496B2F; Sat, 25 Oct 2014 06:48:04 +0000 (UTC) Received: by mail-pd0-f171.google.com with SMTP id r10so2660083pdi.30 for ; Fri, 24 Oct 2014 23:48:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=0W9DkqfiAld4WGNwRxVReCOvtZ/9P41LPKGt4Nci47w=; b=epVQvQOiEMDjjCGFLuY7SL5ke0xemxc8uTqDhVvyNveauSg6B9i8NJi8DnCp3U7vlU zbzCv0QK1I/jKkB70joNDOpMDkBSnR9JVznMLxbYUsqJvOXhWYVDMpN2TzgKFd5GLsK5 dhvnvmP0yXtj0Oi1ees9sCOvSyBtJNWDdXEVgO7dLKmWf8MzDFl4yqYM9PXu4KZ9AMXS 3kG+E40gcOtaHKcNgVCYUkh6+q5IifYCAkj0f6joAo6Lpr7WLo5wbHnn9lD4r/BS55Hf w7UdNjs4a+6tf8gJSeq6EcV20eBBdxKg8rKfrlN/xDsexhnnRZa/otfm/CPLDh4INnzZ FOCw== X-Received: by 10.68.213.101 with SMTP id nr5mr9344918pbc.81.1414219684005; Fri, 24 Oct 2014 23:48:04 -0700 (PDT) Received: from ?IPv6:2601:8:ab80:7d6:4d3d:33a4:5ddc:37e2? ([2601:8:ab80:7d6:4d3d:33a4:5ddc:37e2]) by mx.google.com with ESMTPSA id nq2sm5418288pdb.74.2014.10.24.23.48.03 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 24 Oct 2014 23:48:03 -0700 (PDT) Content-Type: multipart/signed; boundary="Apple-Mail=_FDB23D5C-BB5E-410D-95B3-35CE9ACBE165"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Garrett Cooper In-Reply-To: <544B46BA.4000008@freebsd.org> Date: Fri, 24 Oct 2014 23:48:01 -0700 Message-Id: References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> To: Alfred Perlstein X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 06:48:05 -0000 --Apple-Mail=_FDB23D5C-BB5E-410D-95B3-35CE9ACBE165 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Oct 24, 2014, at 23:44, Alfred Perlstein wrote: > On 10/24/14 9:45 PM, Garrett Cooper wrote: >> On Oct 24, 2014, at 21:20, Craig Rodrigues = wrote: >>=20 >>> On Thu, Oct 23, 2014 at 11:06 PM, K. Macy wrote: >>>=20 >>>>>> (2) Creates a bootable UFS image with makefs >>>>> any chance zfs will be used as well? >>>>>=20 >>>> Seconded. There are residual locking issues issues in ZFS. >>>> Particularly in the less exercised areas. >>>>=20 >>> I think what would be an interesting exercise is to set up a Jenkins = job to >>> build >>> and boot a bhyve VM with ZFS, and then run >>> the ZFS Test Suite, ported by Alan Somers to FreeBSD: >>>=20 >>> = https://lists.freebsd.org/pipermail/freebsd-testing/2014-August/000503.htm= l >>>=20 >>>=20 >>> Are there any volunteers who would like to help set that up and get = it >>> running >>> under Jenkins? >> I think getting tools/regression/zfs working first would be a better = idea (which means that ZFS developers will need to go debug/fix the = issue noted in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D191574 = ). I=92ll go ahead and commit my fixes to head from my github fork so it = runs. >>=20 >> Alan also suggested against integrating the test suite as-is, because = as he said, "Remember, don't run these tests on a production system. = They WILL cause panics and deadlocks, and they may cause data loss too.=94= >>=20 >> Cheers, >> -Garrett >=20 > Wait, we want to sweep those bugs under the rug? What exactly is = wrong with making a test harness that can very easily reproduce a known = problem? The chances are that anyone will dive into it once the bug is = easily reproducible. Sweeping bugs under the rug is not what I plan on doing; I=92m = marking these as expected failures, as opposed to having them = continually panic a machine. Once a ZFS dev takes a look at the issue = and resolves them, then the ZFS dev can remove the =93bail=94 calls I=92m = adding to the testcases. Cheers, -Garrett PS FWIW the panics are recent. As I stated in the first bug, it didn=92t = occur on 10.0-RELEASE. --Apple-Mail=_FDB23D5C-BB5E-410D-95B3-35CE9ACBE165 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUS0eiAAoJEMZr5QU6S73ejxcIALZRkCzcpT6cfWtwDIgts2rz 5zUOJHgmExFkkijOtd7m9HEFnQQcEDQ8JJ0P/s6Fvol2/gKfRCtjd51+ip2Dj8ya g67V81qOxTFomCWo9VxWMWaBtPsowP8wwG/ZL0QkKCYQlLQITNNVu8/FsKODpIbm IKja9Bk97du9oI7+1T6L0FvkRvvk7m7MJxkzI373HKTb6+hmmt7YK9jRQ7fh5efG Qy7Ce4UUAffr6V/PHqhVvxZLiZkSa69hnY9MaAKXkVRPvIgUFTFhdYTCKqw7L8Ua vJx+zflZEzCnA2a9NC0Llcf0roNsnPV/E11F5vBzhhIY1ICx2dx2Gj99298zHPE= =pm2N -----END PGP SIGNATURE----- --Apple-Mail=_FDB23D5C-BB5E-410D-95B3-35CE9ACBE165-- From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 16:49:50 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 30F1BC36; Sat, 25 Oct 2014 16:49:50 +0000 (UTC) Received: from mail-la0-x229.google.com (mail-la0-x229.google.com [IPv6:2a00:1450:4010:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1F886CA9; Sat, 25 Oct 2014 16:49:48 +0000 (UTC) Received: by mail-la0-f41.google.com with SMTP id pn19so4268604lab.0 for ; Sat, 25 Oct 2014 09:49:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=cgm9lfEHWfB6qAKCJlmjuQQ3hPSKCH27q8M4kK3lgUg=; b=t/zm/tpixCC8iLm0cepTwA62bBcKH00ujOBaTZjgFYl0Bo3gLBEwSzKg/CrjCEIWGS GiogtLPx0yMwoo2c6DZct0U0cUATsYJRJfO57owSpHlO3UMj8kJP3GZl9qXgyxRfCamL UjorKSEMtacmJRLL2upi0sTRhku8LSGxeMws02Ke0b7ZHxjat8ujAHmPaPX6lSypOII0 R1cK0S1ssTesQBpX78IwggzDqUkfjAknXFHUTfjM8wXoUnQXXAQUNiLAgulq/n9hN7dA 3sVj+i/FHxqB9SPwJhRlQC2IQcG4ZNYW6RtBZjNIGzx3Rx7RO+kvUSPNCnNPe3yRNatW OsLg== MIME-Version: 1.0 X-Received: by 10.112.57.227 with SMTP id l3mr11901372lbq.68.1414255786825; Sat, 25 Oct 2014 09:49:46 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.84.197 with HTTP; Sat, 25 Oct 2014 09:49:46 -0700 (PDT) In-Reply-To: <544B46BA.4000008@freebsd.org> References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> Date: Sat, 25 Oct 2014 09:49:46 -0700 X-Google-Sender-Auth: GIlVT8zHlo5sp3t_C8L_ce6W7_A Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Craig Rodrigues To: Alfred Perlstein Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-testing@freebsd.org" , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 16:49:50 -0000 On Fri, Oct 24, 2014 at 11:44 PM, Alfred Perlstein wrote: > > On 10/24/14 9:45 PM, Garrett Cooper wrote: > >> I think getting tools/regression/zfs working first would be a better idea >> (which means that ZFS developers will need to go debug/fix the issue noted >> in https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=191574 ). I'll go >> ahead and commit my fixes to head from my github fork so it runs. >> > If tools/regression/zfs has Kyuafiles that makes it easier to run under the kyua tool, that would greatly facilitate running under Jenkins automation, and would be useful. I notice some of the fixes you are applying to the regression/zfs tests is to make certain tests not run on FreeBSD because they cause known kernel panics such as this: https://lists.freebsd.org/pipermail/svn-src-all/2014-October/093671.html I'm not entirely convinced that this is a good way to "fix" a test. If a test causes a panic, then that's what it does, it it should not be swept under the rug, as Alfred has pointed out. Printing out a warning with a pointer to the PR just before running this type of test is OK, though. > >> Alan also suggested against integrating the test suite as-is, because as >> he said, "Remember, don't run these tests on a production system. They >> WILL cause panics and deadlocks, and they may cause data loss too." >> >> Cheers, >> -Garrett >> > > Wait, we want to sweep those bugs under the rug? What exactly is wrong > with making a test harness that can very easily reproduce a known problem? > The chances are that anyone will dive into it once the bug is easily > reproducible. > I agree with Alfred on this. Even though Alan's test suite may kernel panic or cause problems, there is still value in running it and making the results visible on jenkins.freebsd.org. Running these tests inside a VM which is generated during the build will allow these types of test to run, but still keep the test machine usable, even if the VM gets corrupted while running the tests. If we have test suites for ZFS, but no one runs them, then no one will bother to investigate and fix the bugs. Running the test suites under automation that is visible on jenkins.freebsd.org is going to force developers to see problems much sooner than they do now. Just having the tests in the tree and hoping that people run them and care to look into the problems is not enough. -- Craig From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 19:00:02 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4EDB888F; Sat, 25 Oct 2014 19:00:02 +0000 (UTC) Received: from mail-pa0-x229.google.com (mail-pa0-x229.google.com [IPv6:2607:f8b0:400e:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0FD47B34; Sat, 25 Oct 2014 19:00:02 +0000 (UTC) Received: by mail-pa0-f41.google.com with SMTP id rd3so2984995pab.28 for ; Sat, 25 Oct 2014 12:00:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=9YeyI15j5l6S03i2B0aC67DsL83fwHGcb7E4ylFCUPo=; b=ZalMN+CniOIZHtu9McLyADUAUHjlF/0WUq9mQ2ndl3u5QNC2MFjsGKz80ug2PA6fHn 960+25FVAoerVqnFg+a6w0eOEWpD0xWzFgxHfy7T5YBBpBdrmqnzsRSky5NkRhC30RdV lkhsnYF/x2Ke7fA/J5s9f6cjg0LG+XCC7oeK2dkAUCPNy2C/l/cxHrqJx+o3FDQ0MC0f t/gNKU47OB9BW6orLcqPXayH6KwkiMCGKDqLlhZAkS8wBNhRLrmJOkeO7iNyFg+rSKFl FyqtpIdreEo85mCtbfUwS+nZZ5/+GlDgch4/RnMoisTpiVyIIMisyj4KZv357zy1oLfV 5ciQ== X-Received: by 10.68.108.36 with SMTP id hh4mr13000218pbb.108.1414263601437; Sat, 25 Oct 2014 12:00:01 -0700 (PDT) Received: from ?IPv6:2601:8:ab80:7d6:bc27:a626:3844:f04a? ([2601:8:ab80:7d6:bc27:a626:3844:f04a]) by mx.google.com with ESMTPSA id pg9sm6792803pdb.71.2014.10.25.12.00.00 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 25 Oct 2014 12:00:00 -0700 (PDT) Content-Type: multipart/signed; boundary="Apple-Mail=_F769DEA7-56B3-46B3-BAFA-2F238869E838"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Panicking UUTs and integrating the ZFS test suite into the default run From: Garrett Cooper In-Reply-To: Date: Sat, 25 Oct 2014 11:59:59 -0700 Message-Id: <86E89F83-179D-4301-9EC0-0ECC3909B924@gmail.com> References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> To: Craig Rodrigues X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" , Alfred Perlstein X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 19:00:02 -0000 --Apple-Mail=_F769DEA7-56B3-46B3-BAFA-2F238869E838 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 (moving to -testing and renaming because this is turning into a bikeshed = discussion) On Oct 25, 2014, at 9:49, Craig Rodrigues wrote: > On Fri, Oct 24, 2014 at 11:44 PM, Alfred Perlstein = wrote: ... > If tools/regression/zfs has Kyuafiles that makes it easier to > run under the kyua tool, that would greatly facilitate running > under Jenkins automation, and would be useful. > I notice some of the fixes you are applying to the regression/zfs > tests is to make certain tests not run on FreeBSD because they > cause known kernel panics such as this: >=20 > = https://lists.freebsd.org/pipermail/svn-src-all/2014-October/093671.html Yes. > I'm not entirely convinced that this is a good way to "fix" a test. > If a test causes a panic, then that's what it does, it it should not = be > swept under the rug, as Alfred has pointed out. > Printing out a warning with a pointer to the PR just before running = this > type of test is OK, though. I highly disagree. Neither of us can guarantee that these tests aren=92t = being run on either production systems (this would be extremely = foolhardy, but I=92ve seen it happen before several times on the LTP = list), test systems that other developers/testers are working on where = the focus is not ZFS, or that the end-user would care about ZFS in their = test run. Continually panicking the box because of known issues is silly = and wastes developer/tester cycles. Furthermore, running the tests and = getting a subset of coverage [because the box panicked] doesn=92t help = further the goals of testing the overall system. Good developers and = testers care about coverage in addition to =93how green are my tests?=94, = =93how far is feature X to completion?=94. Idioms like the following are in my opinion sufficient for working = around issues. 6=20 7 [ "${os}" =3D "FreeBSD" ] && die "panics FreeBSD; see bug # 194589" 8 1. The issue is documented, so developers and testers know where to get = the bug information 2. The test is marked as a failure. Both of these items encourage testers and other developers to go prod = developers with knowledge and cycles to go fix the bugs so things = eventually turn green. > I agree with Alfred on this. Even though Alan's test suite > may kernel panic or cause problems, there is still value in running it > and making the results visible on jenkins.freebsd.org. > Running these tests inside a VM which is generated during the build > will allow these types of test to run, but still keep the test machine = usable, > even if the VM gets corrupted while running the tests. Again, because of some of the things that Alan said about corruption and = panicking, until the underlying issues are fixed in ZFS, there=92s = absolutely no reason why it should be run by default. As a sidenote, the FreeBSD test suite needs a couple features: 1. Additional configuration options, similar to = =93allow_sysctl_side_effects=94 used by = tests/sys/kern/kern_descrip_test.c (say =93causes_corruption=94 or = =93might_panic=94). If a. These safety belts existed b. Were off by default c. Could be queried via all of the handlers (ATF, TAP, Plain) = via some API (atf_config, etc). then I wouldn't have an issue with someone integrating them in to = the Jenkins runs without the bail outs. 1. and 2. are easy to take care = of, but 3. is missing in all but the ATF handler. 2. A hook to say =93this feature/module is present=94, e.g. =93this test = requires zfs, so if it=92s not kldloaded/statically compiled into the = kernel, don=92t run this test=94 > If we have test suites for ZFS, but no one runs them, then no one will > bother to investigate and fix the bugs. I understand but someone needs to go fix the bugs too before stuff is = turned on by default, otherwise that person will bear the brunt of the = support issues that come up with Jane/Joe user decides to go and run the = test suite and it goes and blows up their machine. > Running the test suites under automation that is visible on = jenkins.freebsd.org is going to force developers to see problems > much sooner than they do now. Just having the tests in the tree > and hoping that people run them and care to look into the problems is > not enough. Cheers. --Apple-Mail=_F769DEA7-56B3-46B3-BAFA-2F238869E838 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUS/MvAAoJEMZr5QU6S73eY1oH/2o0ujMD4+18uW8SInjf6K/D O/PysbLhkcrNKW4Qg7HKOioFjC+EDgx7u/u5+kKnECHRxEf1P5b8Ezzu4Mv7Bm4J OWvONXhS0RDki6eRgGJuVgGzBORdWvgm2xfp6h0Apga7E9yGI9ZvQqtH4hWJ6O88 85Y+PlJLp5obElOG5/y4KAWHnLinI17M6yBv2xXSNvOwMqnsw16xeCv6xIAfkjdO ZaxPCNxrZtIbDHe0DinbO+ajpOa6B5gkhFw5YykRouEiItisR6KlyjiBI/iBSxOG whXgycurIKa6xVChsxJs+NqFUNT+JtWm3BABQlB4B5VtW06Khs5cf6Ho6KxA/+c= =FO8v -----END PGP SIGNATURE----- --Apple-Mail=_F769DEA7-56B3-46B3-BAFA-2F238869E838-- From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 20:20:07 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7E16FD6B; Sat, 25 Oct 2014 20:20:07 +0000 (UTC) Received: from mail-yh0-x236.google.com (mail-yh0-x236.google.com [IPv6:2607:f8b0:4002:c01::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 104C3366; Sat, 25 Oct 2014 20:20:07 +0000 (UTC) Received: by mail-yh0-f54.google.com with SMTP id 29so3165889yhl.13 for ; Sat, 25 Oct 2014 13:20:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=tynCnMKKC0eHZKShqxl5XFhNw74YLNxljWlRLqIOxA4=; b=Pub3QTq2WHLOKa8SdiQLMVF4SbTO2qnYUbR4XG+2yQllUwJPdQbUCWpRbuFy8nd9s8 r8pwHdCwy4EzGeMTtXHkwS7L9TqxbdbXcRNskwWWpPzDxCgil7raUIrZvitMCPnBJphI Y1aY/Zi3+hdvH1rNCXGNYqff8v+MHVALWchY3YxImcLw/VjXavNp1PD56P1vXrpdIR/Z fNNMe+v+M9AwqgBmfgQ9X1246rzuf5eDr+tEAGC/wTNiDB3kXn+PYKvfnmE6ht4LgoqI R6UiXjiXhNkLA2IPvRR1Lih9V4YEYtUgdDH/EuOz9rXOeEbbIGkRh4Xq4xXKIVvCB19C ksfw== MIME-Version: 1.0 X-Received: by 10.170.113.214 with SMTP id f205mr15326025ykb.10.1414268405509; Sat, 25 Oct 2014 13:20:05 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Sat, 25 Oct 2014 13:20:05 -0700 (PDT) In-Reply-To: References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> Date: Sat, 25 Oct 2014 13:20:05 -0700 X-Google-Sender-Auth: Tz1pjYcN82aDbop02VCfh_GmGLU Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: "K. Macy" To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" , Alfred Perlstein , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 20:20:07 -0000 >>> Alan also suggested against integrating the test suite as-is, because a= s he said, "Remember, don't run these tests on a production system. They W= ILL cause panics and deadlocks, and they may cause data loss too.=E2=80=9D >>> >>> Cheers, >>> -Garrett >> >> Wait, we want to sweep those bugs under the rug? What exactly is wrong = with making a test harness that can very easily reproduce a known problem? = The chances are that anyone will dive into it once the bug is easily repro= ducible. > > Sweeping bugs under the rug is not what I plan on doing; I=E2=80= =99m marking these as expected failures, as opposed to having them continua= lly panic a machine. Once a ZFS dev takes a look at the issue and resolves = them, then the ZFS dev can remove the =E2=80=9Cbail=E2=80=9D calls I=E2=80= =99m adding to the testcases. > Cheers, > -Garrett Yes, disabling tests that fail leads to an ineffectual test suite. A test suite that never has any failures is not very useful. However, there are two factors to take in to account in this context: a) frequent failures can lead users to stop running a test suite leading to further regressions b) long-term repeated failures can desensitize users leading them to ignore *new* failures facilitating further regressions Thus it's really a question of what context you're talking about running the test suite in. For purposes of Jenkins we want full visibility in to what is passing and what is failing and how long this has been going on for. Cheers. -K From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 20:46:08 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 256C38D1; Sat, 25 Oct 2014 20:46:08 +0000 (UTC) Received: from mail-wg0-x22c.google.com (mail-wg0-x22c.google.com [IPv6:2a00:1450:400c:c00::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1AE708A5; Sat, 25 Oct 2014 20:46:06 +0000 (UTC) Received: by mail-wg0-f44.google.com with SMTP id y10so3165082wgg.15 for ; Sat, 25 Oct 2014 13:46:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:sender:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=t//pXgEd3Q3UH0iDU41IKFjnnABq5Sm5aPP3WwoOZYY=; b=SVH1blnEsqAYqr3PugLeWMACKsu9FoSXunMO38A0GhtfKZv0/sOTh3ATCkCS9jCQlD DAtFkj8tZBZefRlYGdMmO8HEeASGKxmgLOICXsaUtSVHbLYTBFdei7R3gKKoeKDHu6nt Ml6Jk9Nu7ZACfRC+Pe9L7Rbgb6wIDxIAk0fn91G//Plugu5QCeTEQldd3dXidRp6wOzd 4FQuRRsjKOgvCGhHeLqBnOJ6HQTRSOwbWRaxPWQ2YjRSVIYZ+EJ7FUlLQk5xooiMVr0V 8a0eM26oTPzJVM/gFgFcJPOJvV/GizrxUBE5Q6qiPdBpFQ/ErX4JxJQJREl+y+8+Tygp 2gOw== MIME-Version: 1.0 X-Received: by 10.194.91.176 with SMTP id cf16mr13105133wjb.60.1414269965374; Sat, 25 Oct 2014 13:46:05 -0700 (PDT) Reply-To: matt@ixsystems.com Sender: mattjeet@gmail.com Received: by 10.194.14.40 with HTTP; Sat, 25 Oct 2014 13:46:05 -0700 (PDT) In-Reply-To: References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> Date: Sat, 25 Oct 2014 13:46:05 -0700 X-Google-Sender-Auth: d4LEDwRd3jOMoF28Wxp0hRpOoQk Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Matt Olander To: "K. Macy" Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" , Alfred Perlstein , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 20:46:08 -0000 On Sat, Oct 25, 2014 at 1:20 PM, K. Macy wrote: >>>> Alan also suggested against integrating the test suite as-is, because = as he said, "Remember, don't run these tests on a production system. They = WILL cause panics and deadlocks, and they may cause data loss too.=E2=80=9D >>>> >>>> Cheers, >>>> -Garrett >>> >>> Wait, we want to sweep those bugs under the rug? What exactly is wrong= with making a test harness that can very easily reproduce a known problem?= The chances are that anyone will dive into it once the bug is easily repr= oducible. >> >> Sweeping bugs under the rug is not what I plan on doing; I=E2=80= =99m marking these as expected failures, as opposed to having them continua= lly panic a machine. Once a ZFS dev takes a look at the issue and resolves = them, then the ZFS dev can remove the =E2=80=9Cbail=E2=80=9D calls I=E2=80= =99m adding to the testcases. >> Cheers, >> -Garrett > > Yes, disabling tests that fail leads to an ineffectual test suite. A > test suite that never has any failures is not very useful. However, > there are two factors to take in to account in this context: > a) frequent failures can lead users to stop running a test suite > leading to further regressions > b) long-term repeated failures can desensitize users leading them to > ignore *new* failures facilitating further regressions > > Thus it's really a question of what context you're talking about > running the test suite in. For purposes of Jenkins we want full > visibility in to what is passing and what is failing and how long this > has been going on for. Agreed. We're talking about doing an OpenZFS bug-tracker and maybe we could have tests post there automatically, once the bug is opened. I am going to put this on the board at MeetBSD Ca. next week, so we can have a discussion about it in real-time with some of the people present. Cheers, -matt From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 21:46:48 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 62229484; Sat, 25 Oct 2014 21:46:48 +0000 (UTC) Received: from mail-pa0-x22b.google.com (mail-pa0-x22b.google.com [IPv6:2607:f8b0:400e:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1629EDD4; Sat, 25 Oct 2014 21:46:48 +0000 (UTC) Received: by mail-pa0-f43.google.com with SMTP id eu11so3107879pac.30 for ; Sat, 25 Oct 2014 14:46:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=ROBklQZYzRnOWEg4u6+mXXSEkyoA39Nb6IiVCDo1Xxw=; b=TxhmpWmG3d4QN+IylTmBpQ4iKoZbx8OXhNghtQONHkI+RGzBN2aOzpdlTaPeIua/zr rCd1a80riRlgYFrpHlJ5FuvMCZbPrL6oWUmajDRwVFMUWEILl01baRhHrwaFj6KljQmQ iq5i6PImVvu/vaOv7JJo5bgxDUqKGipAU9HAC/poO7y0DL5yt4Bbs1wvk7gkvLm1Hm0J S/4c5derxn850sFBlqrbnTE42zGlBGDkmbDmZEP43OBvkVdoHFNHBAb219BzbA7TaMP4 ofxFP139tjrcShC8epy6Vrn43rKU1K8lw66K+m9wp9uxbK7ePFV2Wfxd/4UmqlvwTYvW sZtw== X-Received: by 10.70.19.101 with SMTP id d5mr13514906pde.79.1414273607611; Sat, 25 Oct 2014 14:46:47 -0700 (PDT) Received: from [192.168.20.5] (c-98-247-240-204.hsd1.wa.comcast.net. [98.247.240.204]) by mx.google.com with ESMTPSA id jq5sm6960728pbc.32.2014.10.25.14.46.46 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 25 Oct 2014 14:46:46 -0700 (PDT) Content-Type: multipart/signed; boundary="Apple-Mail=_B288C26C-0C84-4CCF-A5EC-92AEB18549DC"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: Garrett Cooper In-Reply-To: Date: Sat, 25 Oct 2014 14:46:46 -0700 Message-Id: <74FE3F75-43D2-4CFC-8B0F-56EF886F4748@gmail.com> References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> To: "K. Macy" X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" , Alfred Perlstein , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 21:46:48 -0000 --Apple-Mail=_B288C26C-0C84-4CCF-A5EC-92AEB18549DC Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Oct 25, 2014, at 13:20, K. Macy wrote: >>>> Alan also suggested against integrating the test suite as-is, = because as he said, "Remember, don't run these tests on a production = system. They WILL cause panics and deadlocks, and they may cause data = loss too.=94 >>>>=20 >>>> Cheers, >>>> -Garrett >>>=20 >>> Wait, we want to sweep those bugs under the rug? What exactly is = wrong with making a test harness that can very easily reproduce a known = problem? The chances are that anyone will dive into it once the bug is = easily reproducible. >>=20 >> Sweeping bugs under the rug is not what I plan on doing; I=92m = marking these as expected failures, as opposed to having them = continually panic a machine. Once a ZFS dev takes a look at the issue = and resolves them, then the ZFS dev can remove the =93bail=94 calls I=92m = adding to the testcases. >=20 > Yes, disabling tests that fail leads to an ineffectual test suite. A > test suite that never has any failures is not very useful. However, > there are two factors to take in to account in this context: > a) frequent failures can lead users to stop running a test suite > leading to further regressions > b) long-term repeated failures can desensitize users leading them to > ignore *new* failures facilitating further regressions >=20 > Thus it's really a question of what context you're talking about > running the test suite in. For purposes of Jenkins we want full > visibility in to what is passing and what is failing and how long this > has been going on for. (seeing as how my other post isn=92t in the -testing archives yet..) Panicking a node (what the tests are doing before last night) = and exiting with a non-zero exit code (what I=92m making them do with = the bail outs in tools/regression/zfs) are both considered test = failures. The difference being that I can safely run all of the tests on = a production or a test machine without having to panic/reboot the box = and I get greater coverage in one fell swoop. If a developer wants they = can always delete the lines that bail out of the tests to get the = desired panic. Cheers, -Garrett --Apple-Mail=_B288C26C-0C84-4CCF-A5EC-92AEB18549DC Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJUTBpGAAoJEMZr5QU6S73eroQH/0EUjiiTQEnkaDgG29GxsSP4 pmerYl9PXlSJiN1t0ApDz7HxxWh0qTsV8+5OIdrWOU4gvmK3DEaq+HOMqVboVtS5 WLFy+/bljFhnEbF5b3UqPUyNFNaCKV7Rdr96oLnA9NVJRvEo6aNHF0+rZKyTDeMd Qb7YDscluRqyqFCugb3rcatFWVHLycQKZdiUSx5Mdc7PnXiJ4nK9nK9nq6gU3lAA u9F3wKFZLHuzI9Ko2MEUp9jqwoK8kL2wx3Q5YBvyCUUI7OmoIf2GNhhxOZwa4Tga lfrTGH4gOcRLzjGmOGCOqXLAG3j1JpPetzJywdYjfZIVu+iEp+FmX4vH6WjF8nw= =a/Nc -----END PGP SIGNATURE----- --Apple-Mail=_B288C26C-0C84-4CCF-A5EC-92AEB18549DC-- From owner-freebsd-testing@FreeBSD.ORG Sat Oct 25 21:55:30 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 49A4D678; Sat, 25 Oct 2014 21:55:30 +0000 (UTC) Received: from mail-yh0-x231.google.com (mail-yh0-x231.google.com [IPv6:2607:f8b0:4002:c01::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CC162EB6; Sat, 25 Oct 2014 21:55:29 +0000 (UTC) Received: by mail-yh0-f49.google.com with SMTP id a41so2850484yho.22 for ; Sat, 25 Oct 2014 14:55:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=kKt6BmrW6I+rLbcDwNkEKwLjVTlpuwf/2Jn9S9xWo14=; b=gvhr0GrDcrOeHTC5nx8pgldKNvSbME/5jkI0kLJkw+pgl/cRnB0k84cK9wV+q56YbJ VG+woGlm/AS5K7No/+0DJPXINLGMUjgDc/4gJ6LaELUcskKepR5yvzetI1yGFu0F4+uO 3jezqTfumx1vwcoS0N/zhpI4fGNsHoBkXXMXigKHe1ZIb2TPyABVzf2nOlpUmMn5Zckn Kj/JRjkXKj182CHGbei9aYR3665sQdve/f4MPMbf5/E2REb+efnTPrGA3+D73rIUZImO RIjCnr39Dex8uCuCmWLyxTR3p30aBEmWJ18u4dlGZQXtp9rIrwShqqj2n3RMEP5uf50j gvfQ== MIME-Version: 1.0 X-Received: by 10.236.209.101 with SMTP id r65mr3468882yho.140.1414274129002; Sat, 25 Oct 2014 14:55:29 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.170.82.197 with HTTP; Sat, 25 Oct 2014 14:55:28 -0700 (PDT) In-Reply-To: <74FE3F75-43D2-4CFC-8B0F-56EF886F4748@gmail.com> References: <20141024053636.GH11222@dft-labs.eu> <81030948-E60F-4AAD-AAF1-16349607917D@gmail.com> <544B46BA.4000008@freebsd.org> <74FE3F75-43D2-4CFC-8B0F-56EF886F4748@gmail.com> Date: Sat, 25 Oct 2014 14:55:28 -0700 X-Google-Sender-Auth: t0VcMeHAoagDRAA2pnmRsZr5KUA Message-ID: Subject: Re: Automatically running /usr/tests on stable/10 branch under Jenkins From: "K. Macy" To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" , Alfred Perlstein , FreeBSD stable , "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Oct 2014 21:55:30 -0000 On Sat, Oct 25, 2014 at 2:46 PM, Garrett Cooper wro= te: > On Oct 25, 2014, at 13:20, K. Macy wrote: > >>>>> Alan also suggested against integrating the test suite as-is, because= as he said, "Remember, don't run these tests on a production system. They= WILL cause panics and deadlocks, and they may cause data loss too.=E2=80= =9D >>>>> >>>>> Cheers, >>>>> -Garrett >>>> >>>> Wait, we want to sweep those bugs under the rug? What exactly is wron= g with making a test harness that can very easily reproduce a known problem= ? The chances are that anyone will dive into it once the bug is easily rep= roducible. >>> >>> Sweeping bugs under the rug is not what I plan on doing; I=E2=80= =99m marking these as expected failures, as opposed to having them continua= lly panic a machine. Once a ZFS dev takes a look at the issue and resolves = them, then the ZFS dev can remove the =E2=80=9Cbail=E2=80=9D calls I=E2=80= =99m adding to the testcases. >> >> Yes, disabling tests that fail leads to an ineffectual test suite. A >> test suite that never has any failures is not very useful. However, >> there are two factors to take in to account in this context: >> a) frequent failures can lead users to stop running a test suite >> leading to further regressions >> b) long-term repeated failures can desensitize users leading them to >> ignore *new* failures facilitating further regressions >> >> Thus it's really a question of what context you're talking about >> running the test suite in. For purposes of Jenkins we want full >> visibility in to what is passing and what is failing and how long this >> has been going on for. > > (seeing as how my other post isn=E2=80=99t in the -testing archives yet..= ) > Panicking a node (what the tests are doing before last night) and= exiting with a non-zero exit code (what I=E2=80=99m making them do with th= e bail outs in tools/regression/zfs) are both considered test failures. The= difference being that I can safely run all of the tests on a production or= a test machine without having to panic/reboot the box and I get greater co= verage in one fell swoop. If a developer wants they can always delete the l= ines that bail out of the tests to get the desired panic. I don't think there is a RIGHT answer. I just ask that it be noisy about the choice that's being made. In other words, it should be very explicit that it's running a "safe" subset of the tests because FreeBSD's ZFS can't actually pass all of them, and that there be a flag to enable those of us who want to see the failure to see it without modifying any scripts or config files. A test framework that requires me to muck with settings to run failing tests isn't living up to its full potential. -K