From owner-freebsd-testing@FreeBSD.ORG Wed Jun 3 19:19:05 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0E0E1551; Wed, 3 Jun 2015 19:19:05 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pa0-x22d.google.com (mail-pa0-x22d.google.com [IPv6:2607:f8b0:400e:c03::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D0AAF1EDA; Wed, 3 Jun 2015 19:19:04 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: by payr10 with SMTP id r10so13284589pay.1; Wed, 03 Jun 2015 12:19:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=jwvci3Wn25Dg1gclzjwOj+PpKTaocPogM3xDrBqyHyA=; b=Td/lHUp788RmpkSenRYlsRZ5LxgIbSmemUpoBeNT/ab0y0BzG/nbDy24YNeYfEpIuy l43ip+vPwP2gquYm5dWLZuhIl5+sw7Q67E2r19XJMqUeZgC1xIvoFHrohlyNbrdkPoBu XaBbGmOhssvZ6FiRkKBDtPKPe+MYQUcEmG25xWuYqQdKBM2ouEWV5+rRM9Y72J3xTuNB sD3MZkBZObK8HE3lNOicT+yqBf9MnB+hUfQ4+ETtsfyqgpsIN5DQE0QP3+dTsC2OUvnn sgGqPAgVbWEvK9hCIHizx7ZrZjFz+KQRnvv71Zs3/qgIERx1sNB9UoC23Wn0uIp5e7Am Sviw== X-Received: by 10.70.129.17 with SMTP id ns17mr62779579pdb.12.1433359144344; Wed, 03 Jun 2015 12:19:04 -0700 (PDT) Received: from [33.161.27.125] ([172.56.32.97]) by mx.google.com with ESMTPSA id qy7sm1529742pbb.12.2015.06.03.12.19.03 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 03 Jun 2015 12:19:03 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: Incorporating the Capsicum test suite From: Garrett Cooper X-Mailer: iPhone Mail (12F70) In-Reply-To: Date: Wed, 3 Jun 2015 12:19:02 -0700 Cc: Ed Maste , "freebsd-testing@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <0582DC06-4021-443D-B693-93F848654441@gmail.com> References: To: Craig Rodrigues X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 19:19:05 -0000 > On Jun 3, 2015, at 08:30, Craig Rodrigues wrote: >=20 >> On Tue, Jun 2, 2015 at 1:40 PM, Ed Maste wrote: >>=20 >>=20 >> I have a couple of questions. First, do we have an existing pattern >> for connecting googletest tests to Kyua? >=20 > No we do not. Googletest looks like it is basically trying to do > the same thing as ATF or CUnit. >=20 > Googletest has a way to generate JUnit XML test output: > https://code.google.com/p/googletest/wiki/AdvancedGuide#Generating_an_XML_= Report >=20 > The easiest thing to do would be to configure the Capsicum tests to > generate this XML test output. This can be imported natively into > Jenkins, which can then parse the test report. > This would require no changes to the Capsicum tests. >=20 > If you want to integrate googletest with kyua, then things get tricky. > On the googletest page: https://code.google.com/p/googletest/ > they list a third party plugin which can take googletest and generate TAP > output. > I don't know how well that works. It would be nice if that functionality > was > integrated in googletest natively. >=20 > Another option would be to modify kyua, so that it can parse the output of= > googletest > natively. This would be quite nice. I haven't found extending kyua to be= > very > easy....it requires good knowledge of C++ and Lua. Also, whoever > contributes > code to kyua must sign a Contributor License Agreement (CLA) with Google. > That's something to keep in mind. >=20 > Another option would be to rewrite the Capsicum tests in ATF. > I don't know how many tests there are, but this might be doable. > ATF and Googletest look similar. >=20 > If you wanted to import the Capsicum tests into FreeBSD as they are right > now, then it looks > like you would need to import googletest into FreeBSD as well. googletest= > is BSD licensed, > so that is not a problem, but we already have ATF in the tree, so I don't > know how > useful it would be to have multiple test frameworks in the base system. Hi Ed, I'll take a look at the test infrastructure and see if I can write a sim= ple shim for ATF to interact with it. I agree with a lot of what Craig said. The Google corporate CLA has impe= ded engineers in EMC / Isilon and some other groups from contributing back t= o the ATF / Kyua projects ;( (this is why I've just been filing bugs against= the projects and have been focusing my efforts on integrating tools/regress= ion tests into Kyua), so anything that needs to be done development wise on t= hese projects will need to be done internal to the FreeBSD project for the t= ime being. Thanks! -NGie=