From owner-freebsd-testing@FreeBSD.ORG Wed Jun 3 15:30:13 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D2D08385; Wed, 3 Jun 2015 15:30:13 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mail-yk0-x22e.google.com (mail-yk0-x22e.google.com [IPv6:2607:f8b0:4002:c07::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 92D8A1493; Wed, 3 Jun 2015 15:30:13 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: by ykfl8 with SMTP id l8so4758310ykf.1; Wed, 03 Jun 2015 08:30:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=sBqtdjNupHM/QqVaIF81R57sI8aELwDjhGv04GFiW+g=; b=OdU5/3G1Bxw3tNwjR4w7VxvjdkFEBxcGKZoJHWsErnHTbQT73uf3RqqYIahW7EME6V KZmCVVvzmmL/oiQb99Rx7hMHzXu1oLfq2ND3/H5sfDfVGG/7UpUy5BXPAonWqNgIVVlD K7PRwMCw7Fr+BlEOxAP9TgI9t0reX1CgE8W2ymZHKN+SNOItFSKmTvoJmW8wJ7nYzmBX UfnvhQiUXlW4uwd4DEZTdsKf+0jnnz/j7PRoViNmUIYZnMMGZFVvjCjJdl3KBO3TimH/ iheAu/YnU5qWyUqEz+I0sc9CpiVg7Hj0raM1J2dsRqVJvqYnF58I0Ful4ttBI0nMC+rz 7gdw== MIME-Version: 1.0 X-Received: by 10.170.125.208 with SMTP id r199mr13580554ykb.15.1433345412668; Wed, 03 Jun 2015 08:30:12 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.13.233.70 with HTTP; Wed, 3 Jun 2015 08:30:12 -0700 (PDT) In-Reply-To: References: Date: Wed, 3 Jun 2015 08:30:12 -0700 X-Google-Sender-Auth: giv6x2CWat6G4c9UzedNnqMrM1Q Message-ID: Subject: Re: Incorporating the Capsicum test suite From: Craig Rodrigues To: Ed Maste Cc: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 15:30:14 -0000 On Tue, Jun 2, 2015 at 1:40 PM, Ed Maste wrote: > > I have a couple of questions. First, do we have an existing pattern > for connecting googletest tests to Kyua? > No we do not. Googletest looks like it is basically trying to do the same thing as ATF or CUnit. Googletest has a way to generate JUnit XML test output: https://code.google.com/p/googletest/wiki/AdvancedGuide#Generating_an_XML_Report The easiest thing to do would be to configure the Capsicum tests to generate this XML test output. This can be imported natively into Jenkins, which can then parse the test report. This would require no changes to the Capsicum tests. If you want to integrate googletest with kyua, then things get tricky. On the googletest page: https://code.google.com/p/googletest/ they list a third party plugin which can take googletest and generate TAP output. I don't know how well that works. It would be nice if that functionality was integrated in googletest natively. Another option would be to modify kyua, so that it can parse the output of googletest natively. This would be quite nice. I haven't found extending kyua to be very easy....it requires good knowledge of C++ and Lua. Also, whoever contributes code to kyua must sign a Contributor License Agreement (CLA) with Google. That's something to keep in mind. Another option would be to rewrite the Capsicum tests in ATF. I don't know how many tests there are, but this might be doable. ATF and Googletest look similar. If you wanted to import the Capsicum tests into FreeBSD as they are right now, then it looks like you would need to import googletest into FreeBSD as well. googletest is BSD licensed, so that is not a problem, but we already have ATF in the tree, so I don't know how useful it would be to have multiple test frameworks in the base system. -- Craig