From owner-freebsd-testing@FreeBSD.ORG Wed Jun 3 19:51:33 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3F274EB5; Wed, 3 Jun 2015 19:51:33 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-ig0-x233.google.com (mail-ig0-x233.google.com [IPv6:2607:f8b0:4001:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 08E1817B6; Wed, 3 Jun 2015 19:51:33 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by igbpi8 with SMTP id pi8so121200490igb.1; Wed, 03 Jun 2015 12:51:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=6N/Vihz0xUPOSyQiU+QU44nweMOt2pwPVfMCaezdLCg=; b=CpQh70/ArSUXOW04pE7BozkETeSpm/rlL+IGu2xX6W8Tvgu1Z+N74DIKKV+d0Xb/OV CMH8ero60TCHZtvq1Ylkdr7uI7vsIGTGd3KyRcEDfi0PZ5y8JNGUfoJXKWNOfJzDKIpL cau2sLNK3muSmzi89P5uRs1GSbxbZjnPpph6t00iyxu4LzAqyUe7kEvAbBh2CzBDvGTI VY/C45WVfBGrrxoOFNCbv8tEboPx2KfGnnb0LtFHIQMua2H2dlyJ1/sC8l5tgQv50Aw3 NMZs4MNJwg0+vKpyg5peUY2TN4QYcu0sd3SA71AD3fbC6R7VVNNqvyGlkFr203kkVF60 SXtA== X-Received: by 10.50.30.105 with SMTP id r9mr29482742igh.11.1433361092508; Wed, 03 Jun 2015 12:51:32 -0700 (PDT) MIME-Version: 1.0 Sender: carpeddiem@gmail.com Received: by 10.107.48.3 with HTTP; Wed, 3 Jun 2015 12:51:11 -0700 (PDT) In-Reply-To: References: From: Ed Maste Date: Wed, 3 Jun 2015 15:51:11 -0400 X-Google-Sender-Auth: b0KK3IvMy5uXVB0Bw3Rz5xFS56c Message-ID: Subject: Re: Incorporating the Capsicum test suite To: Craig Rodrigues Cc: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 19:51:33 -0000 On 3 June 2015 at 11:30, Craig Rodrigues wrote: > On Tue, Jun 2, 2015 at 1:40 PM, Ed Maste wrote: >> >> I have a couple of questions. First, do we have an existing pattern >> for connecting googletest tests to Kyua? > > No we do not. Googletest looks like it is basically trying to do > the same thing as ATF or CUnit. Yes, the intent is the same. My desire to integrate it with Kyua is so that we can add this test coverage without needing the user to run additional commands, parse different output formats, etc. We could integrate the existing tests as-is into Jenkins fairly easily, but I also want us to move to a model where developers consistently run the test suite prior to committing a change. > Another option would be to modify kyua, so that it can parse the output of > googletest > natively. This would be quite nice. I haven't found extending kyua to be > very > easy....it requires good knowledge of C++ and Lua. Also, whoever > contributes > code to kyua must sign a Contributor License Agreement (CLA) with Google. > That's something to keep in mind. It seems to me that this is the best way to go, with connecting the existing tests as-is into Jenkins as my second choice. FreeBSD developers who wish to make changes to Kyua and the Google CLA is an issue we could fork it and make changes in our own version. It's not ideal, but in practice if we don't expect large, conflicting changes from upstream it should be a workable approach. > Another option would be to rewrite the Capsicum tests in ATF. > I don't know how many tests there are, but this might be doable. > ATF and Googletest look similar. I think this is feasible, and may make sense for the Capsicum tests. However, the idea extends beyond the Capsicum tests. Clang/LLVM (and LLDB) have a large test suite that I'd like incorporate into our process, and other 3rd party software does as well. It's not feasible to try to convert all of these to ATF. > If you wanted to import the Capsicum tests into FreeBSD as they are right > now, then it looks > like you would need to import googletest into FreeBSD as well. googletest > is BSD licensed, > so that is not a problem, but we already have ATF in the tree, so I don't > know how > useful it would be to have multiple test frameworks in the base system. The parts of googletest actually required for the capsicum tests are fairly small and self-contained (even though the entire googletest package is included in the capsicum-test repo). Assuming capsicum-test gets stripped it down to just what's required I might just consider the embedded googletest part of capsicum-test, rather than importing it as a standalone project and having capsicum-test use it.