From owner-freebsd-testing@FreeBSD.ORG Tue Jun 2 20:40:55 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 522629FA for ; Tue, 2 Jun 2015 20:40:55 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: from mail-ie0-x231.google.com (mail-ie0-x231.google.com [IPv6:2607:f8b0:4001:c03::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 25B4C1836 for ; Tue, 2 Jun 2015 20:40:55 +0000 (UTC) (envelope-from carpeddiem@gmail.com) Received: by ieczm2 with SMTP id zm2so142510909iec.1 for ; Tue, 02 Jun 2015 13:40:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:from:date:message-id:subject:to:content-type; bh=FALgWuMoBQHqxo0MQDO7x3e3UQdqNnKr8T/KbKJHBpU=; b=fP+WQ+CxXkfV/Inn9ORXAdqX1+7E65mXxZu2mmt/ZvXeQprK/XhDvK8s/+e6Uo3Jft L9tYYWgyyHgwriWnq2z3ycYm/6Wyzn/LJsTYPFba8bnyu/gzHLxx8Bmeae7S3/Y1Z034 nSDuJ0Kgs4IPbkqZCn/s3wgaddlDoc5Bsu0108PNCtSRgkLWlWaerreRJ1WJ5ZlDnaD8 SDB6Y6gGiZBwmCGDGB/RW9ErtzpObs5GM/ExDR1DEzM+YMow4mWz1irwqoSnuk4MtJmV CzSzW9+bKMuzQIqqZN0mXW6ydgDZnBh3RjdBKip/JMK702a0ZjIIFLTwHvbyVA1OraYd WHng== X-Received: by 10.42.176.8 with SMTP id bc8mr25831711icb.22.1433277654353; Tue, 02 Jun 2015 13:40:54 -0700 (PDT) MIME-Version: 1.0 Sender: carpeddiem@gmail.com Received: by 10.107.48.3 with HTTP; Tue, 2 Jun 2015 13:40:33 -0700 (PDT) From: Ed Maste Date: Tue, 2 Jun 2015 16:40:33 -0400 X-Google-Sender-Auth: PWjeTj7oE9hwBCHzLgGlRriJ30g Message-ID: Subject: Incorporating the Capsicum test suite To: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 20:40:55 -0000 I would like to incorporate the Capsicum test suite[1] into our test infrastructure and Jenkins. There are a small handful of minor issues to resolve, but it's very close to a state where it "just builds and passes." It uses googletest, which produces output of the form: [----------] 10 tests from Capmode [ RUN ] Capmode.AllowedIdentifierSyscallsForked [ OK ] Capmode.AllowedIdentifierSyscallsForked (11 ms) [ RUN ] Capmode.AllowedSchedSyscallsForked capmode.cc:289: Failure Expected: (0) <= (sched_rr_get_interval(0, &ts)), actual: 0 vs -1 errno 94 Not permitted in capability mode capmode.cc:276: Failure Value of: rc Actual: 1 Expected: 0 [ FAILED ] Capmode.AllowedSchedSyscallsForked (11 ms) and reports success/failure via the result code. I have a couple of questions. First, do we have an existing pattern for connecting googletest tests to Kyua? We could incorporate it fairly easily by just encapsulating the whole test, but integrating at the level of individual tests would be much better. Second, I wonder if we should import these tests into contrib/ and build them during buildworld, or put them into a port and install them into /usr/local/tests/capsicum. I'd lean towards putting them in contrib since Capsicum is an integrated part of the base system, but it might be a bit easier to maintain if we keep it as a port/package. Presumably the Jenkins run will just execute all tests in /usr/tests/ and /usr/local/tests/. [1] https://github.com/google/capsicum-test