From owner-freebsd-testing@FreeBSD.ORG Sun Jul 27 20:38:20 2014 Return-Path: Delivered-To: testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 53B9A94C; Sun, 27 Jul 2014 20:38:20 +0000 (UTC) Received: from vps.hungerhost.com (vps.hungerhost.com [216.38.53.176]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 27C3925A5; Sun, 27 Jul 2014 20:38:19 +0000 (UTC) Received: from a6.norwich.yourspac.nsdsl.net ([94.229.131.101]:51458 helo=[10.3.6.18]) by vps.hungerhost.com with esmtpsa (TLSv1:DHE-RSA-AES256-SHA:256) (Exim 4.80.1) (envelope-from ) id 1XBVCy-00069X-Ga; Sun, 27 Jul 2014 16:38:16 -0400 From: "George Neville-Neil" To: "Garrett Cooper" Subject: Re: A new way to test systems in multiple machine scenarios... Date: Sun, 27 Jul 2014 21:38:05 +0100 Message-ID: <9207F596-D12C-4624-932C-631E47D40D66@neville-neil.com> In-Reply-To: <19D0342C-3635-4DC1-ACB8-5697F1D579F0@gmail.com> References: <19D0342C-3635-4DC1-ACB8-5697F1D579F0@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Mailer: MailMate (1.8r4214) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - vps.hungerhost.com X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - neville-neil.com X-Get-Message-Sender-Via: vps.hungerhost.com: authenticated_id: gnn@neville-neil.com Cc: "testing@freebsd.org" , "net@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 27 Jul 2014 20:38:20 -0000 On 6 Jul 2014, at 4:52, Garrett Cooper wrote: >> On Jul 5, 2014, at 20:04, "George Neville-Neil" >> wrote: >> >> Hi, >> >> I've coded up a system to allow you to control multiple other systems >> for use in testing. >> >> https://github.com/gvnn3/conductor >> >> It's BSD licensed, of course, and is only alpha quality but I'm using >> it in the test lab >> to control hosts in forwarding tests. >> >> I'll be updating the system frequently over the coming months as I >> build out more test scenarios, >> add documentation and the like. >> >> There are two main scripts, player, and conductor. You run N >> players, one per machine, and >> a single conductor. The conductor controls the players by sending >> down phases which are >> encoded in INI style configs. There are a few, simple, samples in >> the config/ directory >> of the project. >> >> Best, >> George >> >> NOTE: Conductor MUST run as root to be useful. Do NOT run on the >> open Internet. It is meant >> for private test labs. > > I took a quick glance at the code -- have you considered using > multiprocessing managers instead? > > https://docs.python.org/2/library/multiprocessing.html#managers I had not. Thanks. I'll give it a look over. Best, George From owner-freebsd-testing@FreeBSD.ORG Mon Jul 28 00:01:08 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A1C91700 for ; Mon, 28 Jul 2014 00:01:08 +0000 (UTC) Received: from mail-qg0-f49.google.com (mail-qg0-f49.google.com [209.85.192.49]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6486F275D for ; Mon, 28 Jul 2014 00:01:07 +0000 (UTC) Received: by mail-qg0-f49.google.com with SMTP id j107so7621846qga.22 for ; Sun, 27 Jul 2014 17:01:01 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=JGSF2KrvI1m0Jr+UGEgoMMYkU29eGyk7CLVI1BCrgFs=; b=VfGiS0F6eet965gytMZY2ALQS2HWJQQZCCBcPv2eU/HOhUQrkTBQv17zqN2WnTQt9r 6wz3w8WKjPgipK/zuGqVFjrMEI2TKNsp2JLkUB7hZSn5BKTNwBKZ0IUhg4KEb+91te5f Qimix0XjQ5EpdiLZAplGoXnByruIJfRhA0aRLoO/oxq3cAsmjtrYC+o2sUgXTYH1q/G+ g8fgqXxX7kQO87PB7B1MJ0qeRZHMY28xnkHXkzrYNDOBk9SuzNGXLCAnxp3pysk+ernV ga6d2obZteLZsZBxtwzsY4VHUr6qNKzkZXm8IUb0H5o993qLGKT9ykIxkhlH7Ppd6y+a sLCA== X-Gm-Message-State: ALoCoQksqOCWaxOsrlS1yzYcfRPR6vm7ClqEXIx5WsYyjmNUmWpc1ZgE6NSDoLAyfSXRKQKvOW6D X-Received: by 10.224.42.196 with SMTP id t4mr52567304qae.48.1406505661645; Sun, 27 Jul 2014 17:01:01 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Sun, 27 Jul 2014 17:00:41 -0700 (PDT) X-Originating-IP: [50.202.45.106] In-Reply-To: <41d8a2fb.75b47028@fabiankeil.de> References: <41d8a2fb.75b47028@fabiankeil.de> From: Julio Merino Date: Sun, 27 Jul 2014 17:00:41 -0700 X-Google-Sender-Auth: lQX-5yW9PqBTTMYwtLLkg2hN9Ac Message-ID: Subject: Re: atf_check() equivalent to test individual shell functions? To: Fabian Keil Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 00:01:08 -0000 On Sat, Jul 26, 2014 at 4:30 AM, Fabian Keil wrote: > Here's another zogftw-related problem. What's zogftw? > Putting something like this in a test case works as expected: > > | atf_check -s exit:0 -o empty -e empty $ZOGFTW cmd zogftw_string_ends_with "abc" "c" What is "$ZOGFTW cmd ..." supposed to be doing? > It would be preferably to source zogftw once at the beginning > and call the shell function zogftw_string_ends_with() directly > afterwards to reduce the overhead, though. > > The following sort of does this, but the stdout and stderr > output isn't verified, it's tedious to write and if nothing > fails, the "kyua debug" output isn't particular useful: > > | . $ZOGFTW source || atf_fail "Failed to source zogftw" > | > | if ! zogftw_string_ends_with "abc" "c"; then > | atf_fail "Failed to find 'c' at the end of 'abc'" > | fi > > The following also sort of works, but in case of errors the output > isn't helpful (because atf_check_equal() doesn't see the actual > function) and the stderr output isn't verified either: > > | . $ZOGFTW source || atf_fail "Failed to source zogftw" > | > | atf_check_equal $(zogftw_string_ends_with "abc" "c"; echo $?) "0" Or just: zogftw_string_ends_with "abc" "c" || atf_fail "abc doesn't end with c" There is no reason to compare the exit code literally with atf_check_equal. > What I'm looking for is something like this: > > | atf_check_shell_function -s return:0 -o empty -e empty zogftw_string_ends_with "abc" "c" That could be interesting. However, note that atf_check currently is just a thin wrapper over the atf-check binary and that's the reason why you cannot feed shell functions into it. Implementing an atf_check_shell_function would involve rewriting atf-check's functionality in libatf-sh.subr. > atf-sh-api(3) doesn't seem to mention anything like it, > so I'm wondering how other people are testing individual > shell functions? Take a look at the test programs in here for other ideas: https://github.com/jmmv/shtk The background is that atf-sh has been mostly used for integration testing until now so the support for unit-testing shell functions hasn't been necessary. I agree that building some would be good. From owner-freebsd-testing@FreeBSD.ORG Mon Jul 28 14:21:39 2014 Return-Path: Delivered-To: testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 450D2907 for ; Mon, 28 Jul 2014 14:21:39 +0000 (UTC) Received: from mail-qa0-x22a.google.com (mail-qa0-x22a.google.com [IPv6:2607:f8b0:400d:c00::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0AD5B207C for ; Mon, 28 Jul 2014 14:21:38 +0000 (UTC) Received: by mail-qa0-f42.google.com with SMTP id j15so7747102qaq.15 for ; Mon, 28 Jul 2014 07:21:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=ZERrh9fLq8hV2M1WMeesEpwvWESvK/Gl8B3YPrphuWs=; b=xkrPb0Tc/xxblQR2Mf9H3QR3HSF1qnfTTsFJImxMAoAUYcXAgVOwCADZoJHMSrgJgl gDo/+JbFo8jGWkT7PrIHR+DG+pdd7+wkx05kqbFeHQI/owqoJSPMxoGsD+Q7VDdgtE4W R914Z9mRjt0dCPSq+3yBEk8/awfGRjWVq9S4kIX8ifVJYaGnN/8Ut4iUR4bPH9WCOuW0 83B2E8Ejt+gjh2DfwM9L52ioVo4tnGIn4atvJVlkpPvsowDERT4gUxi+JkHTnbgFOgTg ekTjukxm6FynsQ1WHGiRBIawkmq3KO6ux4oOd9Jirt9TlA2k0DgzLSJqjhbshBrac+ET qqSw== MIME-Version: 1.0 X-Received: by 10.224.103.198 with SMTP id l6mr58960072qao.47.1406557298074; Mon, 28 Jul 2014 07:21:38 -0700 (PDT) Sender: sjk4015@gmail.com Received: by 10.140.89.234 with HTTP; Mon, 28 Jul 2014 07:21:38 -0700 (PDT) Date: Mon, 28 Jul 2014 10:21:38 -0400 X-Google-Sender-Auth: 38i9GP-ZkWF2BhGKCPSZyWUYrk0 Message-ID: Subject: Regression tests for true/false From: Steven Kreuzer To: testing@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 14:21:39 -0000 I am in the process of figuring out how I can start writing tests for various utilities and whatnot so I figured I would start out by picking some of the low hanging fruit. However, I am having a bit of an issue finding tutorials or documentation on the preferred way to start add regression tests so I have been looking at some of the existing tests people have been adding. Can someone take a look at this diff to add regression tests that make sure `true` and `false` both exit with the correct value ( http://people.freebsd.org/~skreuzer/patches/true-false.patch ) Also, if anyone has any suggestions or documentation/examples they can point me at to better integrate with ATF, it would be most appreciated. Thanks From owner-freebsd-testing@FreeBSD.ORG Mon Jul 28 16:11:55 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E5E1310B; Mon, 28 Jul 2014 16:11:54 +0000 (UTC) Received: from mail-wi0-x22b.google.com (mail-wi0-x22b.google.com [IPv6:2a00:1450:400c:c05::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B2872F0C; Mon, 28 Jul 2014 16:11:54 +0000 (UTC) Received: by mail-wi0-f171.google.com with SMTP id hi2so4688611wib.4 for ; Mon, 28 Jul 2014 09:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=tYOZnOBMaI7M+zinnlQus0U3Hc9vcvhlBmDxMvv7j9w=; b=yP5YNHlZL9M+D0MZWfWOGbbKOoEFnoJTgjkUuY6pguGfVJM1WpRhZND1w6CScnp4VS 687bXyyEQem3/Jo/0YWyjtV88fLSblTickYQIwfXFU8xTUvs9ZHcAuVxuoIL6rT0FMrR zwz+WtTaYeSM2JL/+jW8ST6Y/fcBGMSQi9riE1GYWFm9+8sB3HGIT0rFSlb6SuBVhO3m Utq+ZegoNqfYNxSofyaTwJLKEF3LjQEBGQC+6WeUOI+BKVGOdGMdwcSr5tvGCqlPAokV USSKu+AvY6WWkZHdC+DxFtePYBuzprYmuWcjG8q1TVyQfQdpUke90YLeHhesO/W1eh/X +yQg== MIME-Version: 1.0 X-Received: by 10.180.39.139 with SMTP id p11mr16030582wik.50.1406563908918; Mon, 28 Jul 2014 09:11:48 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.203.134 with HTTP; Mon, 28 Jul 2014 09:11:48 -0700 (PDT) In-Reply-To: <450da50a.02ce9897@fabiankeil.de> References: <5ee51891.12b049e9@fabiankeil.de> <450da50a.02ce9897@fabiankeil.de> Date: Mon, 28 Jul 2014 10:11:48 -0600 X-Google-Sender-Auth: iQZbtUjyO3FvSkrqxS_hem-U2Dg Message-ID: Subject: Re: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details From: Alan Somers To: Fabian Keil Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 16:11:55 -0000 On Sat, Jul 26, 2014 at 2:58 AM, Fabian Keil wrote: > Alan Somers wrote: > >> On Fri, Jul 25, 2014 at 12:55 PM, Fabian Keil >> wrote: >> > I'm currently evaluating using kyua to test a FreeBSD-specific >> > shell script, however I'm having trouble analysing failing tests. > [...] >> > I get: >> > >> > | fk@r500 ~/git/zogftw/tests $kyua --loglevel debug test -k Kyuafile example:failing_test >> > | example:failing_test -> failed: atf-check failed; see the output of the test for details [0.150s] >> > | >> > | 0/1 passed (1 failed) >> > | Committed action 21 >> > >> > The test failure obviously is expected, but I'm missing the referenced >> > "output of the test". The relevant log excerpt: > [...] >> > I'm looking for something like this: >> > >> > | fk@r500 ~ $/usr/local/libexec/atf-check -s exit:1 -o empty -e empty 'true' >> > | Executing command [ true ] >> > | Fail: incorrect exit status: 0, expected: 1 >> > | stdout: >> > | >> > | stderr: >> > | >> > >> > I didn't read all the docs available yet and I may be missing something, >> > but seeing test failures is something I'd expect to work out of the box. >> > Can someone familiar with kyua confirm that this is working as expected? > >> The test output is stored in Kyua's sqlite database. There are two >> ways to see it: >> 1) Run " kyua report-html" >> 2) Sometimes its easier just to rerun the testcase in debug mode. Do >> "kyua debug example:failing_test" > > Thanks a lot. > > Not showing test failures on the cli by default seems like a strange > design decision to me, but I guess I'll get used to it or write a > wrapper script that calls kyua debug for all tests. I don't recommend calling "kyua debug" for all tests. The best approach is to run "kyua test; kyua report-html ". The HTML report will show you the full stdout and stderr output of each failing test case. > > Fabian From owner-freebsd-testing@FreeBSD.ORG Mon Jul 28 20:59:50 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 02CE06EF for ; Mon, 28 Jul 2014 20:59:50 +0000 (UTC) Received: from mail-qg0-x232.google.com (mail-qg0-x232.google.com [IPv6:2607:f8b0:400d:c04::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BC6B021DF for ; Mon, 28 Jul 2014 20:59:49 +0000 (UTC) Received: by mail-qg0-f50.google.com with SMTP id q108so9287741qgd.37 for ; Mon, 28 Jul 2014 13:59:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=ZERrh9fLq8hV2M1WMeesEpwvWESvK/Gl8B3YPrphuWs=; b=dqcs6zmll254nX2s6b8UlYTDa/qEmBSVrWqh35+a4y+SySruKXZvA+EpAeBchV6QS/ SrFatSHeuXyVOqKBwB2eZiA538UAHvddZsGFAQW+78ZqMOmMoKy2sB1itF+vEnJKNtCQ 7xCaF2WwnU6XvB7HfgR1y/F6GBr1A0gl8DF7tSJXJzuWtbtzZxBBvR03nGeOFeaFKuBA gPe1EvmN/6eRJjHUW9KC+PZmIZ04yM2JOTT+WyKRxyzFChWN+K802Xhz5Q/n32DDbPb3 TtE/xVhLYikYTR/yDxr7uJGzeCuzqGKKCAjtldFawhbTSi2IWUPY517zGmAINQSSe7XO 0HPQ== MIME-Version: 1.0 X-Received: by 10.140.84.21 with SMTP id k21mr47084848qgd.70.1406581188697; Mon, 28 Jul 2014 13:59:48 -0700 (PDT) Sender: sjk4015@gmail.com Received: by 10.140.89.234 with HTTP; Mon, 28 Jul 2014 13:59:48 -0700 (PDT) Date: Mon, 28 Jul 2014 16:59:48 -0400 X-Google-Sender-Auth: uvKpYQw25YJ4rucakyIRQCNTGKE Message-ID: Subject: Regression tests for true/false From: Steven Kreuzer To: freebsd-testing@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 20:59:50 -0000 I am in the process of figuring out how I can start writing tests for various utilities and whatnot so I figured I would start out by picking some of the low hanging fruit. However, I am having a bit of an issue finding tutorials or documentation on the preferred way to start add regression tests so I have been looking at some of the existing tests people have been adding. Can someone take a look at this diff to add regression tests that make sure `true` and `false` both exit with the correct value ( http://people.freebsd.org/~skreuzer/patches/true-false.patch ) Also, if anyone has any suggestions or documentation/examples they can point me at to better integrate with ATF, it would be most appreciated. Thanks From owner-freebsd-testing@FreeBSD.ORG Mon Jul 28 21:05:52 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EF9958DA for ; Mon, 28 Jul 2014 21:05:52 +0000 (UTC) Received: from mail-qg0-f48.google.com (mail-qg0-f48.google.com [209.85.192.48]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B083322A7 for ; Mon, 28 Jul 2014 21:05:52 +0000 (UTC) Received: by mail-qg0-f48.google.com with SMTP id i50so9094237qgf.7 for ; Mon, 28 Jul 2014 14:05:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=6DzOB7uyi0ZEimUvteEa45NkTzI4OzoyVPG0z/H5xHc=; b=K28UacD/matSl36LIQAfjJ906A9s/67g7iS0/aQfEM+blfeCN5oUHOzIdQjZz5GVPm x36NWD173hXnLpt9mapuXkCau4V5JRMK7nJi5gqgQacALEQmeL738WgJNV+bBQ8ZbWmS d/CR/k60Z2uNQ71vAtRsRt1TGMIQPnbQ+P5qz9kOuKYNxf6/w3JXx6m0DAikujtGged7 JjqEHT16cEIIbhxHrsf6BMtZ296cbruQqIAUxNJY3Us4SFKr/PLx0aMzadOTdxmGpRe4 IFK9T+vR4b8OWYngSi5sna72ENKUM2Wasv6SJnb/lWE7ll4WlwFdQ0GFAI1zP1XqJ+NC kcxQ== X-Gm-Message-State: ALoCoQkGsRuB+GuQiYaQOVcuw9FudU2BZv75GD/0mCAokxNrHU1eNpGycc0/AG+0GXZLvPDGX+4k X-Received: by 10.224.120.138 with SMTP id d10mr63991009qar.9.1406581550987; Mon, 28 Jul 2014 14:05:50 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Mon, 28 Jul 2014 14:05:30 -0700 (PDT) X-Originating-IP: [2620:0:1008:100a:1cf4:b8ab:9755:55fa] In-Reply-To: References: From: Julio Merino Date: Mon, 28 Jul 2014 14:05:30 -0700 X-Google-Sender-Auth: cIn6a0Tm5r4GIBBHsVZ6Bx5L33k Message-ID: Subject: Re: Regression tests for true/false To: Steven Kreuzer Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 21:05:53 -0000 On Mon, Jul 28, 2014 at 1:59 PM, Steven Kreuzer wrote: > I am in the process of figuring out how I can start writing tests for > various utilities and whatnot so I figured I would start out by > picking some of the low hanging fruit. > > However, I am having a bit of an issue finding tutorials or > documentation on the preferred way to start add regression tests so I > have been looking at some of the existing tests people have been > adding. > > Can someone take a look at this diff to add regression tests that make > sure `true` and `false` both exit with the correct value ( > http://people.freebsd.org/~skreuzer/patches/true-false.patch ) Can you please upload to Phabricator? From owner-freebsd-testing@FreeBSD.ORG Mon Jul 28 21:18:56 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7E49D3B3 for ; Mon, 28 Jul 2014 21:18:56 +0000 (UTC) Received: from mail-qg0-f53.google.com (mail-qg0-f53.google.com [209.85.192.53]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3B32C24B7 for ; Mon, 28 Jul 2014 21:18:55 +0000 (UTC) Received: by mail-qg0-f53.google.com with SMTP id q107so9172271qgd.40 for ; Mon, 28 Jul 2014 14:18:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=OXOVFajHaPw2iQinMBTIAzT+CqBxJXFbrcKvwcF5n6I=; b=O7ebxW+KpZJhOofatOTeQ/x2GYcU4kUlxx+nWenv5xMitIfcHamKwQSI+an7Oio/JX XYjlaNjJv36qcRkc5xwdZUcqxSC0P5qxjZV1Lj+5y+qX4IYg0PeD79fR9D3Bv9Tzo/m1 sH4wt4BwhR2XR3tsGd5Nk6uHGn5DLDPrznY1hXTU1Tx8WEHaFqmZzlzwCTJPyFMmc1oU 9YJYNRTLwKb79kg5AXpXCnSDKs16Q6jcXktIvCof1X+WUJ5/WnFzg0Kn2r3XPk/bj04/ 4jcBDZKCX5RSbLDGtkGawdc8TXkq8J9xDAC2CeTGzqyy+0wXv8HhMteAjDACtCLYCrI/ lTBA== X-Gm-Message-State: ALoCoQmrMS3UkUg/bdMSmJMARU4GP9k4xRuq+0q2SfDvpmcdhJ8iL+Y8gKprVZ/KcAGv8YDqHKRV X-Received: by 10.224.26.146 with SMTP id e18mr4148462qac.13.1406582328712; Mon, 28 Jul 2014 14:18:48 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Mon, 28 Jul 2014 14:18:28 -0700 (PDT) X-Originating-IP: [2620:0:1008:100a:1cf4:b8ab:9755:55fa] In-Reply-To: References: <5ee51891.12b049e9@fabiankeil.de> <450da50a.02ce9897@fabiankeil.de> From: Julio Merino Date: Mon, 28 Jul 2014 14:18:28 -0700 X-Google-Sender-Auth: PJC_7-u1htxpm-LXGxiNWZ6GLuY Message-ID: Subject: Re: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details To: Alan Somers Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 21:18:56 -0000 On Mon, Jul 28, 2014 at 9:11 AM, Alan Somers wrote: > On Sat, Jul 26, 2014 at 2:58 AM, Fabian Keil > wrote: >> Not showing test failures on the cli by default seems like a strange >> design decision to me, but I guess I'll get used to it or write a >> wrapper script that calls kyua debug for all tests. Test failures are shown on the CLI by default. What is not shown is the output of the test cases. It'd be too verbose and would prevent having any sense of progress of the execution. Making it an option to display the output during "make test" is a reasonable request but it's also non-trivial at the moment. Feel free to file a feature request for this. > I don't recommend calling "kyua debug" for all tests. The best > approach is to run "kyua test; kyua report-html ". The HTML > report will show you the full stdout and stderr output of each failing > test case. We should actually just make "kyua report" show the stdout and stderr, and this is easier to do. Having to resort to report-html is... just ugly. I just haven't thought about a UI for it. Maybe "kyua report --show-output [test cases]" would be good enough? From owner-freebsd-testing@FreeBSD.ORG Mon Jul 28 21:27:09 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BD0BA5AE; Mon, 28 Jul 2014 21:27:09 +0000 (UTC) Received: from mail-wi0-x230.google.com (mail-wi0-x230.google.com [IPv6:2a00:1450:400c:c05::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1078C259E; Mon, 28 Jul 2014 21:27:08 +0000 (UTC) Received: by mail-wi0-f176.google.com with SMTP id bs8so5147273wib.3 for ; Mon, 28 Jul 2014 14:27:07 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=SFDOXdDZ6H7EmmCF47KPF/Nle6ARgOJCFmYoOhKglzw=; b=ND+2gHysSwsRWxmYKBu6huVopg/PykKHf1p6/aEKtmR10f49peQrMS7C/lt8IWebgJ exGn1vOPi0iVVhXBIKnJ9jlvzbg/79t3yYakzBGe4jygUK8x2NrFqXR5J1qWSPWmI1TC DZ7kj8F9qF26KIqnooENrrqLURErz7NYxY070lXkRWF/Dp5NxEbWG48OgqdSTOrcr22p m38NxfkmRbIZkKVHETrtJ5DIcTit+dKAm+SHrXF4/VlC+LrVgt9/4dKxjH9HiEXoc8Jj 0HVRWEu6id5i6ob8rXB9WpyJDyYMxksDjxDoKgFCaNf1ALLtqHJtgBqmaYAWijqvneyl n+XA== MIME-Version: 1.0 X-Received: by 10.194.92.33 with SMTP id cj1mr4170844wjb.66.1406582826969; Mon, 28 Jul 2014 14:27:06 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.203.134 with HTTP; Mon, 28 Jul 2014 14:27:06 -0700 (PDT) In-Reply-To: References: <5ee51891.12b049e9@fabiankeil.de> <450da50a.02ce9897@fabiankeil.de> Date: Mon, 28 Jul 2014 15:27:06 -0600 X-Google-Sender-Auth: j7KUq6su8YMGQNUZykttE5qTHtE Message-ID: Subject: Re: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details From: Alan Somers To: Julio Merino Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 21:27:09 -0000 On Mon, Jul 28, 2014 at 3:18 PM, Julio Merino wrote: > On Mon, Jul 28, 2014 at 9:11 AM, Alan Somers wrote: >> On Sat, Jul 26, 2014 at 2:58 AM, Fabian Keil >> wrote: >>> Not showing test failures on the cli by default seems like a strange >>> design decision to me, but I guess I'll get used to it or write a >>> wrapper script that calls kyua debug for all tests. > > Test failures are shown on the CLI by default. What is not shown is > the output of the test cases. It'd be too verbose and would prevent > having any sense of progress of the execution. > > Making it an option to display the output during "make test" is a > reasonable request but it's also non-trivial at the moment. Feel free > to file a feature request for this. > >> I don't recommend calling "kyua debug" for all tests. The best >> approach is to run "kyua test; kyua report-html ". The HTML >> report will show you the full stdout and stderr output of each failing >> test case. > > We should actually just make "kyua report" show the stdout and stderr, > and this is easier to do. Having to resort to report-html is... just > ugly. > > I just haven't thought about a UI for it. Maybe "kyua report > --show-output [test cases]" would be good enough? How about "kyua report --verbose". Seems more conventional. -Alan From owner-freebsd-testing@FreeBSD.ORG Mon Jul 28 21:42:56 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 56802D33; Mon, 28 Jul 2014 21:42:56 +0000 (UTC) Received: from mail-qa0-x236.google.com (mail-qa0-x236.google.com [IPv6:2607:f8b0:400d:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 060982756; Mon, 28 Jul 2014 21:42:55 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id k15so8855892qaq.27 for ; Mon, 28 Jul 2014 14:42:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=WvzKCI4hAXJ2/RO7caCcQuTYrmBpWKOv7KX9DmHbRbE=; b=UpirMyjuVjy2UT8A3TiYqxYz3VV9ZnJ2LoBV1Ec+IirwTnNe+jhHc7EmEasSU0PILo 22sbQtmEqvQ+yLjBLypsbT4sYr5eWrGP1VjZCZulqWjtAYT/dg/kA6hTB/KBIEjIFjCM ckVjWZoGbGJKxxV+AjbwshJ4ezlloEgQ4wYMzWXR3Vzg5GogXfjl0w3vbUMI5CCQoSwf E1Lp2ezbROvFj8P0/Gv38LrP49kTBWWpp6O2ZIdprGldGUuaNCwpPIWGZZp9jb1Y+ENC cxNuXi5G4Aqi6Dc1DmDkYbX+gua57qnKdbw+Y7axzg8tquIkXEJiyDE/eyTgrOhQNPXR Qq3g== MIME-Version: 1.0 X-Received: by 10.224.103.198 with SMTP id l6mr63005394qao.47.1406583775188; Mon, 28 Jul 2014 14:42:55 -0700 (PDT) Sender: sjk4015@gmail.com Received: by 10.140.89.234 with HTTP; Mon, 28 Jul 2014 14:42:55 -0700 (PDT) In-Reply-To: References: Date: Mon, 28 Jul 2014 17:42:55 -0400 X-Google-Sender-Auth: nYGXJe0MssKkWveaqeaWj50sB5c Message-ID: Subject: Re: Regression tests for true/false From: Steven Kreuzer To: Julio Merino Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 21:42:56 -0000 On Mon, Jul 28, 2014 at 5:05 PM, Julio Merino wrote: > Can you please upload to Phabricator? Review url: https://phabric.freebsd.org/D496 Thanks From owner-freebsd-testing@FreeBSD.ORG Tue Jul 29 00:00:15 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 51381230 for ; Tue, 29 Jul 2014 00:00:15 +0000 (UTC) Received: from mail-qa0-f53.google.com (mail-qa0-f53.google.com [209.85.216.53]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0D09F271A for ; Tue, 29 Jul 2014 00:00:14 +0000 (UTC) Received: by mail-qa0-f53.google.com with SMTP id v10so8521503qac.40 for ; Mon, 28 Jul 2014 17:00:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=gy+VU5iwmfwqHSwjt4sE/eUkN6gL3JStKgV0g4uwnMk=; b=QbLQGASMj5wbdWhB4APlRF/8qoBIrSp70JivsjkMxCcWojGaP/XscwUa+z1oIb0DGk 17oqOnyc/KUjl/aANvdIVZv92QrZBxIcwgU+j0ABGWouuajn6TJMiyFUG097XrQkkQsy lPAhqyDbK+unmw7KjxZnjUsIrdUvcK8JxmF22+yyZg0udGclZliH5jQ3rfqpdCULMyKs ugmUylbaiw1LAz6V33dQLrhBLCEXpK0vuKhG8EQIjZKqVK/fdvPpOzGzcaH0/jlLMNnM dx//7Qcja6Qr4yNfZZ4imOzO4BwaWGimc3qBv0qqzdGzKaLqgyCsWpwO2vDgeZ0KGP73 9MkA== X-Gm-Message-State: ALoCoQmZ8yqxE5OJ4vZyH0cwCgfruVUvaC+BdiO2uAXt92KxU25IPEwZglLyfA2mjrDwc5EXg3cw X-Received: by 10.224.26.146 with SMTP id e18mr5271861qac.13.1406592008725; Mon, 28 Jul 2014 17:00:08 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Mon, 28 Jul 2014 16:59:48 -0700 (PDT) X-Originating-IP: [2620:0:1008:100a:1cf4:b8ab:9755:55fa] In-Reply-To: References: <5ee51891.12b049e9@fabiankeil.de> <450da50a.02ce9897@fabiankeil.de> From: Julio Merino Date: Mon, 28 Jul 2014 16:59:48 -0700 X-Google-Sender-Auth: hzPAHXohrRhzZOfQHNNqGwCTvbE Message-ID: Subject: Re: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details To: Alan Somers Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 00:00:15 -0000 On Mon, Jul 28, 2014 at 2:27 PM, Alan Somers wrote: > On Mon, Jul 28, 2014 at 3:18 PM, Julio Merino wrote: >> I just haven't thought about a UI for it. Maybe "kyua report >> --show-output [test cases]" would be good enough? > > How about "kyua report --verbose". Seems more conventional. Note that we already have --show-context. The side-effect of adding a --verbose flag would imply subsuming the functionality of --show-context into it, which in turns means that querying a single test case would show both the output of the test case plus the context. Not a bad idea now that I think about it, but just saying. From owner-freebsd-testing@FreeBSD.ORG Tue Jul 29 10:37:35 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8158726F for ; Tue, 29 Jul 2014 10:37:35 +0000 (UTC) Received: from smtprelay04.ispgateway.de (smtprelay04.ispgateway.de [80.67.29.8]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 400BC2846 for ; Tue, 29 Jul 2014 10:37:35 +0000 (UTC) Received: from [84.44.154.146] (helo=fabiankeil.de) by smtprelay04.ispgateway.de with esmtpsa (SSLv3:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1XC4mh-00072L-TD for freebsd-testing@freebsd.org; Tue, 29 Jul 2014 12:37:32 +0200 Date: Tue, 29 Jul 2014 12:37:35 +0200 From: Fabian Keil To: Subject: Re: atf_check() equivalent to test individual shell functions? Message-ID: <1552a16f.63e4cafe@fabiankeil.de> In-Reply-To: References: <41d8a2fb.75b47028@fabiankeil.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/0mbW+85UlyoZ+jB1kJO0_kU"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 10:37:35 -0000 --Sig_/0mbW+85UlyoZ+jB1kJO0_kU Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Julio Merino wrote: > On Sat, Jul 26, 2014 at 4:30 AM, Fabian Keil > wrote: > > Here's another zogftw-related problem. >=20 > What's zogftw? The program I'm writing tests for: http://www.fabiankeil.de/gehacktes/zogftw/ > > Putting something like this in a test case works as expected: > > > > | atf_check -s exit:0 -o empty -e empty $ZOGFTW cmd zogftw_string_ends_= with "abc" "c" >=20 > What is "$ZOGFTW cmd ..." supposed to be doing? $ZOGFTW is the path to zogftw ("$(atf_get_srcdir)/../zogftw"), "cmd" is a subcommand that executes the additional arguments in the context of zogftw itself. > > It would be preferably to source zogftw once at the beginning > > and call the shell function zogftw_string_ends_with() directly > > afterwards to reduce the overhead, though. > > > > The following sort of does this, but the stdout and stderr > > output isn't verified, it's tedious to write and if nothing > > fails, the "kyua debug" output isn't particular useful: > > > > | . $ZOGFTW source || atf_fail "Failed to source zogftw" > > | > > | if ! zogftw_string_ends_with "abc" "c"; then > > | atf_fail "Failed to find 'c' at the end of 'abc'" > > | fi > > > > The following also sort of works, but in case of errors the output > > isn't helpful (because atf_check_equal() doesn't see the actual > > function) and the stderr output isn't verified either: > > > > | . $ZOGFTW source || atf_fail "Failed to source zogftw" > > | > > | atf_check_equal $(zogftw_string_ends_with "abc" "c"; echo $?) "0" >=20 > Or just: >=20 > zogftw_string_ends_with "abc" "c" || atf_fail "abc doesn't end with c" In which case unexpected stdout output emitted from zogftw_string_ends_with would not be detected, though. > > What I'm looking for is something like this: > > > > | atf_check_shell_function -s return:0 -o empty -e empty zogftw_string_= ends_with "abc" "c" >=20 > That could be interesting. >=20 > However, note that atf_check currently is just a thin wrapper over the > atf-check binary and that's the reason why you cannot feed shell > functions into it. Implementing an atf_check_shell_function would > involve rewriting atf-check's functionality in libatf-sh.subr. Indeed, I just wondered if someone had already done it. =20 > > atf-sh-api(3) doesn't seem to mention anything like it, > > so I'm wondering how other people are testing individual > > shell functions? >=20 > Take a look at the test programs in here for other ideas: >=20 > https://github.com/jmmv/shtk Will do, thanks. > The background is that atf-sh has been mostly used for integration > testing until now so the support for unit-testing shell functions > hasn't been necessary. I agree that building some would be good. Great. Fabian --Sig_/0mbW+85UlyoZ+jB1kJO0_kU Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlPXeW8ACgkQBYqIVf93VJ3srACbB8jHbT92z36nfc6nfNGSHAfq liQAoKePx8fS+ijmdYi+LTWwpsUVuFqC =VCkC -----END PGP SIGNATURE----- --Sig_/0mbW+85UlyoZ+jB1kJO0_kU-- From owner-freebsd-testing@FreeBSD.ORG Tue Jul 29 11:02:18 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 562D9A92 for ; Tue, 29 Jul 2014 11:02:18 +0000 (UTC) Received: from smtprelay05.ispgateway.de (smtprelay05.ispgateway.de [80.67.31.100]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0FBCE2ABE for ; Tue, 29 Jul 2014 11:02:17 +0000 (UTC) Received: from [84.44.154.146] (helo=fabiankeil.de) by smtprelay05.ispgateway.de with esmtpsa (SSLv3:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1XC50D-0004ch-9T for freebsd-testing@freebsd.org; Tue, 29 Jul 2014 12:51:29 +0200 Date: Tue, 29 Jul 2014 12:51:32 +0200 From: Fabian Keil To: Subject: Re: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details Message-ID: <0d851725.1cb2c6b8@fabiankeil.de> In-Reply-To: References: <5ee51891.12b049e9@fabiankeil.de> <450da50a.02ce9897@fabiankeil.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/e/3Qw.AYZnJuEFA_0zR/370"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 11:02:18 -0000 --Sig_/e/3Qw.AYZnJuEFA_0zR/370 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Julio Merino wrote: > On Mon, Jul 28, 2014 at 9:11 AM, Alan Somers wrote: > > On Sat, Jul 26, 2014 at 2:58 AM, Fabian Keil > > wrote: > > I don't recommend calling "kyua debug" for all tests. The best > > approach is to run "kyua test; kyua report-html ". The HTML > > report will show you the full stdout and stderr output of each failing > > test case. >=20 > We should actually just make "kyua report" show the stdout and stderr, > and this is easier to do. Having to resort to report-html is... just > ugly. Agreed. > I just haven't thought about a UI for it. Maybe "kyua report > --show-output [test cases]" would be good enough? I'd prefer two options, one that only shows the output for failing test cases and one that shows the output for all. Fabian --Sig_/e/3Qw.AYZnJuEFA_0zR/370 Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlPXfLQACgkQBYqIVf93VJ1TFACfWVQe+cFxF4T5MBppBNpJc/gF VBAAn0EhAAyfXVEtuvgz6EnSjGqL9/dW =/PR6 -----END PGP SIGNATURE----- --Sig_/e/3Qw.AYZnJuEFA_0zR/370-- From owner-freebsd-testing@FreeBSD.ORG Tue Jul 29 14:31:28 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9025DE3D for ; Tue, 29 Jul 2014 14:31:28 +0000 (UTC) Received: from mail-qa0-f43.google.com (mail-qa0-f43.google.com [209.85.216.43]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4D49B2709 for ; Tue, 29 Jul 2014 14:31:27 +0000 (UTC) Received: by mail-qa0-f43.google.com with SMTP id w8so9332791qac.2 for ; Tue, 29 Jul 2014 07:31:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=xgFmRX9v+4f2OjTFznC/6A5jy/pRPpHGYtYtv3xB4hk=; b=BwyL34VseXuEKY9sRXBYbJdXcZiXoM201NhkuMjgDNKw1nAIvuiYBo0jFoVCb8GaFZ rcXe5fqu4icslBuTJsQG8C6NkSTDRdtTregrZR5njwJZUJSv26LjGJ6y59flkWlds+In jTuukhTjprqva/CFdTp8MtCNJ/mTmAbYQcdX1cNUTCeEdVs6DR5F6HevCn7XE+G3Np5K P0Q75GCAXIzFQwUfcmUDimmuBpivQyN06hb0lkmOVv8zf55VLptMlv33ApFK45W8HYNT 9/+rt0ip1NoxDCappW4E14MfyZLlPYfa66YazZJqMT+WD5GORday3UXg/gDtm2+e1VDN HZbQ== X-Gm-Message-State: ALoCoQnK+Ia4UNH+V1xypN0GfI+1J7hOp/Ks/Q2RJdieA9p111gGNQPjNVVFRXbcwH9kPh0jEDhr X-Received: by 10.224.120.138 with SMTP id d10mr4113018qar.9.1406644279149; Tue, 29 Jul 2014 07:31:19 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Tue, 29 Jul 2014 07:30:59 -0700 (PDT) X-Originating-IP: [2620:0:1008:100a:d1ec:f72a:142d:2523] In-Reply-To: <0d851725.1cb2c6b8@fabiankeil.de> References: <5ee51891.12b049e9@fabiankeil.de> <450da50a.02ce9897@fabiankeil.de> <0d851725.1cb2c6b8@fabiankeil.de> From: Julio Merino Date: Tue, 29 Jul 2014 07:30:59 -0700 X-Google-Sender-Auth: TpXv4oK7AfLpQ96oYK2xCMNKNM0 Message-ID: Subject: Re: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details To: Fabian Keil Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 Jul 2014 14:31:28 -0000 On Tue, Jul 29, 2014 at 3:51 AM, Fabian Keil wrote: > Julio Merino wrote: >> I just haven't thought about a UI for it. Maybe "kyua report >> --show-output [test cases]" would be good enough? > > I'd prefer two options, one that only shows the output for > failing test cases and one that shows the output for all. Note there already is a --results-filter which defaults to only showing "interesting" test cases (aka everything but passed). From owner-freebsd-testing@FreeBSD.ORG Thu Jul 31 22:01:08 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 785491D6 for ; Thu, 31 Jul 2014 22:01:08 +0000 (UTC) Received: from mail-lb0-x230.google.com (mail-lb0-x230.google.com [IPv6:2a00:1450:4010:c04::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 021272EBB for ; Thu, 31 Jul 2014 22:01:07 +0000 (UTC) Received: by mail-lb0-f176.google.com with SMTP id u10so2515780lbd.21 for ; Thu, 31 Jul 2014 15:01:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:cc:content-type; bh=/zsz+ErfJgVTvNQLRfxwnpwYm6lDPFZaDW+zUXHjYOA=; b=p1L12sZgQOQmQHPf2XcBtXWS4JSTwY2HFPbdhDaGHQztJygGM0JBlXIP7tI4yMz8kj HMOYP7uBMghrNvDwPYkukf1q/rN1JyetZ8+US21ykYUXdsJEM7A9sxgb23P2wPAXLfDk LL7DL0iFMBWhQBHX/nUkE72KEkcqA1IaZOe8TI8aCuOcXeFkqtDYVTiYzobA3JFS/lcr HX6cAhE3Vuqi42sEOohVKejddEcJs3j104TSX7dOJ9oM/gIXX4JuMyZdMUZ1pcUmn8b2 x+ZIwC9eM9W9pPHXIgVWym0lz5aQs4q1o6Vg5Qnw3/LjL9zbF4Bmr5X4oemL2ecodXp5 fZdQ== MIME-Version: 1.0 X-Received: by 10.112.137.136 with SMTP id qi8mr1015247lbb.41.1406844065920; Thu, 31 Jul 2014 15:01:05 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.197.107 with HTTP; Thu, 31 Jul 2014 15:01:05 -0700 (PDT) Date: Thu, 31 Jul 2014 15:01:05 -0700 X-Google-Sender-Auth: OGM2FfaK46JHQW0jPQrTyOqe8jI Message-ID: Subject: Need new release of Kyua From: Craig Rodrigues To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: "freebsd-testing@freebsd.org" , kyua-discuss@googlegroups.com X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 22:01:08 -0000 Julio, Can you do a new release of kyua, which has the "kyua report-junit" support and Garrett's TAP fixes, and update the FreeBSD port and Ubuntu packages? I want to use kyua for a bunch of new tests for something I am working on, and having the junit support is a big must-have. Also, being able to install from packages on FreeBSD and Linux systems will make things easier for setting up a lot of test nodes. I can "configure; make; make intall; create package" myself, but that is a pain in the ass. :) Thanks. -- Craig From owner-freebsd-testing@FreeBSD.ORG Thu Jul 31 22:19:27 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 00146C9E; Thu, 31 Jul 2014 22:19:26 +0000 (UTC) Received: from mail-ig0-x22b.google.com (mail-ig0-x22b.google.com [IPv6:2607:f8b0:4001:c05::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BBD0520B3; Thu, 31 Jul 2014 22:19:26 +0000 (UTC) Received: by mail-ig0-f171.google.com with SMTP id l13so430026iga.10 for ; Thu, 31 Jul 2014 15:19:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=rOZaq0/DXce4KC1g92JQCclYFyOi0Iw5QNUrJn6ir1U=; b=Xb/7vwEDE1YYocJwK03SDZ+UpaAchl6D9snC+J+exjd5Vl9O5K4SgZYOL8IX5SEK3d AA+6+LrjO+44dsgwvTVRhGHoUUi3Ec0mVMpSJhEogj/nCva3wDY9kgDwbKYQBCxKj8Rg Y58BIoGGYMZHs0zztludv9NQG8WqxZir1ztxcDh4Kn0gxX1/0VoNA4eCzlSCXmVufpom /+Ij/ClOFlWDnEWGCox3PBEiJQNaBQIaq3kX13A3lREvo9r2vWZzG4kxiSpCWqPjIt5F Vlo5U2FlPw3OM4rGF6Ft22aCsEJ2Pwh+doblzm8HHypDHy2X+UCSkyGG0i0LySv5yBhG TAfg== MIME-Version: 1.0 X-Received: by 10.50.176.202 with SMTP id ck10mr1210128igc.2.1406845166150; Thu, 31 Jul 2014 15:19:26 -0700 (PDT) Received: by 10.50.213.102 with HTTP; Thu, 31 Jul 2014 15:19:26 -0700 (PDT) In-Reply-To: References: Date: Thu, 31 Jul 2014 15:19:26 -0700 Message-ID: Subject: Re: Need new release of Kyua From: Garrett Cooper To: Craig Rodrigues Content-Type: text/plain; charset=UTF-8 Cc: Julio Merino , "freebsd-testing@freebsd.org" , kyua-discuss@googlegroups.com X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 22:19:27 -0000 Hi Craig! On Thu, Jul 31, 2014 at 3:01 PM, Craig Rodrigues wrote: > Julio, > > Can you do a new release of kyua, which has the "kyua report-junit" support > and Garrett's TAP fixes, and update the FreeBSD port and Ubuntu packages? I just wanted to comment on this really quickly. My group (EMC / Isilon) is working on a CCLA with Google so we can contribute back to the Kyua project. > I want to use kyua for a bunch of new tests for something I am working on, > and having > the junit support is a big must-have. Also, being able to install from > packages > on FreeBSD and Linux systems will make things easier for setting up a lot > of test > nodes. > > I can "configure; make; make intall; create package" myself, but that is a > pain in the ass. :) You could download the port for FreeBSD I created from my GitHub fork: https://github.com/yaneurabeya/freebsd-ports/tree/kyua-testers-tap-fixes/devel/kyua-testers . The sources these patches are based off of are still available in my kyua fork too: https://github.com/yaneurabeya/kyua/tree/parse-all-the-tap-things . This definitely isn't ideal, but until the legal stuff has been cleared between EMC and Google, it's the best route forward. Thanks :), -Garrett From owner-freebsd-testing@FreeBSD.ORG Thu Jul 31 22:32:50 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D13FE37A for ; Thu, 31 Jul 2014 22:32:50 +0000 (UTC) Received: from mail-qg0-f52.google.com (mail-qg0-f52.google.com [209.85.192.52]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8FB36227A for ; Thu, 31 Jul 2014 22:32:50 +0000 (UTC) Received: by mail-qg0-f52.google.com with SMTP id f51so4676086qge.39 for ; Thu, 31 Jul 2014 15:32:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=EVjcJm483N0gBtHdR1pD32aUyXttoFvKQALSpqUGaQc=; b=PLAzk/MJPfyO1ZMMUNcHcJy0kM9zm0g6fywdhSseexw8p3qg0ZuRiIZo9eNjVa8UrG 0kkIVVwYbH7kw4J2yY1afG9KDABZVFCBI+idXmJijFsJOotPxtzsj8iYMTsMPddtNggr Lkho0vg8t7DZF+1pauo4bxYY2ILN0FcBoHCypC7+fqBbKnRWC3YH1QEw52acOL0Ys5Hq OBVOIro3LFezwkMDnVyVr1L3yB1Ey0x/Dgd803bQz8svbppv9oRYg5IE2lymvd49MNdA wHV/6LLV+UwfWO5tJEdT3Q5d86Oi3zX87Y8E0IkqkmNCBsrTNwzsF4QUPY0xMN0r2WYc nKiw== X-Gm-Message-State: ALoCoQndtiyUFFiE/IEN5YbIFremx+tqatBWq1tNwX4YSemVOqUPOwZWQ55lEHkW6CPdfyKUECIt X-Received: by 10.224.135.2 with SMTP id l2mr2167187qat.37.1406845968455; Thu, 31 Jul 2014 15:32:48 -0700 (PDT) MIME-Version: 1.0 Received: by 10.96.83.99 with HTTP; Thu, 31 Jul 2014 15:32:28 -0700 (PDT) X-Originating-IP: [2620:0:1008:100a:e1c4:4f33:3628:e85b] In-Reply-To: References: From: Julio Merino Date: Thu, 31 Jul 2014 15:32:28 -0700 Message-ID: Subject: Re: Need new release of Kyua To: kyua-discuss@googlegroups.com Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 22:32:50 -0000 [ Please do not use my personal address (removed) for open-source related email. The freebsd.org address is cool though. ] On Thu, Jul 31, 2014 at 3:01 PM, Craig Rodrigues wrote: > Julio, > > Can you do a new release of kyua, which has the "kyua report-junit" support > and Garrett's TAP fixes, and update the FreeBSD port and Ubuntu packages? Yes but I want to get the 'remove-actions' branch into the next release. I'm working quite hard this week on getting that finished and it's almost ready for integration. Expect new packages soon. But... Ubuntu packages? Did you mean Fedora or am I missing anything? > I want to use kyua for a bunch of new tests for something I am working on, > and having > the junit support is a big must-have. Also, being able to install from > packages > on FreeBSD and Linux systems will make things easier for setting up a lot of > test > nodes. > > I can "configure; make; make intall; create package" myself, but that is a > pain in the ass. :) Right, and rolling out a Kyua release is a pain in the ass as well -- hence why they are so infrequent. I have a plan to make that much easier but haven't finished drafting it for publication yet :-P From owner-freebsd-testing@FreeBSD.ORG Thu Jul 31 22:53:51 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 04AAD8E4 for ; Thu, 31 Jul 2014 22:53:51 +0000 (UTC) Received: from mail-la0-x233.google.com (mail-la0-x233.google.com [IPv6:2a00:1450:4010:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 811BA25ED for ; Thu, 31 Jul 2014 22:53:50 +0000 (UTC) Received: by mail-la0-f51.google.com with SMTP id pn19so2540067lab.24 for ; Thu, 31 Jul 2014 15:53:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=0B7YDS1VS4AItw3tpTjVgTeHnPjMMgFfegp6IabsvHE=; b=Zr9FlASkYSAEgL72Oum3C6+P8yhV8/wL6LXIKYhrvT9LwlwU+0X3e186anRzfpWHIv +Xvimq/VDWQZq4vW9DQmIsYFzJYgd/7sgimTr7whj0FZgd82WXRcq6f7hsyr5gdDd20B ACyUA0XpJkzUWhbmfDIlzYIhox64rayXVlt62T0D5il1xUATZkDsC8Wv3eZFPXjP6ZrK 1KDsMbGowImKDq9hAmyZvGCk2DMmjnWpKFg8S0mWD9DRIdk+PemWUGCh81DK7iveudLX b90eQRiYFqdVj4m6pgBnetF5MDp1x197WrE6/pkq9pKXI531GVpLanMCGdbWVMNoHgYi Ha/A== MIME-Version: 1.0 X-Received: by 10.112.72.3 with SMTP id z3mr1237090lbu.30.1406847228433; Thu, 31 Jul 2014 15:53:48 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.197.107 with HTTP; Thu, 31 Jul 2014 15:53:48 -0700 (PDT) In-Reply-To: References: Date: Thu, 31 Jul 2014 15:53:48 -0700 X-Google-Sender-Auth: gqnCE9oItRMAOiGlzTdCmuTWYyk Message-ID: Subject: Re: Need new release of Kyua From: Craig Rodrigues To: kyua-discuss@googlegroups.com Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 31 Jul 2014 22:53:51 -0000 On Thu, Jul 31, 2014 at 3:32 PM, Julio Merino wrote: > > On Thu, Jul 31, 2014 at 3:01 PM, Craig Rodrigues > wrote: > > Julio, > > > > Can you do a new release of kyua, which has the "kyua report-junit" > support > > and Garrett's TAP fixes, and update the FreeBSD port and Ubuntu packages? > > Yes but I want to get the 'remove-actions' branch into the next > release. I'm working quite hard this week on getting that finished and > it's almost ready for integration. Expect new packages soon. > OK, that sounds fine. If you have new packages available in the next 2 weeks or so, that would be great, since I am starting a bunch of stuff. > > But... Ubuntu packages? Did you mean Fedora or am I missing anything? > Yes, I meant Fedora. It would be nice if Ubuntu packages of Kyua were available also, but that's a separate item. -- Craig