From owner-freebsd-testing@FreeBSD.ORG Mon Jul 28 16:11:55 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E5E1310B; Mon, 28 Jul 2014 16:11:54 +0000 (UTC) Received: from mail-wi0-x22b.google.com (mail-wi0-x22b.google.com [IPv6:2a00:1450:400c:c05::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B2872F0C; Mon, 28 Jul 2014 16:11:54 +0000 (UTC) Received: by mail-wi0-f171.google.com with SMTP id hi2so4688611wib.4 for ; Mon, 28 Jul 2014 09:11:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=tYOZnOBMaI7M+zinnlQus0U3Hc9vcvhlBmDxMvv7j9w=; b=yP5YNHlZL9M+D0MZWfWOGbbKOoEFnoJTgjkUuY6pguGfVJM1WpRhZND1w6CScnp4VS 687bXyyEQem3/Jo/0YWyjtV88fLSblTickYQIwfXFU8xTUvs9ZHcAuVxuoIL6rT0FMrR zwz+WtTaYeSM2JL/+jW8ST6Y/fcBGMSQi9riE1GYWFm9+8sB3HGIT0rFSlb6SuBVhO3m Utq+ZegoNqfYNxSofyaTwJLKEF3LjQEBGQC+6WeUOI+BKVGOdGMdwcSr5tvGCqlPAokV USSKu+AvY6WWkZHdC+DxFtePYBuzprYmuWcjG8q1TVyQfQdpUke90YLeHhesO/W1eh/X +yQg== MIME-Version: 1.0 X-Received: by 10.180.39.139 with SMTP id p11mr16030582wik.50.1406563908918; Mon, 28 Jul 2014 09:11:48 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.203.134 with HTTP; Mon, 28 Jul 2014 09:11:48 -0700 (PDT) In-Reply-To: <450da50a.02ce9897@fabiankeil.de> References: <5ee51891.12b049e9@fabiankeil.de> <450da50a.02ce9897@fabiankeil.de> Date: Mon, 28 Jul 2014 10:11:48 -0600 X-Google-Sender-Auth: iQZbtUjyO3FvSkrqxS_hem-U2Dg Message-ID: Subject: Re: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details From: Alan Somers To: Fabian Keil Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jul 2014 16:11:55 -0000 On Sat, Jul 26, 2014 at 2:58 AM, Fabian Keil wrote: > Alan Somers wrote: > >> On Fri, Jul 25, 2014 at 12:55 PM, Fabian Keil >> wrote: >> > I'm currently evaluating using kyua to test a FreeBSD-specific >> > shell script, however I'm having trouble analysing failing tests. > [...] >> > I get: >> > >> > | fk@r500 ~/git/zogftw/tests $kyua --loglevel debug test -k Kyuafile example:failing_test >> > | example:failing_test -> failed: atf-check failed; see the output of the test for details [0.150s] >> > | >> > | 0/1 passed (1 failed) >> > | Committed action 21 >> > >> > The test failure obviously is expected, but I'm missing the referenced >> > "output of the test". The relevant log excerpt: > [...] >> > I'm looking for something like this: >> > >> > | fk@r500 ~ $/usr/local/libexec/atf-check -s exit:1 -o empty -e empty 'true' >> > | Executing command [ true ] >> > | Fail: incorrect exit status: 0, expected: 1 >> > | stdout: >> > | >> > | stderr: >> > | >> > >> > I didn't read all the docs available yet and I may be missing something, >> > but seeing test failures is something I'd expect to work out of the box. >> > Can someone familiar with kyua confirm that this is working as expected? > >> The test output is stored in Kyua's sqlite database. There are two >> ways to see it: >> 1) Run " kyua report-html" >> 2) Sometimes its easier just to rerun the testcase in debug mode. Do >> "kyua debug example:failing_test" > > Thanks a lot. > > Not showing test failures on the cli by default seems like a strange > design decision to me, but I guess I'll get used to it or write a > wrapper script that calls kyua debug for all tests. I don't recommend calling "kyua debug" for all tests. The best approach is to run "kyua test; kyua report-html ". The HTML report will show you the full stdout and stderr output of each failing test case. > > Fabian