From owner-freebsd-testing@FreeBSD.ORG Fri Jul 25 19:16:55 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 28B2BF5A for ; Fri, 25 Jul 2014 19:16:55 +0000 (UTC) Received: from mail-wi0-x22c.google.com (mail-wi0-x22c.google.com [IPv6:2a00:1450:400c:c05::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B5C89239F for ; Fri, 25 Jul 2014 19:16:54 +0000 (UTC) Received: by mail-wi0-f172.google.com with SMTP id n3so1492875wiv.5 for ; Fri, 25 Jul 2014 12:16:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=vqncrcYJSiuNqyfdSSxT6G8CaR9jax3LuGVUaiGqa48=; b=aH9BN79JXirsY+PhF+N4x+T+zNCSkkbXitSObGjQsDRk3TsvvcHtfjpCZN6BmBamtG WJvR7KypxKhnnEtFxoGoGJTApNF2Qw/bqNArF+n7atMmpeBhMQnO8vdpaqBrgufR/+Tq XGXCQipn4rVsEG68XxDuw42dzRwWhVJW3T6hMK2SCDGa5aNslVJBL/XaRO6lUI8b+Yfz EdJERm/qo/9kheiEw4N3AnaTIVmrMCKoaBRWkh/ANejpySZNZ3cMi5f/vrJsN65aAo/u D1stOEUa0U3Jbo6Iju+l192sO/aRRvMFMxB57azgkZsvQ9Xw6z5yS1c1NhybGWOCP6Tg /Etg== MIME-Version: 1.0 X-Received: by 10.180.39.33 with SMTP id m1mr7976653wik.82.1406315812454; Fri, 25 Jul 2014 12:16:52 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.203.134 with HTTP; Fri, 25 Jul 2014 12:16:52 -0700 (PDT) In-Reply-To: <5ee51891.12b049e9@fabiankeil.de> References: <5ee51891.12b049e9@fabiankeil.de> Date: Fri, 25 Jul 2014 13:16:52 -0600 X-Google-Sender-Auth: 7KsEStRZ7rTibBJSrVOib7XVwKs Message-ID: Subject: Re: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details From: Alan Somers To: Fabian Keil Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 19:16:55 -0000 On Fri, Jul 25, 2014 at 12:55 PM, Fabian Keil wrote: > I'm currently evaluating using kyua to test a FreeBSD-specific > shell script, however I'm having trouble analysing failing tests. > > For example, using: > > | #!/usr/bin/env atf-sh > | > | atf_test_case failing_test > | failing_test_body() { > | atf_check -s exit:1 -o empty -e empty 'true' > | } > | > | atf_init_test_cases() { > | atf_add_test_case failing_test > | } > > I get: > > | fk@r500 ~/git/zogftw/tests $kyua --loglevel debug test -k Kyuafile exam= ple:failing_test > | example:failing_test -> failed: atf-check failed; see the output of t= he test for details [0.150s] > | > | 0/1 passed (1 failed) > | Committed action 21 > > The test failure obviously is expected, but I'm missing the referenced > "output of the test". The relevant log excerpt: > > ######## > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO test_cases (test_program_id, name, metadata_id) VALUES (:test= _program_id, :name, :metadata_id) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: example:failing_= test -> > 20140725-183911 D 71935 utils/process/child.cpp:350: Spawned process 7195= 0: stdout=3D/tmp/kyua.7ESBAU/stdout.txt, stderr=3D/tmp/kyua.7ESBAU/stderr.t= xt > 20140725-183911 D 71935 utils/process/child.cpp:167: Executing /usr/local= /libexec/kyua-atf-tester -t300 test -vhas.cleanup=3Dfalse -vunprivileged-us= er=3Dtests /home/fk/git/zogftw/tests/example failing_test /tmp/kyua.7ESBAU/= result.txt > 20140725-183911 D 71935 utils/process/child.cpp:145: Waiting for pid=3D71= 950 > 20140725-183911 D 71935 store/transaction.cpp:896: Storing __STDOUT__ (/t= mp/kyua.7ESBAU/stdout.txt) of test case 42 > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO files (contents) VALUES (:contents) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO test_case_files (test_case_id, file_name, file_id) VALUES (:t= est_case_id, :file_name, :file_id) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 D 71935 store/transaction.cpp:896: Storing __STDERR__ (/t= mp/kyua.7ESBAU/stderr.txt) of test case 42 > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO files (contents) VALUES (:contents) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO test_case_files (test_case_id, file_name, file_id) VALUES (:t= est_case_id, :file_name, :file_id) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO test_results (test_case_id, result_type, = result_reason, start_time, end_time) VALUES= (:test_case_id, :result_type, :result_reason, :start_time, :end_ti= me) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: failed: atf-chec= k failed; see the output of the test for details [0.150s] > 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: > 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: 0/1 passed (1 fa= iled) > 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: Committed action= 21 > 20140725-183911 I 71935 cli/main.cpp:326: Clean exit with code 1 > ######## > > I'm looking for something like this: > > | fk@r500 ~ $/usr/local/libexec/atf-check -s exit:1 -o empty -e empty 'tr= ue' > | Executing command [ true ] > | Fail: incorrect exit status: 0, expected: 1 > | stdout: > | > | stderr: > | > > I didn't read all the docs available yet and I may be missing something, > but seeing test failures is something I'd expect to work out of the box. > Can someone familiar with kyua confirm that this is working as expected? > > I'm currently using: > > fk@r500 ~ $uname -a > FreeBSD r500.local 11.0-CURRENT FreeBSD 11.0-CURRENT #651 r269009+6d12a21= (fk): Wed Jul 23 17:49:07 CEST 2014 fk@r500.local:/usr/obj/usr/src/sys/= ZOEY amd64 > fk@r500 ~ $pkg info kyua\* > kyua-20140215,1 > kyua-atf-compat-0.2_2 > kyua-cli-0.8_3 > kyua-testers-0.2_3 > > Fabian The test output is stored in Kyua's sqlite database. There are two ways to see it: 1) Run " kyua report-html" 2) Sometimes its easier just to rerun the testcase in debug mode. Do "kyua debug example:failing_test" -Alan