From owner-freebsd-testing@FreeBSD.ORG Fri Jul 25 18:56:07 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DF3EB56A for ; Fri, 25 Jul 2014 18:56:07 +0000 (UTC) Received: from smtprelay02.ispgateway.de (smtprelay02.ispgateway.de [80.67.31.40]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 772482188 for ; Fri, 25 Jul 2014 18:56:06 +0000 (UTC) Received: from [78.35.187.212] (helo=fabiankeil.de) by smtprelay02.ispgateway.de with esmtpsa (SSLv3:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1XAker-0004E9-Lm for freebsd-testing@freebsd.org; Fri, 25 Jul 2014 20:55:57 +0200 Date: Fri, 25 Jul 2014 20:55:59 +0200 From: Fabian Keil To: Subject: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details Message-ID: <5ee51891.12b049e9@fabiankeil.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/9OaCUxQFeVDQVhxqf1DcW4i"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 18:56:07 -0000 --Sig_/9OaCUxQFeVDQVhxqf1DcW4i Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable I'm currently evaluating using kyua to test a FreeBSD-specific shell script, however I'm having trouble analysing failing tests. For example, using: | #!/usr/bin/env atf-sh | | atf_test_case failing_test | failing_test_body() { | atf_check -s exit:1 -o empty -e empty 'true' | } | | atf_init_test_cases() { | atf_add_test_case failing_test | } I get: | fk@r500 ~/git/zogftw/tests $kyua --loglevel debug test -k Kyuafile exampl= e:failing_test | example:failing_test -> failed: atf-check failed; see the output of the= test for details [0.150s] | | 0/1 passed (1 failed) | Committed action 21 The test failure obviously is expected, but I'm missing the referenced "output of the test". The relevant log excerpt: ######## 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO test_cases (test_program_id, name, metadata_id) VALUES (:test_p= rogram_id, :name, :metadata_id) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: example:failing_te= st -> =20 20140725-183911 D 71935 utils/process/child.cpp:350: Spawned process 71950:= stdout=3D/tmp/kyua.7ESBAU/stdout.txt, stderr=3D/tmp/kyua.7ESBAU/stderr.txt 20140725-183911 D 71935 utils/process/child.cpp:167: Executing /usr/local/l= ibexec/kyua-atf-tester -t300 test -vhas.cleanup=3Dfalse -vunprivileged-user= =3Dtests /home/fk/git/zogftw/tests/example failing_test /tmp/kyua.7ESBAU/re= sult.txt 20140725-183911 D 71935 utils/process/child.cpp:145: Waiting for pid=3D71950 20140725-183911 D 71935 store/transaction.cpp:896: Storing __STDOUT__ (/tmp= /kyua.7ESBAU/stdout.txt) of test case 42 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO files (contents) VALUES (:contents) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO test_case_files (test_case_id, file_name, file_id) VALUES (:tes= t_case_id, :file_name, :file_id) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 D 71935 store/transaction.cpp:896: Storing __STDERR__ (/tmp= /kyua.7ESBAU/stderr.txt) of test case 42 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO files (contents) VALUES (:contents) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO test_case_files (test_case_id, file_name, file_id) VALUES (:tes= t_case_id, :file_name, :file_id) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO test_results (test_case_id, result_type, = result_reason, start_time, end_time) VALUES (= :test_case_id, :result_type, :result_reason, :start_time, :end_time) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: failed: atf-check = failed; see the output of the test for details [0.150s] 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout:=20 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: 0/1 passed (1 fail= ed) 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: Committed action 21 20140725-183911 I 71935 cli/main.cpp:326: Clean exit with code 1 ######## I'm looking for something like this: | fk@r500 ~ $/usr/local/libexec/atf-check -s exit:1 -o empty -e empty 'true' | Executing command [ true ] | Fail: incorrect exit status: 0, expected: 1 | stdout: | | stderr: | I didn't read all the docs available yet and I may be missing something, but seeing test failures is something I'd expect to work out of the box. Can someone familiar with kyua confirm that this is working as expected? I'm currently using: fk@r500 ~ $uname -a FreeBSD r500.local 11.0-CURRENT FreeBSD 11.0-CURRENT #651 r269009+6d12a21(f= k): Wed Jul 23 17:49:07 CEST 2014 fk@r500.local:/usr/obj/usr/src/sys/ZO= EY amd64 fk@r500 ~ $pkg info kyua\* kyua-20140215,1 kyua-atf-compat-0.2_2 kyua-cli-0.8_3 kyua-testers-0.2_3 Fabian --Sig_/9OaCUxQFeVDQVhxqf1DcW4i Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlPSqD8ACgkQBYqIVf93VJ2hrQCgphDcCWsi0L2w/J/mybeEHYae CtgAn3hVPRzcSDRIGi99+00ZRc3t12tG =tKn1 -----END PGP SIGNATURE----- --Sig_/9OaCUxQFeVDQVhxqf1DcW4i--