From owner-freebsd-testing@FreeBSD.ORG Sun Nov 24 22:02:58 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 191E2EE5 for ; Sun, 24 Nov 2013 22:02:58 +0000 (UTC) Received: from mail-pb0-x22e.google.com (mail-pb0-x22e.google.com [IPv6:2607:f8b0:400e:c01::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E90C82853 for ; Sun, 24 Nov 2013 22:02:57 +0000 (UTC) Received: by mail-pb0-f46.google.com with SMTP id md12so4414048pbc.33 for ; Sun, 24 Nov 2013 14:02:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:date:cc :content-transfer-encoding:message-id:references:to; bh=Ry2fL4mvUsyfec3KdJSamSF99oCzBO3wV23gC91eI+8=; b=P8p/z4E5+TMcG3QGBRRgj2oKatqMVImQF5C4hunDEL/h9hXvNwzvQHAEvVjLqIZuwA gwhm9G2hcFHfKwjX4gvmg64Qsp1J2UkNmdpJYYjr1mOE+lSmZ4GBg8bcxNIZEhIAEvrE z5h3jvGuuKBKkeI4UsluMVFoOxgEzgbKaKXQ2umxoppobEDKdACbA/g81DI33AUcXLDz 7b8X+uGvQ7uiEifIGn4LyVSw+PP1ICQAxf3TJ9l3HEf8dTJ/NBl3yC39et/ehtvWYSpR P0pqZOg6b6IyaoatpxiaylNEuZPdJtw4/daIQ0eV/BKwKPtgA+qzHxT1QUL8+yEHlWPU 6jcg== X-Received: by 10.66.149.231 with SMTP id ud7mr24618490pab.8.1385330577403; Sun, 24 Nov 2013 14:02:57 -0800 (PST) Received: from [192.168.20.5] (c-50-181-163-89.hsd1.wa.comcast.net. [50.181.163.89]) by mx.google.com with ESMTPSA id qw8sm68465094pbb.27.2013.11.24.14.02.56 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 24 Nov 2013 14:02:56 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Fwd: Confusion over BSD.tests.dist From: Garrett Cooper Date: Sun, 24 Nov 2013 14:02:54 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: References: <83E6FED5-2E76-4E43-9547-C0DC1C90DBBD@gmail.com> To: Julio Merino X-Mailer: Apple Mail (2.1822) Cc: freebsd-testing@FreeBSD.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2013 22:02:58 -0000 Forwarding this message to freebsd-testing for others to have = context in terms of what I=92m proposing be done. Thanks! -Garrett Begin forwarded message: > From: Garrett Cooper > Subject: Confusion over BSD.tests.dist > Date: November 24, 2013 at 2:01:25 PM PST > To: Julio Merino >=20 > Hi Julio, > I think there=92s some confusion over what the intent of this = file was. I originally planned for it to be strictly for tests, and it = seems to have morphed into a tests and test infrastructure mtree file. = This is mostly ok, except for the fact that now if I specify TESTSBASE = !=3D /usr/tests, things will fail in the build. > I=92d like to fix this, but that means that the ATF/Kyua = infrastructure pieces really need to be moved out into their own mtree = file. Does that sound ok? > Thanks! > -Garrett From owner-freebsd-testing@FreeBSD.ORG Sun Nov 24 22:05:27 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 060F6B8 for ; Sun, 24 Nov 2013 22:05:27 +0000 (UTC) Received: from mail-lb0-f173.google.com (mail-lb0-f173.google.com [209.85.217.173]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 88F23287C for ; Sun, 24 Nov 2013 22:05:25 +0000 (UTC) Received: by mail-lb0-f173.google.com with SMTP id u14so2545824lbd.4 for ; Sun, 24 Nov 2013 14:05:18 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=7suHc1GmDX2apBZV9gIU+pTBHp7vfemHIsij/qECx50=; b=JfalSoP8AjCitwrnMjlyFEFVn0XaOu3da81BdmlZtsU7aQZiRXUhjNlDjELuAq+1NV vcl+OgVoEvQ7U6J0Nw4Kp56EkksYSb1hq8JUN09CEFteZgfi04HWCitPK1LFyZ+mf9yH nj+raw8hxqBB2vWjFId7qqE7ntFLJDgZWX6E1PdTHe0RqEPA6vpBGvgbMcPY+MJCNoNx k0v2awrhLduDW+k2wgM6nWTnHhNk24S8ToMgleKL50zkXCdpG7lKizaBIP6ZD3KVWHBz JYF4IgFPY1bldsefr5ZjSMFizDJenTwdJEXxorHVG1LGCcPApVIDwgojtkzTJLk61syM 7dkA== X-Gm-Message-State: ALoCoQm1rGZ0FRqprPBahOjZME8/my9iuN0iFMA/FN7hoaYdlo2gnRwNHv1eeIx1SVmcvhJp10L+ X-Received: by 10.112.218.74 with SMTP id pe10mr18608672lbc.19.1385330718461; Sun, 24 Nov 2013 14:05:18 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.168.65 with HTTP; Sun, 24 Nov 2013 14:04:58 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: References: <83E6FED5-2E76-4E43-9547-C0DC1C90DBBD@gmail.com> From: Julio Merino Date: Sun, 24 Nov 2013 17:04:58 -0500 Message-ID: Subject: Re: Confusion over BSD.tests.dist To: Garrett Cooper Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2013 22:05:27 -0000 On Sun, Nov 24, 2013 at 5:02 PM, Garrett Cooper wro= te: > Forwarding this message to freebsd-testing for others to have con= text in terms of what I=92m proposing be done. > Thanks! > -Garrett > > Begin forwarded message: > >> From: Garrett Cooper >> Subject: Confusion over BSD.tests.dist >> Date: November 24, 2013 at 2:01:25 PM PST >> To: Julio Merino >> >> Hi Julio, >> I think there=92s some confusion over what the intent of this file= was. I originally planned for it to be strictly for tests, and it seems to= have morphed into a tests and test infrastructure mtree file. This is most= ly ok, except for the fact that now if I specify TESTSBASE !=3D /usr/tests,= things will fail in the build. >> I=92d like to fix this, but that means that the ATF/Kyua infrastru= cture pieces really need to be moved out into their own mtree file. Does th= at sound ok? Is TESTSBASE supposed to be customizable? (And before answering that: are things like LIBDIR or INCLUDEDIR user-tunabale?) --=20 Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Sun Nov 24 22:08:53 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 07DD9FB for ; Sun, 24 Nov 2013 22:08:53 +0000 (UTC) Received: from mail-yh0-x235.google.com (mail-yh0-x235.google.com [IPv6:2607:f8b0:4002:c01::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C13012898 for ; Sun, 24 Nov 2013 22:08:52 +0000 (UTC) Received: by mail-yh0-f53.google.com with SMTP id b20so2314607yha.40 for ; Sun, 24 Nov 2013 14:08:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=mtXRmGFiAWweHFm02Or78O56JGwwk6KtTwPoItLljvI=; b=PgZcry1viutyrqUuKdp5VG3cbE5ebLr1l1fr3xRyHzmu09ymsUrorYmJ7MogfE0FmJ UyRROYp4I3JdQRU0Jtmh9KUn6ajgvZoIMP3qHKqtAjWBp/NwNoMO+jdFO8AHSW3Ro9UO nwIbHbCggEChkGETK60nc70ZeMPbq5reB4NJpbQAMlIV+qNSGsjbosVmbjuWUdVEqhPv ZftfCJBj67w01RUW5HqFoiAW0pQAPnpXAeYVUBOECJ1LoaoqeUzPkXwGW7LwA9ZsFymp ZjQg2hHCOnqeAdxl4u/M7zKbn7DVs1ButdsC8PEU2OL5GPwd8VQhby6z7EdPPn6mwM4c r9SQ== X-Received: by 10.236.90.65 with SMTP id d41mr9958712yhf.28.1385330932029; Sun, 24 Nov 2013 14:08:52 -0800 (PST) Received: from ?IPv6:2601:8:ab80:7d6:608b:3f59:7890:c8fe? ([2601:8:ab80:7d6:608b:3f59:7890:c8fe]) by mx.google.com with ESMTPSA id r64sm10310287yhc.23.2013.11.24.14.08.51 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 24 Nov 2013 14:08:51 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Re: Confusion over BSD.tests.dist From: Garrett Cooper In-Reply-To: Date: Sun, 24 Nov 2013 14:08:50 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: References: <83E6FED5-2E76-4E43-9547-C0DC1C90DBBD@gmail.com> To: Julio Merino X-Mailer: Apple Mail (2.1822) Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2013 22:08:53 -0000 On Nov 24, 2013, at 2:04 PM, Julio Merino wrote: > On Sun, Nov 24, 2013 at 5:02 PM, Garrett Cooper = wrote: >> Forwarding this message to freebsd-testing for others to have = context in terms of what I=92m proposing be done. >> Thanks! >> -Garrett >>=20 >> Begin forwarded message: >>=20 >>> From: Garrett Cooper >>> Subject: Confusion over BSD.tests.dist >>> Date: November 24, 2013 at 2:01:25 PM PST >>> To: Julio Merino >>>=20 >>> Hi Julio, >>> I think there=92s some confusion over what the intent of this = file was. I originally planned for it to be strictly for tests, and it = seems to have morphed into a tests and test infrastructure mtree file. = This is mostly ok, except for the fact that now if I specify TESTSBASE = !=3D /usr/tests, things will fail in the build. >>> I=92d like to fix this, but that means that the ATF/Kyua = infrastructure pieces really need to be moved out into their own mtree = file. Does that sound ok? >=20 > Is TESTSBASE supposed to be customizable? (And before answering that: It can be: # grep -r TESTSBASE share/mk share/mk/bsd.README:TESTSDIR Target directory relative to = ${TESTSBASE} for tests and share/mk/bsd.own.mk:TESTSBASE?=3D /usr/tests > are things like LIBDIR or INCLUDEDIR user-tunabale?) Those are user tunable too, but generally not tweaked, except when = dealing with packages that use bsd.*.mk (e.g. ports): # egrep --include \*.mk -r '^INCLUDEDIR|^LIBDIR' share/mk share/mk/bsd.own.mk:LIBDIR?=3D /usr/lib share/mk/bsd.own.mk:INCLUDEDIR?=3D /usr/include Cheers! -Garrett= From owner-freebsd-testing@FreeBSD.ORG Sun Nov 24 22:10:01 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BAB2110A for ; Sun, 24 Nov 2013 22:10:01 +0000 (UTC) Received: from mail-pd0-x22d.google.com (mail-pd0-x22d.google.com [IPv6:2607:f8b0:400e:c02::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 97EF1289E for ; Sun, 24 Nov 2013 22:10:01 +0000 (UTC) Received: by mail-pd0-f173.google.com with SMTP id p10so4244418pdj.32 for ; Sun, 24 Nov 2013 14:10:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:subject:date:message-id :cc:to:mime-version; bh=V9JYzziCZnM9XL3lkO779GB3qwbOUlMnO5XT1SpXd4Y=; b=CsvuMbYWrjAehz/UdAzoy8cxYFpG7AM8EjMKlyZwQii1oausa2Uc1UCIanvvupbzf9 xwK7p4hv5eR/j6hb7u4w6h5IS1030T/r8GrRebpP7LyvvNhTP2rUm8mafV4d1/tqjDI5 4U9eKJpAESHWKIG3FP74ZGzFXuQu5kg4zDjND5k394UqiJieP53Ym2jAPFEWV6YYDzKH 8AG4pZTRHNDU5xv7hLj5ThcwfQYvzCTkf2iQ333FLGQLnfuM2YSyZLwCoNEIAA+Gc9or UuNabdAdbasFvp/TqO/BblqT5pN3euyZDRPRZS4Iw4nZHmZCLNbaTUA1swSCPFCPZfmR QukQ== X-Received: by 10.66.122.40 with SMTP id lp8mr23890253pab.82.1385331001346; Sun, 24 Nov 2013 14:10:01 -0800 (PST) Received: from [192.168.20.5] (c-50-181-163-89.hsd1.wa.comcast.net. [50.181.163.89]) by mx.google.com with ESMTPSA id kd1sm77594018pab.20.2013.11.24.14.10.00 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 24 Nov 2013 14:10:00 -0800 (PST) From: Garrett Cooper Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: Adding new "test interfaces" to work with Kyua? Date: Sun, 24 Nov 2013 14:09:59 -0800 Message-Id: <82E5F5E2-EA2C-46D2-9068-C419A834686C@gmail.com> To: Julio Merino Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) X-Mailer: Apple Mail (2.1822) Cc: freebsd-testing@FreeBSD.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2013 22:10:01 -0000 Trying to reconcile my diffs and get things sane again, I = realize that due to refactoring (and especially after I installed = devel/kyua) that prove.test.mk no longer functions for me off my branch. = Are there directions for how I can program in a new test interface so I = can integrate prove testing into Kyua? Thanks! -Garrett= From owner-freebsd-testing@FreeBSD.ORG Sun Nov 24 22:24:33 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3D7423FD for ; Sun, 24 Nov 2013 22:24:33 +0000 (UTC) Received: from mail-la0-f43.google.com (mail-la0-f43.google.com [209.85.215.43]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id C0543293B for ; Sun, 24 Nov 2013 22:24:32 +0000 (UTC) Received: by mail-la0-f43.google.com with SMTP id n7so2388899lam.2 for ; Sun, 24 Nov 2013 14:24:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=SKxsS11w5v+uqeXoDBcThZpg07S6ZZRVJ0NbovhwA1k=; b=ktHBhjB1pG6Cftua3+LzdIieZ34lVynHxBJzX3ruLPxdj1G3EAL1ZAbFYZa2EAHukE 9sFdW79eZpNKeRQhmHmZgJcc4PELGIvphMAwLPTW9DvPI316S+Y3GQcM9qFFDORnJh2S apObqJaJZKq3lyj3VFL858qxZQj+GUQattmdwNmkHQrw4o/faEHkVMS3PQYz+AEDs1WY mI/+9jxjPLzCZp73mzKYgHXJtQ+MkowgWKgr4fbUxHi1NOsltHf7904/fTtmL0GE9aSH 93zRAWrEA+YsUVdt3E5N/04gM65o7biKXcM3iqym0uLDFwBaiMKWXcwP1Xedgsm7tOCK EKkA== X-Gm-Message-State: ALoCoQm5fpTfZFMUm7p7OXmpENOoNv/TnmSa3RBi3xNTuTaY09Foz55gNKOPvKGqIZwEjmjC3eup X-Received: by 10.153.7.106 with SMTP id db10mr12273161lad.9.1385331864620; Sun, 24 Nov 2013 14:24:24 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.168.65 with HTTP; Sun, 24 Nov 2013 14:24:04 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: References: <83E6FED5-2E76-4E43-9547-C0DC1C90DBBD@gmail.com> From: Julio Merino Date: Sun, 24 Nov 2013 17:24:04 -0500 Message-ID: Subject: Re: Confusion over BSD.tests.dist To: Garrett Cooper Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2013 22:24:33 -0000 On Sun, Nov 24, 2013 at 5:08 PM, Garrett Cooper wrote: > On Nov 24, 2013, at 2:04 PM, Julio Merino wrote: >> Is TESTSBASE supposed to be customizable? (And before answering that: > > It can be: > > # grep -r TESTSBASE share/mk > share/mk/bsd.README:TESTSDIR Target directory relative to ${TESTSBASE} for tests and > share/mk/bsd.own.mk:TESTSBASE?= /usr/tests I know it _can_ be, but the question is: do we want to support that as a use case? I'm not sure why anybody would want to move /usr/tests anywhere else. If there is no real reason other than "just because", I don't think the build system should make any accommodations to make it trivial. (Because if it's trivial, people _will_ move it and when things break, it's one more thing to support in bug reports.) >> are things like LIBDIR or INCLUDEDIR user-tunabale?) > > Those are user tunable too, but generally not tweaked, except when dealing with packages that use bsd.*.mk (e.g. ports): > > # egrep --include \*.mk -r '^INCLUDEDIR|^LIBDIR' share/mk > share/mk/bsd.own.mk:LIBDIR?= /usr/lib > share/mk/bsd.own.mk:INCLUDEDIR?= /usr/include Right, so they are tunable when bsd.*.mk are abused to build things from ports (and in that case mtree doesn't apply). But I believe they are not tunable to tell the base system where the libraries or headers should be; if they were, I'm pretty sure things would break in obscure ways and it'd be a "support" headache. -- Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Sun Nov 24 22:30:02 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3B9B645E for ; Sun, 24 Nov 2013 22:30:02 +0000 (UTC) Received: from mail-lb0-f178.google.com (mail-lb0-f178.google.com [209.85.217.178]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id BE524295A for ; Sun, 24 Nov 2013 22:30:01 +0000 (UTC) Received: by mail-lb0-f178.google.com with SMTP id c11so2558982lbj.23 for ; Sun, 24 Nov 2013 14:29:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=Frb5gadeB98qv1dhrWEGLKxg+zAv39ftm/l4ydRTEX4=; b=f7NSdZLcvGa9EfFR1YkxicnDMNqlTVrIlxNgFqb+QexWVSr+XeW66EuVJjcOzvUoBq 5DeNn9KnduzlrMgMFShwfhWtrKslXQ49dwPjVIGpVFwOWhq2zPKZbpEvbr0be/wqQMjx NdhpX2+K0T2BuvKPR0uR2nWRx3O6va8ZJ6fF5KG76sw7POQH5dW5ZOLeaqlYbzygSXUK Yonl0zo/LE/2mrCjOLcqhXcHgGqjRnYXt6d2Yl+jJ9tDXa9GLCelN2zfhzhjTjpbi+hc d4DVjSO4uxUp2vXq2DtHEL9qektNt3MDOPS99uu9/EltN78SBBdqr/Du2aWQO+ME3tw8 d1dw== X-Gm-Message-State: ALoCoQkLipYWY4fgwaSt3gctGp9+ivECCfPYxMeZYAMXgHCclkAvawSW0rkveplqSuW46OC+2QNT X-Received: by 10.152.116.46 with SMTP id jt14mr3654lab.31.1385332193563; Sun, 24 Nov 2013 14:29:53 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.168.65 with HTTP; Sun, 24 Nov 2013 14:29:33 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: <82E5F5E2-EA2C-46D2-9068-C419A834686C@gmail.com> References: <82E5F5E2-EA2C-46D2-9068-C419A834686C@gmail.com> From: Julio Merino Date: Sun, 24 Nov 2013 17:29:33 -0500 Message-ID: Subject: Re: Adding new "test interfaces" to work with Kyua? To: Garrett Cooper Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2013 22:30:02 -0000 On Sun, Nov 24, 2013 at 5:09 PM, Garrett Cooper wro= te: > Trying to reconcile my diffs and get things sane again, I realize= that due to refactoring (and especially after I installed devel/kyua) that= prove.test.mk no longer functions for me off my branch. Are there directio= ns for how I can program in a new test interface so I can integrate prove t= esting into Kyua? What kind of interface do prove-based test programs have? Are they binaries/scripts that you can just run and assume that a 0 exit code means passed and 1 means failed? If yes, you'd just funnel them through the 'plain' interface and they should just work as a first cut. (I believe this approach works for most test programs out there.) But there are some details here: https://code.google.com/p/kyua/wiki/TestersDesign and more in the kyua-tester manual page and its references. That said... I'm not sure now if I have detailed the "protocol" in detail anywhere... and if not, that's something for me to start doing :P --=20 Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Sun Nov 24 22:34:31 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2D60C48E for ; Sun, 24 Nov 2013 22:34:31 +0000 (UTC) Received: from mail-yh0-x22d.google.com (mail-yh0-x22d.google.com [IPv6:2607:f8b0:4002:c01::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E6E772996 for ; Sun, 24 Nov 2013 22:34:30 +0000 (UTC) Received: by mail-yh0-f45.google.com with SMTP id v1so1275066yhn.4 for ; Sun, 24 Nov 2013 14:34:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=IZAz+zqZfwkRR9wHDFlctwDVMh5gCy9Mb5DzkOWA/2w=; b=R3Y0LdaoJXOH3/ADqgFDn3U8PBmp11dxtiChVaoj7MQIBgFNq6S7Aw4RsLUjoo2c/m J2gH5b+5p9RUY1ebSvNP+Jl2dCB8iPiPPp6F3FiRK7m3Sw26VQgExuISHJKdeyMR8CE9 GQzmRW65lUjUNfaRuHanVnk7JH0LK5L04+XiWMOBiYr8sb+j12xnFRDRWS+gymx+ZhPV 0Jg3dGbv0IVwxPY1DR3CirjUDYj5vmeEAcyIOkWRmkooQHmfqN6xHRQ0m3qUILkugHNz RVu3hn8OvIJOPaQ5z2qZobDXPYNdqf4oj8LcZqE8OJlKpVxrQz8oTi/p2QlyAHHiweYf as4w== X-Received: by 10.236.120.74 with SMTP id o50mr24489498yhh.45.1385332468903; Sun, 24 Nov 2013 14:34:28 -0800 (PST) Received: from ?IPv6:2601:8:ab80:7d6:608b:3f59:7890:c8fe? ([2601:8:ab80:7d6:608b:3f59:7890:c8fe]) by mx.google.com with ESMTPSA id m29sm70010611yho.14.2013.11.24.14.34.28 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 24 Nov 2013 14:34:28 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Re: Adding new "test interfaces" to work with Kyua? From: Garrett Cooper In-Reply-To: Date: Sun, 24 Nov 2013 14:34:26 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <751B8D9F-9A1B-46B5-80F8-CB9E44F98BFC@gmail.com> References: <82E5F5E2-EA2C-46D2-9068-C419A834686C@gmail.com> To: Julio Merino X-Mailer: Apple Mail (2.1822) Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2013 22:34:31 -0000 On Nov 24, 2013, at 2:29 PM, Julio Merino wrote: > On Sun, Nov 24, 2013 at 5:09 PM, Garrett Cooper = wrote: >> Trying to reconcile my diffs and get things sane again, I = realize that due to refactoring (and especially after I installed = devel/kyua) that prove.test.mk no longer functions for me off my branch. = Are there directions for how I can program in a new test interface so I = can integrate prove testing into Kyua? >=20 > What kind of interface do prove-based test programs have? Are they > binaries/scripts that you can just run and assume that a 0 exit code > means passed and 1 means failed? If yes, you'd just funnel them > through the 'plain' interface and they should just work as a first > cut. (I believe this approach works for most test programs out > there.) >=20 > But there are some details here: >=20 > https://code.google.com/p/kyua/wiki/TestersDesign >=20 > and more in the kyua-tester manual page and its references. It=92s a bit more complex than that. nose and prove both implement = collection and execution test features, meaning that it goes and figures = out what to execute (for nose it=92s a variety of parameters =97 = filename, mode, etc =97 that determine whether or not it=92s testable; = for prove it=92s all files with a .t extension), then executes it. I=92d like to take the error messages and make them more meaningful = though, because it doesn=92t help if something fails and I need to = diagnose why it failed, but lost the info I need to debug the problem... > That said... I'm not sure now if I have detailed the "protocol" in > detail anywhere... and if not, that's something for me to start doing > :P That would be nice :). Thanks! -Garrett= From owner-freebsd-testing@FreeBSD.ORG Sun Nov 24 22:39:57 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 93BBA5C4 for ; Sun, 24 Nov 2013 22:39:57 +0000 (UTC) Received: from mail-yh0-x230.google.com (mail-yh0-x230.google.com [IPv6:2607:f8b0:4002:c01::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 572B829C8 for ; Sun, 24 Nov 2013 22:39:57 +0000 (UTC) Received: by mail-yh0-f48.google.com with SMTP id f73so2325665yha.7 for ; Sun, 24 Nov 2013 14:39:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=15pBijMEt3N2ar+UD+0rf8gOF0RRXTHJtPdsI6pdHCM=; b=vmH3MsfrdWdE5bpfymuYaW9/7P8Ukt7f7vyHI9TpCwTqVMrWgvVvx3JD4c8JHBDibc +JOfEFRtCIWt+JbQnBLpK6pPkFm9etcNF6CjhV4C2HGiGLar/DrtROa2wZrx6pbaIxao ZV/B5zfsVXhnA7gaNRgNokI5T1yiTKzUQ0VYcDHoRaJDY/kGzVMUcdu4FOd5yGdvPO/C q4o60NlboXdQjdCmGBmSpIxPMSDJn31DRWlxhCYv2OF+PFYp9JPPD0VhN5I6N+gt6bWz LTMOVpz51PpY+C6miVzgkel09ZdMP/ATtRAhJw1CHZygkUlhcL5I+RFGh9cUTnYGPl2B Zojw== X-Received: by 10.236.85.97 with SMTP id t61mr24355038yhe.11.1385332796572; Sun, 24 Nov 2013 14:39:56 -0800 (PST) Received: from ?IPv6:2601:8:ab80:7d6:608b:3f59:7890:c8fe? ([2601:8:ab80:7d6:608b:3f59:7890:c8fe]) by mx.google.com with ESMTPSA id h23sm13462008yhc.0.2013.11.24.14.39.55 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 24 Nov 2013 14:39:56 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Re: Confusion over BSD.tests.dist From: Garrett Cooper In-Reply-To: Date: Sun, 24 Nov 2013 14:39:54 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <500AA3D4-6910-406A-A093-1B925DF0612D@gmail.com> References: <83E6FED5-2E76-4E43-9547-C0DC1C90DBBD@gmail.com> To: Julio Merino X-Mailer: Apple Mail (2.1822) Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2013 22:39:57 -0000 On Nov 24, 2013, at 2:24 PM, Julio Merino wrote: > On Sun, Nov 24, 2013 at 5:08 PM, Garrett Cooper = wrote: >> On Nov 24, 2013, at 2:04 PM, Julio Merino wrote: >>> Is TESTSBASE supposed to be customizable? (And before answering = that: >>=20 >> It can be: >>=20 >> # grep -r TESTSBASE share/mk >> share/mk/bsd.README:TESTSDIR Target directory relative to = ${TESTSBASE} for tests and >> share/mk/bsd.own.mk:TESTSBASE?=3D /usr/tests >=20 > I know it _can_ be, but the question is: do we want to support that as > a use case? I'm not sure why anybody would want to move /usr/tests > anywhere else. If there is no real reason other than "just because", > I don't think the build system should make any accommodations to make > it trivial. (Because if it's trivial, people _will_ move it and when > things break, it's one more thing to support in bug reports.) Fair enough. The problem is that there are some organizations (like the = one I just left =97 EMC) that use other paths for testing (i.e. not = /usr/tests) because adjusting existing infrastructure to match new stuff = is difficult, introduces unnecessary risk, and could break generic = tools. >>> are things like LIBDIR or INCLUDEDIR user-tunabale?) >>=20 >> Those are user tunable too, but generally not tweaked, except when = dealing with packages that use bsd.*.mk (e.g. ports): >>=20 >> # egrep --include \*.mk -r '^INCLUDEDIR|^LIBDIR' share/mk >> share/mk/bsd.own.mk:LIBDIR?=3D /usr/lib >> share/mk/bsd.own.mk:INCLUDEDIR?=3D /usr/include >=20 > Right, so they are tunable when bsd.*.mk are abused to build things > from ports (and in that case mtree doesn't apply). But I believe they > are not tunable to tell the base system where the libraries or headers > should be; if they were, I'm pretty sure things would break in obscure > ways and it'd be a "support" headache. It=92s not really abuse; there are also some packagers/third party = groups that implement bsd.*.mk properly with the intent to integrate = better into *BSD (by and large the Makefile snippets are consistent = between the BSDs in many cases, so there=92s some degree of = portability), in part because the original upstream source may have done = such a shoddy job writing configure scripts or were so Linux centric = that it=92s better to write something simple from scratch for an initial = port. Cheers! -Garrett= From owner-freebsd-testing@FreeBSD.ORG Mon Nov 25 01:40:05 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6A63A39A for ; Mon, 25 Nov 2013 01:40:05 +0000 (UTC) Received: from mail-lb0-f177.google.com (mail-lb0-f177.google.com [209.85.217.177]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id EA99621F3 for ; Mon, 25 Nov 2013 01:40:04 +0000 (UTC) Received: by mail-lb0-f177.google.com with SMTP id w7so2620336lbi.36 for ; Sun, 24 Nov 2013 17:39:56 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=Py8yigIf1EsDDw2nJCQabutBpkF4N4EQzU67ZzYRJ/w=; b=kxRfjMRZdoiisi3Ufe177cQAIGiMyLX6fX7jLmSyUEozuKdJgi0tI/rC6pFEzOvPkE i4pra/F+Ev2UhikszJr7zXdJTykICuLIaCA2nkc5ZIIoETbx7yFMytZpgg8nxIGAWFyl I+9M+s5toHjd/EWHi7gexNzMDsKRB2JRYPE9W7C5OZ43HBXDEfrihcSVSvTPgNC3Thp8 oLziy0Rs1YmiBe89mni9A9fesgLgU71SqYMZRWwR8suQX1J8v51XqiUG7BFaQaSo/efS GdsME1IFTVfcuKixX2pvgPqpLxVIIPzVKv+6tc4r9+QFL13nMFdgmcY8s2WsU/Joji6+ v4+g== X-Gm-Message-State: ALoCoQkDuN1/DRn2OOOlPWz/N4S4kQm0kROQyTaj+DazsPHc9dbFcG0YqpubmIPyoPa4oALI27F+ X-Received: by 10.112.136.163 with SMTP id qb3mr19323929lbb.14.1385343596561; Sun, 24 Nov 2013 17:39:56 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.168.65 with HTTP; Sun, 24 Nov 2013 17:39:36 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: <500AA3D4-6910-406A-A093-1B925DF0612D@gmail.com> References: <83E6FED5-2E76-4E43-9547-C0DC1C90DBBD@gmail.com> <500AA3D4-6910-406A-A093-1B925DF0612D@gmail.com> From: Julio Merino Date: Sun, 24 Nov 2013 20:39:36 -0500 Message-ID: Subject: Re: Confusion over BSD.tests.dist To: Garrett Cooper Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2013 01:40:05 -0000 On Sun, Nov 24, 2013 at 5:39 PM, Garrett Cooper wro= te: > On Nov 24, 2013, at 2:24 PM, Julio Merino wrote: > >> On Sun, Nov 24, 2013 at 5:08 PM, Garrett Cooper = wrote: >>> On Nov 24, 2013, at 2:04 PM, Julio Merino wrote: >>>> Is TESTSBASE supposed to be customizable? (And before answering that: >>> >>> It can be: >>> >>> # grep -r TESTSBASE share/mk >>> share/mk/bsd.README:TESTSDIR Target directory relative to ${TESTSBAS= E} for tests and >>> share/mk/bsd.own.mk:TESTSBASE?=3D /usr/tests >> >> I know it _can_ be, but the question is: do we want to support that as >> a use case? I'm not sure why anybody would want to move /usr/tests >> anywhere else. If there is no real reason other than "just because", >> I don't think the build system should make any accommodations to make >> it trivial. (Because if it's trivial, people _will_ move it and when >> things break, it's one more thing to support in bug reports.) > > Fair enough. The problem is that there are some organizations (like the o= ne I just left =97 EMC) that use other paths for testing (i.e. not /usr/tes= ts) because adjusting existing infrastructure to match new stuff is difficu= lt, introduces unnecessary risk, and could break generic tools. Right... so that's the use case I was looking for: organization that "cannot" change existing infrastructure to match the /usr/tests path. I don't know the details of your previous company at all, but I'm wondering if the path is the only thing that matters or it's just one tiny detail among many? I understand the use case... so the question is: is it worth supporting it (going all the way through to support this and other customizations that such scheme may require)? (Personally I tend to think it's not, but I don't have a strong opinion as long as the proposed modifications are simple enough.) --=20 Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Mon Nov 25 02:01:54 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6995C608 for ; Mon, 25 Nov 2013 02:01:54 +0000 (UTC) Received: from mail-la0-f51.google.com (mail-la0-f51.google.com [209.85.215.51]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id E847D2307 for ; Mon, 25 Nov 2013 02:01:53 +0000 (UTC) Received: by mail-la0-f51.google.com with SMTP id ec20so2541124lab.10 for ; Sun, 24 Nov 2013 18:01:46 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=Cvr4WZqLQqG9KsGqIhpVHVEKPm7AviRBajFkrgzZl0I=; b=QjVeLXfABT3nTBUsRj2ypf6qRicCGSqmCjpNTzygv2RcHkNo6ZZDpjaFhh7HjgS9+b BMSf1u4E1LFgcOwcPUvmKdqCOoSe+DP4DDif4hBUupHbI3ZahNPTNFXPdMkvz1TnA/d7 Djso6y0oricw+4dI23epvQNTZv04Cz6bzEwklfIy6KaXwE5xhBgGb1nPB4xesL/7rHKe aZDEchHaseIERB3gnyhkgfTJ1aqKsxgQCh2iD63Phmuy74rU5+5mI9+N8y2z52guYQBZ RUucEdQo5zES5MqtcgKLxY/BT3s9iIBmt2547JjBiYhJCbaTO6SgS03xQ+L5mU/zgosu z9Nw== X-Gm-Message-State: ALoCoQlKMInHXXsBm6MXrPKYzOhLicHyD38T4s79t/kYyC8KabOVYrN7Zatshr8Xt8AOySRY3tNE X-Received: by 10.112.172.137 with SMTP id bc9mr19671289lbc.21.1385344510906; Sun, 24 Nov 2013 17:55:10 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.168.65 with HTTP; Sun, 24 Nov 2013 17:54:50 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: <751B8D9F-9A1B-46B5-80F8-CB9E44F98BFC@gmail.com> References: <82E5F5E2-EA2C-46D2-9068-C419A834686C@gmail.com> <751B8D9F-9A1B-46B5-80F8-CB9E44F98BFC@gmail.com> From: Julio Merino Date: Sun, 24 Nov 2013 20:54:50 -0500 Message-ID: Subject: Re: Adding new "test interfaces" to work with Kyua? To: Garrett Cooper Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2013 02:01:54 -0000 On Sun, Nov 24, 2013 at 5:34 PM, Garrett Cooper wro= te: > On Nov 24, 2013, at 2:29 PM, Julio Merino wrote: > >> On Sun, Nov 24, 2013 at 5:09 PM, Garrett Cooper = wrote: >>> Trying to reconcile my diffs and get things sane again, I realiz= e that due to refactoring (and especially after I installed devel/kyua) tha= t prove.test.mk no longer functions for me off my branch. Are there directi= ons for how I can program in a new test interface so I can integrate prove = testing into Kyua? >> >> What kind of interface do prove-based test programs have? Are they >> binaries/scripts that you can just run and assume that a 0 exit code >> means passed and 1 means failed? If yes, you'd just funnel them >> through the 'plain' interface and they should just work as a first >> cut. (I believe this approach works for most test programs out >> there.) >> >> But there are some details here: >> >> https://code.google.com/p/kyua/wiki/TestersDesign >> >> and more in the kyua-tester manual page and its references. > > It=92s a bit more complex than that. nose and prove both implement collec= tion and execution test features, meaning that it goes and figures out what= to execute (for nose it=92s a variety of parameters =97 filename, mode, et= c =97 that determine whether or not it=92s testable; for prove it=92s all f= iles with a .t extension), then executes it. I need to read about that a bit more then. Is there a web site for these tools? Their names are so generic that it's hard to find stuff. That said, I have started to take a look at the existing .t tests... and the majority of them seem to be just wrappers that execute a shell script. In these cases, I think we should just drop the .t and use the plain interface to execute the existing shell script. By pure coincidence, I had been working today on "porting" a couple of these tests; expect a patch for the proof of concept soon. > I=92d like to take the error messages and make them more meaningful thoug= h, because it doesn=92t help if something fails and I need to diagnose why = it failed, but lost the info I need to debug the problem... As long as these print the error messages to either stdout or stderr, they will be captured by Kyua and stored into the reports. --=20 Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Mon Nov 25 17:21:02 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9DA294E4 for ; Mon, 25 Nov 2013 17:21:02 +0000 (UTC) Received: from mail-la0-f42.google.com (mail-la0-f42.google.com [209.85.215.42]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2B07829D1 for ; Mon, 25 Nov 2013 17:21:01 +0000 (UTC) Received: by mail-la0-f42.google.com with SMTP id ec20so3271490lab.15 for ; Mon, 25 Nov 2013 09:21:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=b6ID2bn73fL+m+yG7JhBbHY28F0TwxKhJyXN50Cxzrg=; b=hBup9m06wzUIhXjdH+a6j0ZvnoHfRKI5JHmUzgyKbiFbL+83YXnRmDHOLv7Behpeod yMrxldtK07SabmsznG5e0XLnM5NWSSIfOh7q1O95a7hMf9zyhWAeF0oEo5xPI0bdIBy0 +qZtisisSPuKYzcDRX5k2bPgDwbw2VAN8G7BS4HIOO2MEI/qqZQbOCyxj9mwiqF2+va4 IZqhSZlpCEuEhPFa+gO5HoF4AanjeDyfte4c/DYmhs+nCUwJGX+2iIF9gJcc8zSndvG+ VsXtk04CwavednQJM6Q1qHxddKykDHHi4wbuiuSYUDmFtVgmvGruKw71DWWfR4awIs5+ o59A== X-Gm-Message-State: ALoCoQkHkKLHl8cXt2D/BlqKIGySjJZM0AhekwNAlZpI9NraqwWC9I0UY6Y25epS9u42UBMBELY1 X-Received: by 10.152.116.109 with SMTP id jv13mr2328103lab.30.1385399645848; Mon, 25 Nov 2013 09:14:05 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.133.74 with HTTP; Mon, 25 Nov 2013 09:13:45 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: <751B8D9F-9A1B-46B5-80F8-CB9E44F98BFC@gmail.com> References: <82E5F5E2-EA2C-46D2-9068-C419A834686C@gmail.com> <751B8D9F-9A1B-46B5-80F8-CB9E44F98BFC@gmail.com> From: Julio Merino Date: Mon, 25 Nov 2013 12:13:45 -0500 Message-ID: Subject: Re: Adding new "test interfaces" to work with Kyua? To: Garrett Cooper Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2013 17:21:02 -0000 On Sun, Nov 24, 2013 at 5:34 PM, Garrett Cooper wro= te: > On Nov 24, 2013, at 2:29 PM, Julio Merino wrote: > >> On Sun, Nov 24, 2013 at 5:09 PM, Garrett Cooper = wrote: >>> Trying to reconcile my diffs and get things sane again, I realiz= e that due to refactoring (and especially after I installed devel/kyua) tha= t prove.test.mk no longer functions for me off my branch. Are there directi= ons for how I can program in a new test interface so I can integrate prove = testing into Kyua? >> >> What kind of interface do prove-based test programs have? Are they >> binaries/scripts that you can just run and assume that a 0 exit code >> means passed and 1 means failed? If yes, you'd just funnel them >> through the 'plain' interface and they should just work as a first >> cut. (I believe this approach works for most test programs out >> there.) >> >> But there are some details here: >> >> https://code.google.com/p/kyua/wiki/TestersDesign >> >> and more in the kyua-tester manual page and its references. > > It=92s a bit more complex than that. nose and prove both implement collec= tion and execution test features, meaning that it goes and figures out what= to execute (for nose it=92s a variety of parameters =97 filename, mode, et= c =97 that determine whether or not it=92s testable; for prove it=92s all f= iles with a .t extension), then executes it. Bleh. Turns out that prove-based test programs do not return 1 on failure. Time to write a prove tester given that I have some spare time this week :-P --=20 Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Mon Nov 25 18:29:57 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 32B8279D for ; Mon, 25 Nov 2013 18:29:57 +0000 (UTC) Received: from mail-la0-f42.google.com (mail-la0-f42.google.com [209.85.215.42]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 44E3A2E3D for ; Mon, 25 Nov 2013 18:29:56 +0000 (UTC) Received: by mail-la0-f42.google.com with SMTP id ec20so3423883lab.29 for ; Mon, 25 Nov 2013 10:29:54 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc :content-type; bh=d+rOrBz3vFyR0kZO7eEb0bEjqqQDeIGaTrG/PJ6rpUY=; b=HcEhJjeSPYlN4QedY91qFxIw0tD2EcSX8z6fci4k2VEF7FdCHArmuzRXDcDbMzBbz/ /w8dsrC0OAYQ3T89DFerlq/JgCCSQgJ05a6EyRwQeGrf6BwquazL4KkM4LIVVl5WM9sK 7IMOKbkXQOD9s92BtfbNenltvcI0dIaHzFd5NVoQfrIq6XitbcZxtlbmwXdpVMhwaDYK VfGoXOmGIXhh0Wruh0dYNlNMN8DhZGKJXVfVj8HMz+2PaJHiiqpS+EJfHiKA/MMKRwW9 MdXAChYSUs0tRNzTmLQv1azwoFUjCfIU/17RR6PQbHHeTihXf8UXoDpHBKtvhyD367ct tBnw== X-Gm-Message-State: ALoCoQmpB0UJclyvh9l/oeNHAZ71N4dzRG1/+7Ynf/gsHLnbnOexnOGBv950e4DiKSsElkj1P9we X-Received: by 10.152.120.135 with SMTP id lc7mr2073991lab.38.1385404194098; Mon, 25 Nov 2013 10:29:54 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.133.74 with HTTP; Mon, 25 Nov 2013 10:29:33 -0800 (PST) X-Originating-IP: [108.176.158.82] From: Julio Merino Date: Mon, 25 Nov 2013 13:29:33 -0500 Message-ID: Subject: Migrating tools/regression/ tests (and dealing with TAP) To: freebsd-testing@freebsd.org Content-Type: multipart/mixed; boundary=089e012281acdc2e8004ec048c0f X-Content-Filtered-By: Mailman/MimeDel 2.1.16 Cc: Garrett Cooper , rpaulo@freebsd.org, Simon Gerraty X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2013 18:29:57 -0000 --089e012281acdc2e8004ec048c0f Content-Type: text/plain; charset=ISO-8859-1 Hello all, Here come sample patches to move existing tests in tools/regression/bin to the new layout ('tests' subdirectories within the corresponding tools) and to hook these tests into the test suite. The patches are quite straightforward: I just did an 'svn mv' of the existing shell scripts files, wrote a Makefile using plain.test.mk (here is where things get interesting) and applied minor changes to get all things sorted out (e.g. mtree directory registration). Consider this a non-working proof of concept. Again: this does NOT work (well, it does, but yields invalid results). The problem is that TAP-based test programs (like the ones to be run by prove) do NOT report 1 on failure and thus the plain Kyua interface is unsuitable because it always reports everything as passed. There are two solutions: 1) Patch every existing TAP-based test to propagate a 1 error code if any of its tests fails. Doable, yes, but it will take a while. (Mind you: this is the approach we tried in NetBSD and not all old tests are gone yet, years later... it's a significant amount of work to manually tweak all test programs!) I'd rather keep existing code as is *for now*; it simplifies the review of the changes and it doesn't immediately break people's knowledge of existing test code. (If there is a need for a rewrite, I'd suggest to go and use the ATF interface instead. But that's even more work.) 2) Add a TAP-compliant backend to Kyua. This would allow us to plug pretty much all existing tests in tools/regression/ without changes to the code. I believe this is a worthy goal for the transitional step and an easy first cut. But this means I've gotta implement this backend (fun!). I've been reading the protocol and seems reasonably simple, so I expect to have something workable this week. I'm choosing option 2 for now while leaving open the option of rewriting tests using ATF *only after* they have been hooked into the build. (What I'm shooting for, if you haven't guessed yet, is having a single tests tree instead of 3 in src ASAP... Keeping multiple trees is not a pretty situation for long. Once the new Kyua backend is ready, the proposed patches in this mail should change minimally: basically do s/plain/prove/ in the Makefiles. This is why I'm sending this mail out already: you can get a pretty good idea of what I'd like to do and your feedback would be useful! There is a "tricky" side change to be made to bsd.progs.mk so that it is possible to have a directory with SCRIPTS but no PROGS. make tinderbox is happy. Comments? Thanks! --089e012281acdc2e8004ec048c0f Content-Type: text/x-diff; charset=us-ascii; name="handle-scripts.diff" Content-Disposition: attachment; filename="handle-scripts.diff" Content-Transfer-Encoding: base64 X-Attachment-Id: 6c1840eb6ca50d84_0.2 TWFrZSBic2QucHJvZ3MubWsgd29yayBpbiBkaXJlY3RvcmllcyB3aXRoIFNDUklQVFMgYnV0IG5v IFBST0dTLg0KDQpUaGlzIGNoYW5nZSBmaXhlcyBzb21lIGNhc2VzIHdoZXJlIGJzZC5wcm9ncy5t ayB3b3VsZCBmYWlsIHRvIGhhbmRsZQ0KZGlyZWN0b3JpZXMgd2l0aCBTQ1JJUFRTIGJ1dCBubyBQ Uk9HUy4gIEluIHBhcnRpY3VsYXIsICJpbnN0YWxsIiBkaWQNCm5vdCBoYW5kbGUgc3VjaCBzY3Jp cHRzIG5vciBkZXBlbmRlbnQgZmlsZXMgd2hlbiBic2Quc3ViZGlyLm1rIHdhcw0KYWRkZWQgdG8g dGhlIG1peC4NCg0KVGhpcyBpcyAibWFrZSB0aW5kZXJib3giIGNsZWFuLg0KDQpJbmRleDogc2hh cmUvbWsvYnNkLnByb2dzLm1rDQo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09DQotLS0gc2hhcmUvbWsvYnNkLnByb2dzLm1r CShyZXZpc2lvbiAyNTg1NTIpDQorKysgc2hhcmUvbWsvYnNkLnByb2dzLm1rCSh3b3JraW5nIGNv cHkpDQpAQCAtNzMsNyArNzMsNyBAQA0KICMgaGFuZGxlIGJlaW5nIGNhbGxlZCBbYnNkLl1wcm9n cy5taw0KIC5pbmNsdWRlIDxic2QucHJvZy5taz4NCiANCi0uaWZuZGVmIFBST0cNCisuaWZuZGVm IF9SRUNVUlNJTkdfUFJPR1MNCiAjIHRlbGwgcHJvZ3MubWsgd2UgbWlnaHQgd2FudCB0byBpbnN0 YWxsIHRoaW5ncw0KIFBST0dTX1RBUkdFVFMrPSBjbGVhbmRlcGVuZCBjbGVhbmRpciBjbGVhbm9i aiBkZXBlbmQgaW5zdGFsbA0KIA0KQEAgLTg0LDExICs4NCwxMyBAQA0KIC5lbmRpZg0KIA0KICRw ICR7cH1fcDogLlBIT05ZIC5NQUtFDQotCShjZCAkey5DVVJESVJ9ICYmICR7TUFLRX0gLWYgJHtN QUtFRklMRX0gUFJPRz0kcCAke3guJHB9KQ0KKwkoY2QgJHsuQ1VSRElSfSAmJiAke01BS0V9IC1m ICR7TUFLRUZJTEV9IF9SRUNVUlNJTkdfUFJPR1M9IFwNCisJICAgIFNVQkRJUj0gUFJPRz0kcCAk e3guJHB9KQ0KIA0KIC5mb3IgdCBpbiAke1BST0dTX1RBUkdFVFM6Tzp1fQ0KICRwLiR0OiAuUEhP TlkgLk1BS0UNCi0JKGNkICR7LkNVUkRJUn0gJiYgJHtNQUtFfSAtZiAke01BS0VGSUxFfSBQUk9H PSRwICR7eC4kcH0gJHtAOkV9KQ0KKwkoY2QgJHsuQ1VSRElSfSAmJiAke01BS0V9IC1mICR7TUFL RUZJTEV9IF9SRUNVUlNJTkdfUFJPR1M9IFwNCisJICAgIFNVQkRJUj0gUFJPRz0kcCAke3guJHB9 ICR7QDpFfSkNCiAuZW5kZm9yDQogLmVuZGZvcg0KIA0KQEAgLTk2LDQgKzk4LDE4IEBADQogJHQ6 ICR7UFJPR1M6JT0lLiR0fQ0KIC5lbmRmb3INCiANCitTQ1JJUFRTX1RBUkdFVFMrPSBjbGVhbmRl cGVuZCBjbGVhbmRpciBjbGVhbm9iaiBkZXBlbmQgaW5zdGFsbA0KKw0KKy5mb3IgcCBpbiAke1ND UklQVFN9DQorLmZvciB0IGluICR7U0NSSVBUU19UQVJHRVRTOk86dX0NCiskcC4kdDogLlBIT05Z IC5NQUtFDQorCShjZCAkey5DVVJESVJ9ICYmICR7TUFLRX0gLWYgJHtNQUtFRklMRX0gX1JFQ1VS U0lOR19QUk9HUz0gXA0KKwkgICAgU1VCRElSPSBTQ1JJUFQ9JHAgJHt4LiRwfSAke0A6RX0pDQor LmVuZGZvcg0KKy5lbmRmb3INCisNCisuZm9yIHQgaW4gJHtTQ1JJUFRTX1RBUkdFVFM6Tzp1fQ0K KyR0OiAke1NDUklQVFM6JT0lLiR0fQ0KKy5lbmRmb3INCisNCiAuZW5kaWYNCg== --089e012281acdc2e8004ec048c0f-- From owner-freebsd-testing@FreeBSD.ORG Mon Nov 25 20:22:16 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6979A2B7; Mon, 25 Nov 2013 20:22:16 +0000 (UTC) Received: from co9outboundpool.messaging.microsoft.com (co9ehsobe001.messaging.microsoft.com [207.46.163.24]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 37D0A26EE; Mon, 25 Nov 2013 20:22:15 +0000 (UTC) Received: from mail134-co9-R.bigfish.com (10.236.132.254) by CO9EHSOBE030.bigfish.com (10.236.130.93) with Microsoft SMTP Server id 14.1.225.22; Mon, 25 Nov 2013 20:22:15 +0000 Received: from mail134-co9 (localhost [127.0.0.1]) by mail134-co9-R.bigfish.com (Postfix) with ESMTP id 26B6F1405B0; Mon, 25 Nov 2013 20:22:15 +0000 (UTC) X-Forefront-Antispam-Report: CIP:66.129.239.16; KIP:(null); UIP:(null); IPV:NLI; H:P-EMF02-SAC.jnpr.net; RD:none; EFVD:NLI X-SpamScore: 8 X-BigFish: VPS8(zzda00hdc73hzz1f42h208ch1ee6h1de0h1fdah2073h2146h1202h1e76h1d1ah1d2ah1fc6h1082kzz1de097hz31h2a8h839hd25hf0ah1288h12a5h12a9h12bdh12e5h137ah139eh13b6h1441h14ddh1504h1537h162dh1631h1758h1898h18e1h1946h19b5h1ad9h1b0ah1b2fh1b88h224fh1fb3h1d0ch1d2eh1d3fh1de2h1dfeh1dffh1e23h1fe8h1ff5h2218h2216h226dh22d0h222dm1155h) Received-SPF: softfail (mail134-co9: transitioning domain of juniper.net does not designate 66.129.239.16 as permitted sender) client-ip=66.129.239.16; envelope-from=sjg@juniper.net; helo=P-EMF02-SAC.jnpr.net ; SAC.jnpr.net ; Received: from mail134-co9 (localhost.localdomain [127.0.0.1]) by mail134-co9 (MessageSwitch) id 1385410934134114_14808; Mon, 25 Nov 2013 20:22:14 +0000 (UTC) Received: from CO9EHSMHS004.bigfish.com (unknown [10.236.132.226]) by mail134-co9.bigfish.com (Postfix) with ESMTP id 1222920009F; Mon, 25 Nov 2013 20:22:14 +0000 (UTC) Received: from P-EMF02-SAC.jnpr.net (66.129.239.16) by CO9EHSMHS004.bigfish.com (10.236.130.14) with Microsoft SMTP Server (TLS) id 14.16.227.3; Mon, 25 Nov 2013 20:22:13 +0000 Received: from magenta.juniper.net (172.17.27.123) by P-EMF02-SAC.jnpr.net (172.24.192.21) with Microsoft SMTP Server (TLS) id 14.3.146.0; Mon, 25 Nov 2013 12:22:12 -0800 Received: from chaos.jnpr.net (chaos.jnpr.net [172.24.29.229]) by magenta.juniper.net (8.11.3/8.11.3) with ESMTP id rAPKMBL95023; Mon, 25 Nov 2013 12:22:11 -0800 (PST) (envelope-from sjg@juniper.net) Received: from chaos.jnpr.net (localhost [127.0.0.1]) by chaos.jnpr.net (Postfix) with ESMTP id 772175807E; Mon, 25 Nov 2013 12:22:11 -0800 (PST) To: Julio Merino Subject: Re: Migrating tools/regression/ tests (and dealing with TAP) In-Reply-To: References: Comments: In-reply-to: Julio Merino message dated "Mon, 25 Nov 2013 13:29:33 -0500." From: "Simon J. Gerraty" X-Mailer: MH-E 7.82+cvs; nmh 1.3; GNU Emacs 22.3.1 Date: Mon, 25 Nov 2013 12:22:11 -0800 Message-ID: <20131125202211.772175807E@chaos.jnpr.net> MIME-Version: 1.0 Content-Type: text/plain X-OriginatorOrg: juniper.net X-FOPE-CONNECTOR: Id%0$Dn%*$RO%0$TLS%0$FQDN%$TlsDn% Cc: freebsd-testing@freebsd.org, Garrett Cooper , rpaulo@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Nov 2013 20:22:16 -0000 >I'm choosing option 2 for now while leaving open the option of rewriting Sounds good. >There is a "tricky" side change to be made to bsd.progs.mk so that it is >possible to have a directory with SCRIPTS but no PROGS. make tinderbox Looks reasonable. From owner-freebsd-testing@FreeBSD.ORG Tue Nov 26 05:52:43 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 40004FCA; Tue, 26 Nov 2013 05:52:43 +0000 (UTC) Received: from mail-pd0-x230.google.com (mail-pd0-x230.google.com [IPv6:2607:f8b0:400e:c02::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1421223D0; Tue, 26 Nov 2013 05:52:43 +0000 (UTC) Received: by mail-pd0-f176.google.com with SMTP id w10so6941085pde.7 for ; Mon, 25 Nov 2013 21:52:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=wj9gE4CvitlpBXiHLA85YzFnwHr/hPpTrwZ+NmcLzDM=; b=mCNZHiSsHb+hqTz0eoVeDbHC/2SbUPXJn6S7OEMBOT0vy4nO93OoxUr62GVm6MZyEl bMgy/30nq5gX+Rk/80Uus9w9dkGAJTiD312+gcnyh1+T3RYAqWuzUMQzRnZeIyw4Uy+o obv8zNF5GNoCFFl2ltBV3dwDacR3wjRfCGV0lAvuoJqkmG0/lUre4ODSx0/pzpR0z9y1 nUWvl6YrVVAE0EoMi+OhdcDRA7xyOmCyEeJxE3SkSahS1o/hlnXfhJLA/ipZ/5jo/ELM HCK1pVzX5K54EWILj/mIotL13F1bPdxe2tKkKcQRqmffGLWDqcY4/IxatSgnNlvt3Bf/ Lscw== X-Received: by 10.67.30.70 with SMTP id kc6mr31106006pad.32.1385445162698; Mon, 25 Nov 2013 21:52:42 -0800 (PST) Received: from [192.168.20.5] (c-50-181-163-89.hsd1.wa.comcast.net. [50.181.163.89]) by mx.google.com with ESMTPSA id sy10sm88109904pac.15.2013.11.25.21.52.40 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 25 Nov 2013 21:52:41 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Re: Migrating tools/regression/ tests (and dealing with TAP) From: Garrett Cooper In-Reply-To: Date: Mon, 25 Nov 2013 21:52:39 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <9EA03717-C36F-4BA7-A47D-DD68FB6D3EAE@gmail.com> References: To: Julio Merino X-Mailer: Apple Mail (2.1822) Cc: freebsd-testing@freebsd.org, rpaulo@freebsd.org, Simon Gerraty X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2013 05:52:43 -0000 On Nov 25, 2013, at 10:29 AM, Julio Merino wrote: > Hello all, >=20 > Here come sample patches to move existing tests in = tools/regression/bin > to the new layout ('tests' subdirectories within the corresponding = tools) > and to hook these tests into the test suite. >=20 > The patches are quite straightforward: I just did an 'svn mv' of the > existing shell scripts files, wrote a Makefile using plain.test.mk = (here > is where things get interesting) and applied minor changes to get all > things sorted out (e.g. mtree directory registration). >=20 > Consider this a non-working proof of concept. Again: this does NOT = work > (well, it does, but yields invalid results). The problem is that > TAP-based test programs (like the ones to be run by prove) do NOT = report > 1 on failure and thus the plain Kyua interface is unsuitable because = it > always reports everything as passed. >=20 > There are two solutions: >=20 > 1) Patch every existing TAP-based test to propagate a 1 error code if > any of its tests fails. Doable, yes, but it will take a while. > (Mind you: this is the approach we tried in NetBSD and not all old > tests are gone yet, years later... it's a significant amount of work > to manually tweak all test programs!) >=20 > I'd rather keep existing code as is *for now*; it simplifies the > review of the changes and it doesn't immediately break people's > knowledge of existing test code. (If there is a need for a rewrite, > I'd suggest to go and use the ATF interface instead. But that's = even > more work.) >=20 > 2) Add a TAP-compliant backend to Kyua. This would allow us to plug > pretty much all existing tests in tools/regression/ without changes > to the code. I believe this is a worthy goal for the transitional > step and an easy first cut. >=20 > But this means I've gotta implement this backend (fun!). I've been > reading the protocol and seems reasonably simple, so I expect to = have > something workable this week. >=20 > I'm choosing option 2 for now while leaving open the option of = rewriting > tests using ATF *only after* they have been hooked into the build. > (What I'm shooting for, if you haven't guessed yet, is having a single > tests tree instead of 3 in src ASAP... Keeping multiple trees is not = a > pretty situation for long. >=20 > Once the new Kyua backend is ready, the proposed patches in this mail > should change minimally: basically do s/plain/prove/ in the Makefiles. > This is why I'm sending this mail out already: you can get a pretty > good idea of what I'd like to do and your feedback would be useful! >=20 > There is a "tricky" side change to be made to bsd.progs.mk so that it = is > possible to have a directory with SCRIPTS but no PROGS. make = tinderbox > is happy. >=20 > Comments? >=20 > Thanks! > The big concern is that you=92re shuffling around tests that are = being actively developed by other developers. Have you provided a heads = up to Jilles first at least..? Thanks, -Garrett From owner-freebsd-testing@FreeBSD.ORG Tue Nov 26 14:22:09 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 886CA56C for ; Tue, 26 Nov 2013 14:22:09 +0000 (UTC) Received: from mail-lb0-f177.google.com (mail-lb0-f177.google.com [209.85.217.177]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 161442218 for ; Tue, 26 Nov 2013 14:22:08 +0000 (UTC) Received: by mail-lb0-f177.google.com with SMTP id w7so4360923lbi.8 for ; Tue, 26 Nov 2013 06:22:00 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=mHLOTh/+ZQX8vp4r/VcQiiOHmJ1B4+3gLpcOdqHZmu0=; b=TcsT5uLI00pr7Bd0H9mUeBIgzC9O+35Y/QLDQ4ZO2zUOyx/BTbQPQYjtp8kxVGAWBu w/YU+pQMVY5b/Roh2cxVAjSRkKuuK24p74TMefgCU3sK57Z64ybQkOjZDdp+YiOdNQMD kVCikwQZZowrnqGm+25Rg6PTFottaHlCrRGySF1aAzV5nVw217hRK4SmglbZNIyTQsJf N9GCQTjdWoqJRBTQ3SM0lzsNwEN82pK0ZBed/VeIy/Uj92tUGSz+z8vw79cfrc+ccjqq q+dpX7PWND0L9iZksHZp/pMnmB/nvcV21j16aIpGY0Nn9DAPzqWc+cYNkYYh5B0M4esL HmBw== X-Gm-Message-State: ALoCoQmejy0aEkOO665mc0VDgRSSqbJr+ap2q8YMx5/oTiUN/cCTF5bWYVgyr42QiOU4j1nzaEU8 X-Received: by 10.152.234.37 with SMTP id ub5mr160451lac.51.1385475720316; Tue, 26 Nov 2013 06:22:00 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.133.74 with HTTP; Tue, 26 Nov 2013 06:21:40 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: <9EA03717-C36F-4BA7-A47D-DD68FB6D3EAE@gmail.com> References: <9EA03717-C36F-4BA7-A47D-DD68FB6D3EAE@gmail.com> From: Julio Merino Date: Tue, 26 Nov 2013 09:21:40 -0500 Message-ID: Subject: Re: Migrating tools/regression/ tests (and dealing with TAP) To: Garrett Cooper Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-testing@freebsd.org, rpaulo@freebsd.org, Simon Gerraty X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2013 14:22:09 -0000 On Tue, Nov 26, 2013 at 12:52 AM, Garrett Cooper wrote= : > > The big concern is that you=92re shuffling around tests that are = being actively developed by other developers. Have you provided a heads up = to Jilles first at least..? Fair enough. Sending a separate email to Jilles to keep an eye on this thre= ad. --=20 Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Tue Nov 26 16:10:11 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7A15090; Tue, 26 Nov 2013 16:10:11 +0000 (UTC) Received: from mail-pb0-x231.google.com (mail-pb0-x231.google.com [IPv6:2607:f8b0:400e:c01::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 4DD1229D0; Tue, 26 Nov 2013 16:10:11 +0000 (UTC) Received: by mail-pb0-f49.google.com with SMTP id jt11so8185854pbb.8 for ; Tue, 26 Nov 2013 08:10:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=uLMoQsCFGgx6HWc6XOD9aedwFjkBlE9UXutkM4rGUdk=; b=W1mnoOvUAldt7zD/W/6mgYvrMG1ACjUW2SpT0unyyZY/zni3QxjlXX73VDIoF2dYJ3 lX+5Rs0/o+I/9VGUhOi7IHeL6CQrjWaPg7Gtw1T7VijLrfmV7RaaeGixcTFgH9fsS4wb 953xDi1FejggpXrVBH10iI3xOSZSMSqTfmLqiaqYRI25uPTFWpa6c/qa49gTrrqNacqR 3IluLQmuabYXEq5qMvKnp0Qfs4mKVrIc0nWkrSJ+D5rtvrqDx2qTXBborb/fco+Lu2Ce dogznFbj+O1YPKU8JQ5XuoIeVwVXHZz5PBVj4csEuKdYrEyw8b3lCc0mUImi5bJAxorD 7Ltw== X-Received: by 10.68.59.202 with SMTP id b10mr132957pbr.78.1385482210129; Tue, 26 Nov 2013 08:10:10 -0800 (PST) Received: from [192.168.20.5] (c-50-181-163-89.hsd1.wa.comcast.net. [50.181.163.89]) by mx.google.com with ESMTPSA id oj6sm92415430pab.9.2013.11.26.08.10.08 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 26 Nov 2013 08:10:09 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.0 \(1822\)) Subject: Re: Migrating tools/regression/ tests (and dealing with TAP) From: Garrett Cooper In-Reply-To: Date: Tue, 26 Nov 2013 08:10:07 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: <6F9EF37C-B53B-47C8-AF2C-FF1441054CCB@gmail.com> References: <9EA03717-C36F-4BA7-A47D-DD68FB6D3EAE@gmail.com> To: Julio Merino X-Mailer: Apple Mail (2.1822) Cc: freebsd-testing@freebsd.org, rpaulo@freebsd.org, Simon Gerraty X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2013 16:10:11 -0000 On Nov 26, 2013, at 6:21 AM, Julio Merino wrote: > On Tue, Nov 26, 2013 at 12:52 AM, Garrett Cooper = wrote: >>=20 >> The big concern is that you=92re shuffling around tests that = are being actively developed by other developers. Have you provided a = heads up to Jilles first at least..? >=20 > Fair enough. Sending a separate email to Jilles to keep an eye on this = thread. Also, I=92ve had bin/sh integrated into ATF for almost 2 years = now. I=92ll shoot you a patch with that, jilles@ and freebsd-testing@ = CCed as there=92s no need to keep the prove tests if they=92ve already = been converted over. Thanks, -Garrett= From owner-freebsd-testing@FreeBSD.ORG Tue Nov 26 16:15:29 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 242686C2 for ; Tue, 26 Nov 2013 16:15:29 +0000 (UTC) Received: from mail-la0-f51.google.com (mail-la0-f51.google.com [209.85.215.51]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A4B472A41 for ; Tue, 26 Nov 2013 16:15:28 +0000 (UTC) Received: by mail-la0-f51.google.com with SMTP id ec20so4354793lab.24 for ; Tue, 26 Nov 2013 08:15:26 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type:content-transfer-encoding; bh=Amog6eRYOvsddUVMupYIDumZlNm2JWEwN2aOKx1teyg=; b=e7FJM27DRYydYHpl9i5CBL7KEZTThlRxgglvRm2X+ljSMZfwQprJzEZ2fHicBRKCJ1 i2SyXMhEQV92OECB65OOAZTdQnIOHWSsLgYCZ6O1N2gx6FmYvRY03/WEC0bo/7fXjSzk EKQz3KMjSaHvpkvq1ilSxRjXcTpA3lyVPVKj7fMVUjzkO9NHSRTUHOus6eoZxc6BVOzi JBcSXbS7+5hckngAbtZsaSeisedEjAk55RUDrSjReFjZ6Bj/lCkGiHDu7C065J+x42MD oXgrODM4/m3x+cjKbRNvqfMuEl2ovL9zZdul/m55PU/FCINXKxf2fF1K4Lc5sWkb999G DQmg== X-Gm-Message-State: ALoCoQmoAABs2TIKcaHg00HWplnZV0lLyEY7AvQro26k7BFYvnMdMfStyMMvWom/bLM5M+xAAdTt X-Received: by 10.152.120.135 with SMTP id lc7mr5253569lab.38.1385482526545; Tue, 26 Nov 2013 08:15:26 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.133.74 with HTTP; Tue, 26 Nov 2013 08:15:06 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: <6F9EF37C-B53B-47C8-AF2C-FF1441054CCB@gmail.com> References: <9EA03717-C36F-4BA7-A47D-DD68FB6D3EAE@gmail.com> <6F9EF37C-B53B-47C8-AF2C-FF1441054CCB@gmail.com> From: Julio Merino Date: Tue, 26 Nov 2013 11:15:06 -0500 Message-ID: Subject: Re: Migrating tools/regression/ tests (and dealing with TAP) To: Garrett Cooper Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable Cc: freebsd-testing@freebsd.org, rpaulo@freebsd.org, Simon Gerraty X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2013 16:15:29 -0000 On Tue, Nov 26, 2013 at 11:10 AM, Garrett Cooper wr= ote: > On Nov 26, 2013, at 6:21 AM, Julio Merino wrote: > >> On Tue, Nov 26, 2013 at 12:52 AM, Garrett Cooper wr= ote: >>> >>> The big concern is that you=92re shuffling around tests that are= being actively developed by other developers. Have you provided a heads up= to Jilles first at least..? >> >> Fair enough. Sending a separate email to Jilles to keep an eye on this t= hread. > > Also, I=92ve had bin/sh integrated into ATF for almost 2 years no= w. I=92ll shoot you a patch with that, jilles@ and freebsd-testing@ CCed as= there=92s no need to keep the prove tests if they=92ve already been conver= ted over. That'd be nice! --=20 Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Tue Nov 26 16:30:06 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E987DC75 for ; Tue, 26 Nov 2013 16:30:06 +0000 (UTC) Received: from mail-lb0-f170.google.com (mail-lb0-f170.google.com [209.85.217.170]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 76AD22B2B for ; Tue, 26 Nov 2013 16:30:06 +0000 (UTC) Received: by mail-lb0-f170.google.com with SMTP id w7so4739825lbi.1 for ; Tue, 26 Nov 2013 08:30:04 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=jZxhweJkaAKio8RD0NXiX66/whyJoUj50+5P4vqks/4=; b=d0w/hVuZ/0Isgs2EziNq/8s8gqkbBvT/fAp59LPycnmRShIjQBF4TjOwWLiJ+nmPdE wfrellBjh6wDu4Z6i0esTXMskICbnt581KVC6cQnKOvHqUbyDqQFecQbvf4ynv95pT4w Bhk8++X8XMiJh+fxBZKcyXsXPj14beYw2nGShiAI/A3dUt1dopu+xboeV8Dn+D1WyZ8N tPcrI6EzLGuforAHohPuk0TWwmGwFTThbeunuH8sYE2Z8rneY6+bRIMe5A4PjUswJJ7D UI5Q/spY5A+MEunplBnaRVQa/9h3rA4CGauuCBUgZwg10cUDKAsgghDkCaOI1ZqfXisX pLDA== X-Gm-Message-State: ALoCoQkn2umFztxP2Xme9KEEWkFmn/XGIwPyRZliFqJw4r5cA6BurSCR+tV8CzRnP0qNOrVuiwiR X-Received: by 10.112.89.232 with SMTP id br8mr25571578lbb.20.1385483404398; Tue, 26 Nov 2013 08:30:04 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.133.74 with HTTP; Tue, 26 Nov 2013 08:29:44 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: References: <82E5F5E2-EA2C-46D2-9068-C419A834686C@gmail.com> <751B8D9F-9A1B-46B5-80F8-CB9E44F98BFC@gmail.com> From: Julio Merino Date: Tue, 26 Nov 2013 11:29:44 -0500 Message-ID: Subject: Re: Adding new "test interfaces" to work with Kyua? To: Garrett Cooper Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-testing@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2013 16:30:07 -0000 On Mon, Nov 25, 2013 at 12:13 PM, Julio Merino wrote: > Bleh. Turns out that prove-based test programs do not return 1 on > failure. Time to write a prove tester given that I have some spare > time this week :-P Alright! I've committed a preliminary implementation of a tester for TAP. See https://code.google.com/p/kyua/issues/detail?id=74 for the relevant commits. Testing welcome. -- Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Tue Nov 26 19:51:10 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 391FDA9A for ; Tue, 26 Nov 2013 19:51:10 +0000 (UTC) Received: from sanddollar.geekisp.com (sanddollar.geekisp.com [216.168.135.167]) by mx1.freebsd.org (Postfix) with SMTP id BD8EC2874 for ; Tue, 26 Nov 2013 19:51:09 +0000 (UTC) Received: (qmail 28790 invoked by uid 1003); 26 Nov 2013 19:51:03 -0000 Received: from unknown (HELO kiwi.corp.lookout.com) (tyler@monkeypox.org@64.125.69.200) by mail.geekisp.com with (DHE-RSA-AES256-SHA encrypted) SMTP; 26 Nov 2013 19:51:02 -0000 Date: Tue, 26 Nov 2013 11:50:57 -0800 From: "R. Tyler Croy" To: freebsd-testing@freebsd.org Subject: [Offtopic] Anybody want to talk about Kyua/etc at FOSDEM 14? Message-ID: <20131126195057.GF23246@kiwi.corp.lookout.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ULyIDA2m8JTe+TiX" Content-Disposition: inline User-Agent: Mutt/1.5.21 (2010-09-15) X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Nov 2013 19:51:10 -0000 --ULyIDA2m8JTe+TiX Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Apologies if this is off-topic, but I run the Testing and Automation devroom at FOSDEM and I certainly wouldn't mind if some folks from the FreeBSD testing community came and spoke about what you all are working on here. The CFP is open for a few more days until the 30th (I might extend it if you need me to ;)) I'm personally interested the testing work that Julio has been heading up, but I'm sure there are plenty of others who would be interested. Cheers - R. Tyler Croy -------------------------------------- Code: https://github.com/rtyler Chatter: https://twitter.com/agentdero rtyler@jabber.org --ULyIDA2m8JTe+TiX Content-Type: application/pgp-signature; name="signature.asc" Content-Description: Digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlKU+6EACgkQFCbH3D9R4W+C3ACfdnTqLM+8YT+qHxU56+nCY+tk WXwAn2ZIPadzN/v02r++I5YhyXenmygU =yHkC -----END PGP SIGNATURE----- --ULyIDA2m8JTe+TiX-- From owner-freebsd-testing@FreeBSD.ORG Wed Nov 27 04:21:25 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E3E1F206 for ; Wed, 27 Nov 2013 04:21:25 +0000 (UTC) Received: from mail-wi0-x22f.google.com (mail-wi0-x22f.google.com [IPv6:2a00:1450:400c:c05::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 80FCA252F for ; Wed, 27 Nov 2013 04:21:25 +0000 (UTC) Received: by mail-wi0-f175.google.com with SMTP id hi5so6181382wib.14 for ; Tue, 26 Nov 2013 20:21:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=QSxzmoDO+VuCxbJJdjOOhobp5JUNu6xW92DfHMnimBw=; b=ocl20E1KfvrA649VhRJ4+u4393kIWBXNaZgD0yYA/mjX/nhbWWMMhNe9qHQt9PLPHa USLGDCA6wJaaSU4zgEgwRMi+OFDuWYAVirUNK6qMTgfRkWrCT2rCjwDQenMWW13KAwIf aj3yKXWDa5HhyZC7wiexas4bUNw1nr/MMju7R1iU1LQWjAGs1xoo1FivEFo37r7fSivK ag8oVzp912QGvpj7nsGl1LxBZ3ZOS0aFJUzZidSAn5x8jp3+EzuaETqwE4uwfHCzLQqS jdqRivJ6RSVlHGm2+Y2TU4lhCt9kP4F2WmX/g6IQbbMxTLUPhXZ9dCj5xmL3D/OImapx dFCA== MIME-Version: 1.0 X-Received: by 10.194.232.133 with SMTP id to5mr9802770wjc.41.1385526084002; Tue, 26 Nov 2013 20:21:24 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.171.35 with HTTP; Tue, 26 Nov 2013 20:21:23 -0800 (PST) In-Reply-To: References: <82E5F5E2-EA2C-46D2-9068-C419A834686C@gmail.com> <751B8D9F-9A1B-46B5-80F8-CB9E44F98BFC@gmail.com> Date: Tue, 26 Nov 2013 21:21:23 -0700 X-Google-Sender-Auth: kqaB_iUE08v07DejG6cs-MmZbtQ Message-ID: Subject: Re: Adding new "test interfaces" to work with Kyua? From: Alan Somers To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2013 04:21:26 -0000 It looks like you're interfacing to TAP through its command-line interface. Unfortunately, as your man page notes, that means that "kyua list" can't list bundled tests, and thus it will treat them as a single unit. Within FreeBSD, most if not all of the TAP-based tests use Perl's Prove module. Have you considered writing a Prove backend for Kyua instead of a TAP backend? If you write it in Perl, it could have much more intimate access to the tests. Instead of interfacing through the CLI, it could dynamically load the tests and interrogate them directly. That's the approach that I'm taking with the Python backend that I've been working on. I'll post more details about that in a separate thread. -Alan On Tue, Nov 26, 2013 at 9:29 AM, Julio Merino wrote: > On Mon, Nov 25, 2013 at 12:13 PM, Julio Merino wrote: >> Bleh. Turns out that prove-based test programs do not return 1 on >> failure. Time to write a prove tester given that I have some spare >> time this week :-P > > Alright! I've committed a preliminary implementation of a tester for > TAP. See https://code.google.com/p/kyua/issues/detail?id=74 for the > relevant commits. > > Testing welcome. > > -- > Julio Merino / @jmmv > _______________________________________________ > freebsd-testing@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-testing > To unsubscribe, send any mail to "freebsd-testing-unsubscribe@freebsd.org" From owner-freebsd-testing@FreeBSD.ORG Wed Nov 27 04:47:31 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D48067F6 for ; Wed, 27 Nov 2013 04:47:31 +0000 (UTC) Received: from mail-we0-x236.google.com (mail-we0-x236.google.com [IPv6:2a00:1450:400c:c03::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5F336264E for ; Wed, 27 Nov 2013 04:47:31 +0000 (UTC) Received: by mail-we0-f182.google.com with SMTP id q59so6383392wes.27 for ; Tue, 26 Nov 2013 20:47:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:message-id:subject:from:to:content-type; bh=x93q5qDPfbIC1EhTrsFqJvAxDWX+AwIXC45z44YxsuQ=; b=wxX87cNcljlEvz3l3bu/Z2LMUDrr4K6Vn8fN1bUrTM7+zHwaXkV70XsNZ4WU0TyQw6 1AoL9KibK+K2MvOwh5dn0kZKAKU2YndSj5feLXNOVLhTXuYBZ4wqvckpuWYTYk7h+C0v qT1P9wwlBUt1PxIJxpcKmVjqn5oowXcl2OK5NltDxO6d3eTPo7rQ2IbkuHPDjRoFeu3x XEzPWgjdFlJe726ARlOKa1DdX6STT8RJwXEGznq8/x7azGnaCwgqLt3pB4HKGjNOtisQ zWPsFki0g1XCmQOlFHAgVCekpac6a9ierVMcVkRQk8LXIIsROuXZmNNi1gvWBgSjTbNy DjyA== MIME-Version: 1.0 X-Received: by 10.194.122.99 with SMTP id lr3mr28952581wjb.21.1385527649916; Tue, 26 Nov 2013 20:47:29 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.171.35 with HTTP; Tue, 26 Nov 2013 20:47:29 -0800 (PST) Date: Tue, 26 Nov 2013 21:47:29 -0700 X-Google-Sender-Auth: H6Ixf14lV0q45l_8LwfFCEG8-1A Message-ID: Subject: Python unittest backend for Kyua From: Alan Somers To: kyua-discuss@googlegroups.com, "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.16 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2013 04:47:32 -0000 I wrote a Kyua backend for Python programs that use the unittest module from the standard library. It's incomplete (no man page, no tests, no test case isolation, and no tested version of python other than 2.7), but I'm posting it to discuss the approach. Do you think it's a good start? Rather than use unittest's CLI, I wrote the backend to dynamically load the file under test and interrogate the test using unittest as a library. This allowed more intimate access. For example, the unittest CLI doesn't have a "list" command, but the backend can still list tests. Does it look good so far? -Alan From owner-freebsd-testing@FreeBSD.ORG Wed Nov 27 13:37:31 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8F531392 for ; Wed, 27 Nov 2013 13:37:31 +0000 (UTC) Received: from mail-lb0-f172.google.com (mail-lb0-f172.google.com [209.85.217.172]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 1B28D20C7 for ; Wed, 27 Nov 2013 13:37:30 +0000 (UTC) Received: by mail-lb0-f172.google.com with SMTP id z5so5455825lbh.3 for ; Wed, 27 Nov 2013 05:37:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=a5Mng/yu6wkl7f+FczU6p8YDFhZgqodaAZRRreJBwM8=; b=jVwV+d4x1KUquESuxhy/NIrMHf3/I/FV8X7278sUof86ttZtn/bAmmlafVCQPDW3n7 SH3fiNnkMBTVFILmPjygdnJu7my7ocmCGkPYaL2/Ap+6FJT26cPIRRqPz/z4PfCyurxm RoziQ1Dj07aFTY/DQUk1a7g1DzJ8tsOb8WKqNjQ3wWmyboGhOvzd8DzjXlWvN+ONqc7Q kbJ8ycJFa3K+wudzGjQ+M6xumrE17+qj1DW6n9u4An49oXLJvCaCoxEARD8AHPOOE8JO kzG+vuF2n1FBqrVVY3M1WBFWld9H//+ul+MiQRcZ5ypDmXK5zibj1jbbfdNGz+90qGeD g1bQ== X-Gm-Message-State: ALoCoQkPm6JOOwsqttAUw5vksTbeFKV16nESo7Qr2gQmkBh6VODy33xulHc3a6M99WRk1MQpVg/Y X-Received: by 10.152.225.161 with SMTP id rl1mr29526728lac.5.1385559442967; Wed, 27 Nov 2013 05:37:22 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.133.74 with HTTP; Wed, 27 Nov 2013 05:37:02 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: References: <82E5F5E2-EA2C-46D2-9068-C419A834686C@gmail.com> <751B8D9F-9A1B-46B5-80F8-CB9E44F98BFC@gmail.com> From: Julio Merino Date: Wed, 27 Nov 2013 08:37:02 -0500 Message-ID: Subject: Re: Adding new "test interfaces" to work with Kyua? To: Alan Somers Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2013 13:37:31 -0000 On Tue, Nov 26, 2013 at 11:21 PM, Alan Somers wrote: > It looks like you're interfacing to TAP through its command-line > interface. Unfortunately, as your man page notes, that means that > "kyua list" can't list bundled tests, and thus it will treat them as a > single unit. > > Within FreeBSD, most if not all of the TAP-based tests use Perl's > Prove module. Have you considered writing a Prove backend for Kyua > instead of a TAP backend? If you write it in Perl, it could have much > more intimate access to the tests. Instead of interfacing through the > CLI, it could dynamically load the tests and interrogate them > directly. I hadn't considered that, but it is a good idea. However, having separate TAP and Prove backends are not mutually exclusive goals. That said, I don't find the Prove backend a pressing need nor I think it's as useful as a TAP backend, at least for the time being: what I'd like is a simple way to transition existing tests to the new layout, and the TAP backend works for that -- with the added benefit that it works also for non-Perl tests! Later on, if people want to keep the Perl-based tests and make them more versatile within Kyua, we can do that pretty easily. (And of course, if anyone wants to start writing a more versatile backend explicitly for Perl tests anyway, please do!) -- Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Wed Nov 27 14:02:44 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7AE72E3C for ; Wed, 27 Nov 2013 14:02:44 +0000 (UTC) Received: from mail-lb0-f174.google.com (mail-lb0-f174.google.com [209.85.217.174]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 08E702216 for ; Wed, 27 Nov 2013 14:02:43 +0000 (UTC) Received: by mail-lb0-f174.google.com with SMTP id c11so5456810lbj.33 for ; Wed, 27 Nov 2013 06:02:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=2ulRR/gUrhqNgYOHTexyJbXdVNWTgRYneRpHSPAJuKM=; b=QZSA/KseUd5oC5VJDRPB4HiFxT7X/ix97vua+p2/hfEgqDVcUws+TEBLLK1fRwjT0z ig/xnOREUCn0sXYreayrIG5vq7r4QdzWkghywF838NoLfmGEpI+tnlulljqzxslip4L8 RSU8wXJJ7cPv8c8a3gMIDQi4yr/kQTXKgbkfieHKsnRtveWOhuyev4DPFeKUHGe2IzJ5 YrQg7kJry/JB6j7AscRkaHN7Fq4U/vGwslEbVgx7tXdklYM8p31+grnV8Lt0INa2dTh1 71HvwWxnzOP3+JHyhl5Zl7OCUP4SkodR8saWMdSwKNxMXfRkydr6CrO2BH8zrbzV9126 06SQ== X-Gm-Message-State: ALoCoQnVUByOpn+ZjM4RC5VSQEebbXShitM3qPx20OMGuLx7OB21Y1YAr88tr/LMEKMD7NyT6fR9 X-Received: by 10.112.210.66 with SMTP id ms2mr3507lbc.51.1385560606031; Wed, 27 Nov 2013 05:56:46 -0800 (PST) MIME-Version: 1.0 Received: by 10.112.133.74 with HTTP; Wed, 27 Nov 2013 05:56:25 -0800 (PST) X-Originating-IP: [108.176.158.82] In-Reply-To: References: From: Julio Merino Date: Wed, 27 Nov 2013 08:56:25 -0500 Message-ID: Subject: Re: Python unittest backend for Kyua To: kyua-discuss@googlegroups.com Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2013 14:02:44 -0000 On Tue, Nov 26, 2013 at 11:47 PM, Alan Somers wrote: > I wrote a Kyua backend for Python programs that use the unittest > module from the standard library. It's incomplete (no man page, no > tests, no test case isolation, and no tested version of python other > than 2.7), but I'm posting it to discuss the approach. Do you think > it's a good start? > > Rather than use unittest's CLI, I wrote the backend to dynamically > load the file under test and interrogate the test using unittest as a > library. This allowed more intimate access. For example, the > unittest CLI doesn't have a "list" command, but the backend can still > list tests. > > Does it look good so far? I like the idea a lot (and I had not considered it earlier). The code is a good start, but see below. One concern I have is regarding how you will implement isolation features. The more I think about them, the more I think that they are not specific to the ATF interface at all: i.e. all tests should be executed with the same level of isolation. Now, in your case, it seems you'd have to reimplement these features in Python -- which means that there will be two different implementations for the same thing and this can easily lead to inconsistencies. (atf had 3 versions of this in the very beginning, one for each supported language, and it wasn't pretty!) But having a pure Python binary has its benefits, if only for simplicity. The alternatives would be to either 1) hook the isolation C code into a Python module and call it from your tester, or 2) write the tester in C using all existing code (including CLI parsing for a consistent interface, isolation features, etc.) and just call into the Python interpreter to list tests or run one of them. What do you think? -- Julio Merino / @jmmv From owner-freebsd-testing@FreeBSD.ORG Wed Nov 27 23:47:27 2013 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0A465496 for ; Wed, 27 Nov 2013 23:47:27 +0000 (UTC) Received: from mail-wg0-x234.google.com (mail-wg0-x234.google.com [IPv6:2a00:1450:400c:c00::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 94B7A68D for ; Wed, 27 Nov 2013 23:47:26 +0000 (UTC) Received: by mail-wg0-f52.google.com with SMTP id x13so7546879wgg.7 for ; Wed, 27 Nov 2013 15:47:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=EMY8i9Fre+AMadk59LGFWIIS8Vtmw9BrxmhfgcZpcAM=; b=LrsT3VEnzM5gQUwikjFmlK2W9moAzq1/IAtPGEdsrEJNxjIqUWHzTdrsJffRFcNc2n JHviVro4Zt3PPfIiTRp1CWut7SJmcC3J6AW12BpIGK/jGAT0meh6Wq3Y1BnjhSloWYUt 9kYdiVS5mjClPLF8nn9hJ0bTnra56mxRDyMzHacU1t80fvOXFwLZuptfIjaZJJZvB+5/ 6cjn8X939svcrDxe4A5mH+hu+yDUhqSfnyGbkn5Rt/0wKYVLBbpC+m8qEETh/+I8ehmH Rb2xUG4wx+wiqPf1pVZb3rR7E9r2mCo16jKxezRAOXmI8xVxTsK5l/7uL+c8kmk7wRIA Os8w== MIME-Version: 1.0 X-Received: by 10.180.98.229 with SMTP id el5mr285106wib.29.1385596044863; Wed, 27 Nov 2013 15:47:24 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.171.35 with HTTP; Wed, 27 Nov 2013 15:47:24 -0800 (PST) In-Reply-To: References: Date: Wed, 27 Nov 2013 16:47:24 -0700 X-Google-Sender-Auth: drNDbV3P_tiVaMUTTym03lI5JLU Message-ID: Subject: Re: Python unittest backend for Kyua From: Alan Somers To: kyua-discuss@googlegroups.com Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Nov 2013 23:47:27 -0000 On Wed, Nov 27, 2013 at 6:56 AM, Julio Merino wrote: > On Tue, Nov 26, 2013 at 11:47 PM, Alan Somers wrote: >> I wrote a Kyua backend for Python programs that use the unittest >> module from the standard library. It's incomplete (no man page, no >> tests, no test case isolation, and no tested version of python other >> than 2.7), but I'm posting it to discuss the approach. Do you think >> it's a good start? >> >> Rather than use unittest's CLI, I wrote the backend to dynamically >> load the file under test and interrogate the test using unittest as a >> library. This allowed more intimate access. For example, the >> unittest CLI doesn't have a "list" command, but the backend can still >> list tests. >> >> Does it look good so far? > > I like the idea a lot (and I had not considered it earlier). The code > is a good start, but see below. > > One concern I have is regarding how you will implement isolation > features. The more I think about them, the more I think that they are > not specific to the ATF interface at all: i.e. all tests should be > executed with the same level of isolation. > > Now, in your case, it seems you'd have to reimplement these features > in Python -- which means that there will be two different > implementations for the same thing and this can easily lead to > inconsistencies. (atf had 3 versions of this in the very beginning, > one for each supported language, and it wasn't pretty!) But having a > pure Python binary has its benefits, if only for simplicity. The other benefit of pure Python is that Python doesn't become a build-time dependency of kyua-testers. As long as the tester is pure python, then Python is only a runtime dependency, and it can be ignored if you don't have any Python test programs. > > The alternatives would be to either 1) hook the isolation C code into > a Python module and call it from your tester, or 2) write the tester > in C using all existing code (including CLI parsing for a consistent > interface, isolation features, etc.) and just call into the Python > interpreter to list tests or run one of them. I've never programmed around the Python/C interface. I'll take a look and see how hard it would be. Is there any list of which isolation features a tester needs? The ones I know of are: * Separate process for each test case * Catch segfaults and other test deaths * setuid * setgid * timeout * chdir * sanitize ENV variables * core size * reset signal handlers * umask -Alan > > What do you think? > > -- > Julio Merino / @jmmv > > -- > You received this message because you are subscribed to the Google Groups "kyua-discuss" group. > To unsubscribe from this group and stop receiving emails from it, send an email to kyua-discuss+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.