From owner-freebsd-testing@FreeBSD.ORG Sun Jan 26 08:21:11 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CE0EB6A2 for ; Sun, 26 Jan 2014 08:21:11 +0000 (UTC) Received: from mail-pd0-x22a.google.com (mail-pd0-x22a.google.com [IPv6:2607:f8b0:400e:c02::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A0DE41392 for ; Sun, 26 Jan 2014 08:21:11 +0000 (UTC) Received: by mail-pd0-f170.google.com with SMTP id p10so4688518pdj.1 for ; Sun, 26 Jan 2014 00:21:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=NvrFDvU525SEtgtRTyIrdA2crnsGpYp3SqL7WXL8mfM=; b=linSXr5Zj1lR96lSUGrgCYxLNDXXymb+BWonsNtxUl8QYhI26I+27PXDNEsyfeZMl9 rBX9I/GS9qjWCwRJmwG3fm6sSg8xCz1QYZff3G4B4g6jrLnVdcIPgqmnPiKpHI+z6+7x xYUInpJpgewYwYVakai8I1AVO7bS9xfDkFZlz8GxU6NY0WR8EJzN6R1WRLBp4lqIwA4N /MJzuSEax/HDKGXo1uULjv06FsvVd3K2BuYclh4EzRaRixjGwNQD8FEiUWqDbQyB6T3C oPrM9RITh3ljLL/ZtvkGc8idMvsB5tShb3+HkwZA8xrvtVz5WqUEPszaSWZ1b5QN+WNf v97w== X-Received: by 10.68.139.100 with SMTP id qx4mr1646169pbb.144.1390724471157; Sun, 26 Jan 2014 00:21:11 -0800 (PST) Received: from [192.168.20.5] (c-50-181-163-89.hsd1.wa.comcast.net. [50.181.163.89]) by mx.google.com with ESMTPSA id vx10sm53683128pac.17.2014.01.26.00.21.09 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sun, 26 Jan 2014 00:21:10 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: [PATCH] convert /bin/sh tests over to ATF From: Garrett Cooper In-Reply-To: <20140124162759.GB90996@stack.nl> Date: Sun, 26 Jan 2014 00:21:07 -0800 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20140124162759.GB90996@stack.nl> To: Jilles Tjoelker X-Mailer: Apple Mail (2.1827) Cc: Julio Merino , freebsd-testing@FreeBSD.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 Jan 2014 08:21:11 -0000 On Jan 24, 2014, at 8:27 AM, Jilles Tjoelker wrote: > On Mon, Jan 20, 2014 at 08:44:58AM -0800, Garrett Cooper wrote: >> Also, I added some functionality so the tests expect failures on >> certain versions of FreeBSD due to missing functionality or >> conformance modifications (see the .valid_osreldate files). >=20 > I think this part is a bit too simplistic and inflexible. I have = always > dealt with changes affecting the tests by MFCing the changes to the > tests along with the changes to sh itself, and this has worked well > enough. On the other hand, I also run -current's sh and tests on my > -stable machine, without installing the tests (and often also without > installing the new sh). Therefore, the osreldate of the machine does = not > imply that the sh is also of that version. >=20 > There are also changes like r222684 that need an adjustment to the = tests > which are hard to handle properly using a .valid_osreldate file. Fair enough. Long story short, when I wrote this I was still working at = EMC on a FreeBSD port project where we were porting = revision-by-revision, so in order for the tests to not fail [so much], = it helped to write in these checks. I agree though, this shouldn=92t hit = mainline FreeBSD at all (will yank it). > There may be some point in marking tests that are supposed to work on > any POSIX-compliant sh and ones that are inherently FreeBSD-specific, = or > possibly more granular. However, this would be most useful when = someone > who cares about another sh implementation works on it. Ok. Thanks! -Garrett=