From owner-freebsd-testing@FreeBSD.ORG Fri Jan 24 16:28:04 2014 Return-Path: Delivered-To: freebsd-testing@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 439C0C95 for ; Fri, 24 Jan 2014 16:28:04 +0000 (UTC) Received: from mx1.stack.nl (relay04.stack.nl [IPv6:2001:610:1108:5010::107]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 093131ECC for ; Fri, 24 Jan 2014 16:28:00 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id A6DC4B8439; Fri, 24 Jan 2014 17:27:59 +0100 (CET) Received: by snail.stack.nl (Postfix, from userid 1677) id 93ED128497; Fri, 24 Jan 2014 17:27:59 +0100 (CET) Date: Fri, 24 Jan 2014 17:27:59 +0100 From: Jilles Tjoelker To: Garrett Cooper Subject: Re: [PATCH] convert /bin/sh tests over to ATF Message-ID: <20140124162759.GB90996@stack.nl> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-testing@FreeBSD.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Jan 2014 16:28:04 -0000 On Mon, Jan 20, 2014 at 08:44:58AM -0800, Garrett Cooper wrote: > Also, I added some functionality so the tests expect failures on > certain versions of FreeBSD due to missing functionality or > conformance modifications (see the .valid_osreldate files). I think this part is a bit too simplistic and inflexible. I have always dealt with changes affecting the tests by MFCing the changes to the tests along with the changes to sh itself, and this has worked well enough. On the other hand, I also run -current's sh and tests on my -stable machine, without installing the tests (and often also without installing the new sh). Therefore, the osreldate of the machine does not imply that the sh is also of that version. There are also changes like r222684 that need an adjustment to the tests which are hard to handle properly using a .valid_osreldate file. There may be some point in marking tests that are supposed to work on any POSIX-compliant sh and ones that are inherently FreeBSD-specific, or possibly more granular. However, this would be most useful when someone who cares about another sh implementation works on it. -- Jilles Tjoelker