From owner-freebsd-testing@FreeBSD.ORG Fri Jul 25 01:23:05 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 380A0D63 for ; Fri, 25 Jul 2014 01:23:05 +0000 (UTC) Received: from mail-wg0-f52.google.com (mail-wg0-f52.google.com [74.125.82.52]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BEF812F42 for ; Fri, 25 Jul 2014 01:23:03 +0000 (UTC) Received: by mail-wg0-f52.google.com with SMTP id a1so3519482wgh.35 for ; Thu, 24 Jul 2014 18:23:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=6yG2wZitMZpeWMj/fEQrunzBDJnjpfazmwPu/HStw38=; b=U612X4NXC9WLxjwW4TqPUV6KslADJfny/JK+9ZjJiMcFAVs4MCZ4qLx2VTEAH3WCx8 dwJep7Bf3Fg7PEaEXl6/JsuUprSopvpehSwzSEE6nKpey/h+Mqd6mZfSG+FmFMCSuoS3 +ipHqxIlK2/r8HoL7Cr7i9d1/H+9MGPYNsIXFTftBFV3Gif+tQ4xHyJhfzsCQ0kra2+w b1m+i4eseMRjU6xo5aqQyBLPYHZR/oUG5d7W5BGvUGZ5NFi0AdBfIC40HYc4o8eDuyt2 XqIuVn+Hcss11pzB0aieKd9NgU/NyuPKO5Tl1yYcRXA/YsiT8WNbBW0UCY6U0X8fBu8H OR/Q== X-Gm-Message-State: ALoCoQmGre59t5UCEMA3EwA/qG6LrCbLgIj0x/AjvKlpJQ3J8XAy6CI2CKFlYXI1sWD865wJCiZ7 X-Received: by 10.194.63.37 with SMTP id d5mr17465047wjs.92.1406250902663; Thu, 24 Jul 2014 18:15:02 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.194.203.3 with HTTP; Thu, 24 Jul 2014 18:14:42 -0700 (PDT) X-Originating-IP: [184.153.117.174] In-Reply-To: References: <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com> From: Julio Merino Date: Thu, 24 Jul 2014 21:14:42 -0400 X-Google-Sender-Auth: DRY63jysvCAT5SoyHlH99yWUmOk Message-ID: Subject: Re: Need input on preference on location of 3rd party tests vs FreeBSD tests To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Garrett Cooper , "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 01:23:05 -0000 On Thu, Jul 24, 2014 at 4:39 PM, Garrett Cooper wro= te: > On Thu, Jul 24, 2014 at 12:24 PM, Julio Merino wrote: >> On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper wro= te: >>> On Jul 18, 2014, at 7:45 AM, Alan Somers wrote: >>> >>>> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper wrote: >>>>> Hi all, >>>>> One of the things that I've done on my fork of FreeBSD is I've impo= rted ATF test suites from NetBSD and I have integrated existing test suites= from freebsd's tools/regression tree into Kyua as well. Due to the size an= d difference in test content/coverage, I pulled lib/libc and lib/msun from = bother sources and integrated them into Kyua. What I did was I put the netb= sd testcases into the tests/ subdirectory and put the FreeBSD test suites i= nto a tests/legacy subdirectory. The goal was that the legacy directory wou= ld eventually be converted over to atf testcases and then could be removed = once the conversion was complete. >>>>> I'm not sure if this scheme makes sense though. Does anyone have a = preference as to whether or not this makes sense? >>>>> Thanks! >>>>> -Garrett >>>> >>>> >>>> I don't understand. What did you put in tests/legacy? >>> >>> The tests from tools/regression. tests/ contains the tests from NetBSD. >> >> I do not think adding tests under src/tests/ is a good idea. We have >> chosen to put tests under the 'tests' subdirectories of the affected >> components and we should stick to this rule except for very specific >> cases (see src/tests/sys/). Creating an artificial barrier between >> tests imported from NetBSD and native tests to FreeBSD is not >> beneficial (e.g. why should users/developers care where the tests came >> from?). > > Sorry if this was a bit misleading in my original email... > > Basically I'm trying to solve the problem of keeping the FreeBSD > testcases isolated from the NetBSD testcases. This only impacted two > components (so far) -- libc and msun. What I did was placed the libc > testcases from NetBSD in lib/libc/tests and the libc testcases from > FreeBSD in lib/libc/tests/legacy (kind of matching the model that we > have for TAP/plain scripts with the "legacy" name, but in name only). Right... but I'm not sure if that's what FreeBSD should be doing. I can see how, in the context of Isilon, this was the best thing to do: keep yourself out of the business of writing tests for the base system where possible and just reuse what NetBSD had already done. However, we are now working towards the test suite that ships with FreeBSD itself, and in this case we really should be fully on-board with maintaining the tests themselves within the project. Compare this to the development of libc itself, for example. My biggest fear is that we will perform the initial import of tests from NetBSD (as a bunch of files littered throughout the tree, with no links to where they came from), then hack on the code to make it work for FreeBSD, then add a bunch of new tests by various users... and then never import updates from NetBSD again. There are plenty of examples of code that was shared across projects and that never suffered further reimports due to the difficulty of it; see the truly obsolete smbfs in NetBSD, the pf in FreeBSD, or the way tmpfs has gone different paths in NetBSD and FreeBSD. This is why I wouldn't introduce any intentional divergence between NetBSD and FreeBSD tests in the tree, in particular to prevent any internal fears about "this code is not ours so better not touch it much". Just import the code and, if we want to pull in new tests from NetBSD, do it on a case-by-case basis. (Note that I don't object to keeping the legacy tests in a subdirectory though as that sounds reasonable. But I feel that this is orthogonal to whether we (re)import tests from NetBSD or we write our shiny new test programs!)