From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 20:39:50 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A285B228; Thu, 24 Jul 2014 20:39:50 +0000 (UTC) Received: from mail-ie0-x229.google.com (mail-ie0-x229.google.com [IPv6:2607:f8b0:4001:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 64B8F25A1; Thu, 24 Jul 2014 20:39:50 +0000 (UTC) Received: by mail-ie0-f169.google.com with SMTP id rd18so2797835iec.0 for ; Thu, 24 Jul 2014 13:39:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=L9CCJ10uDRW+q2wlpvqXiWnL2chh8AfBMCR8/ejGFPM=; b=Lt9cgWkfmJ8tgG6aEsrfa4v2Mw9bCf2scO4ir3AkeYc5zN+3c40sez/DK5BoGNPrlO 476IXK9cM+0fP3//G7ANkxXd/XhdlJQR/s5akHvpO02nAuLpnroJHtS2E8AesHlC5AAz b9MQCQ56UQVwft758oWpNOskOjQPH34f8mspCvrf/MUlt6ekvFmdv8DephJ2yJ6DM4Xh tUGlMK3UHAJ02KyYP5P7NC6BoWT3QX10wDdByn1gs8xfFfLJGMutwo7UXzOYodCaxpUz 4vnT8071jFO1wh+8oxGPM76lqyaGaOlu9QP4xavM08GVPAFhniIRAuyVfvdDLV3xq/P+ 7P/g== MIME-Version: 1.0 X-Received: by 10.42.198.75 with SMTP id en11mr15304278icb.7.1406234389824; Thu, 24 Jul 2014 13:39:49 -0700 (PDT) Received: by 10.50.213.102 with HTTP; Thu, 24 Jul 2014 13:39:49 -0700 (PDT) In-Reply-To: References: <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com> Date: Thu, 24 Jul 2014 13:39:49 -0700 Message-ID: Subject: Re: Need input on preference on location of 3rd party tests vs FreeBSD tests From: Garrett Cooper To: Julio Merino Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Garrett Cooper , "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 20:39:50 -0000 On Thu, Jul 24, 2014 at 12:24 PM, Julio Merino wrote: > On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper wrot= e: >> On Jul 18, 2014, at 7:45 AM, Alan Somers wrote: >> >>> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper = wrote: >>>> Hi all, >>>> One of the things that I've done on my fork of FreeBSD is I've impor= ted ATF test suites from NetBSD and I have integrated existing test suites = from freebsd's tools/regression tree into Kyua as well. Due to the size and= difference in test content/coverage, I pulled lib/libc and lib/msun from b= other sources and integrated them into Kyua. What I did was I put the netbs= d testcases into the tests/ subdirectory and put the FreeBSD test suites in= to a tests/legacy subdirectory. The goal was that the legacy directory woul= d eventually be converted over to atf testcases and then could be removed o= nce the conversion was complete. >>>> I'm not sure if this scheme makes sense though. Does anyone have a p= reference as to whether or not this makes sense? >>>> Thanks! >>>> -Garrett >>> >>> >>> I don't understand. What did you put in tests/legacy? >> >> The tests from tools/regression. tests/ contains the tests from NetBSD. > > I do not think adding tests under src/tests/ is a good idea. We have > chosen to put tests under the 'tests' subdirectories of the affected > components and we should stick to this rule except for very specific > cases (see src/tests/sys/). Creating an artificial barrier between > tests imported from NetBSD and native tests to FreeBSD is not > beneficial (e.g. why should users/developers care where the tests came > from?). Sorry if this was a bit misleading in my original email... Basically I'm trying to solve the problem of keeping the FreeBSD testcases isolated from the NetBSD testcases. This only impacted two components (so far) -- libc and msun. What I did was placed the libc testcases from NetBSD in lib/libc/tests and the libc testcases from FreeBSD in lib/libc/tests/legacy (kind of matching the model that we have for TAP/plain scripts with the "legacy" name, but in name only). Example: - https://github.com/yaneurabeya/freebsd/tree/isilon-atf/lib/libc/tests - the NetBSD testcases (mostly) - https://github.com/yaneurabeya/freebsd/tree/isilon-atf/lib/libc/tests/leg= acy - the testcases from tools/regression on FreeBSD head > "Reusing the NetBSD Makefiles" is not a strong enough reason because > those are usually trivial -- and if they aren't, that's an indication > that they are wrong! Well, they're really trivial, yes, and apart from small caveats with bsd.progs.mk in HEAD today, there isn't a reason for the testcase Makefiles to be divergent, except when dealing with FreeBSD vs NetBSD MK_ knobs. > We have two options: > > 1) We treat the NetBSD tests as upstream. In this case, we ought to do > the right thing, which is to put them in a vendor branch, merge into > contrib, and add reachover Makefiles where necessary. This is what I was thinking of doing because I don't have a compelling argument and it doesn't really seem to make sense to fork [at this point]. The modifications that Peter and I have been making to the isilon-atf branch are relatively trivial/documented with #if defined(__FreeBSD__) or #if defined(__NetBSD__) (still not happy with the __arraycount vs nitems divergence, but it's a simple sed one-liner). I've pushed a number of patches out to NetBSD GNATS to fix issues with the testcases (mostly header discrepancies, but also some valid testcase bugs with NetBSD's copy of the tests). > OR > > 2) We just copy the code (effectively a fork), and when doing so we > can do as we wish with the structure. > > So... > > Option 1 may sound nice, but it will be problematic long-term without > collaboration from NetBSD: I don't think we want to treat NetBSD's > tests as upstream if NetBSD themselves don't consider the tests as a > "first class project" (and they don't currently). Not to mention that > this will require a significant amount of local patching with the > corresponding maintenance headaches. > > Option 2 is definitely easier to handle on our side, but with the > obvious disadvantage that we won't be able to easily import tests from > NetBSD. How big of a problem is that today? Honestly the fact that our > codebases are separate for things like libc is more of a problem than > having diverging tests. > > Option 3 (yes, another one!) would be to start a third-party > repository holding "generic Unix tests" (a POSIX test suite?) that > both NetBSD and FreeBSD import as a third-party product. Then, each > system would just extend these suites with local patches to verify > their system-specific behavior. A very nice option in my opinion, but > this is a major effort. (And this can be tackled later once we the > FreeBSD Test Suite is more complete and we can easily spot how much it > overlaps to the NetBSD one.) This is an alternative as well, but the tricky thing is that it moves away from the whole idea of "having the tests in the tree" which was part of the point behind doing things this way (to get developers in the habit of writing tests that match the test code), unless both FreeBSD and NetBSD collaboratively pull test code from the "BSD Test Project". Thanks! -Garrett