From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 19:24:34 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1CFEBF89 for ; Thu, 24 Jul 2014 19:24:34 +0000 (UTC) Received: from mail-qg0-f44.google.com (mail-qg0-f44.google.com [209.85.192.44]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CD79C2F07 for ; Thu, 24 Jul 2014 19:24:33 +0000 (UTC) Received: by mail-qg0-f44.google.com with SMTP id e89so3910699qgf.3 for ; Thu, 24 Jul 2014 12:24:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=Wc93h6g/uoVGNh65heinmgA6lvmWGlqPpiKkbl9Dp2I=; b=eXb4wJxR/2zZQufr3XbYoy3UrGH2C78OqrPR6zQSrSRK60kR6TW06Z+Co3t2kdbGxz 7TfKWQa01g8XFvoAfqfwEIfEyYvdC85uqudK5ZWoS+34wBt07DhLBhm5NMF+Q3urNWM0 jNwG0ZB40M0KdvEt73ruPMt5VxdJRzI9Qq3GgQS9WzvtuOsZ6nuZrdjkzzxg6v+diiI5 4sFxY28P58USkTFMbvPzqVZWddh8YsKAWQkbylObyjXfVGfDhwEqK/5611XuZg7qaPJj uTVFdiJAE1Y7FFdP/cINNcU82l/bcn0hFr7pNH2/yDQbtK1QqxBqe1kk4kmwVNjnqz8k FNxQ== X-Gm-Message-State: ALoCoQkAjWdvXp8D3eeLI9aXZO4gnRAbpFKnMhzNfsrxcEbs+e+E2+6J/4jLD+7vT5AlpN3oX23N X-Received: by 10.140.109.118 with SMTP id k109mr18080907qgf.98.1406229867260; Thu, 24 Jul 2014 12:24:27 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Thu, 24 Jul 2014 12:24:07 -0700 (PDT) X-Originating-IP: [2620:0:1003:1007:ac3e:4bce:435e:6e80] In-Reply-To: <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com> References: <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com> From: Julio Merino Date: Thu, 24 Jul 2014 15:24:07 -0400 X-Google-Sender-Auth: 1qssWOGXL8oaxzYNDjnrfXnuHRY Message-ID: Subject: Re: Need input on preference on location of 3rd party tests vs FreeBSD tests To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 19:24:34 -0000 On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper wrote: > On Jul 18, 2014, at 7:45 AM, Alan Somers wrote: > >> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper = wrote: >>> Hi all, >>> One of the things that I've done on my fork of FreeBSD is I've import= ed ATF test suites from NetBSD and I have integrated existing test suites f= rom freebsd's tools/regression tree into Kyua as well. Due to the size and = difference in test content/coverage, I pulled lib/libc and lib/msun from bo= ther sources and integrated them into Kyua. What I did was I put the netbsd= testcases into the tests/ subdirectory and put the FreeBSD test suites int= o a tests/legacy subdirectory. The goal was that the legacy directory would= eventually be converted over to atf testcases and then could be removed on= ce the conversion was complete. >>> I'm not sure if this scheme makes sense though. Does anyone have a pr= eference as to whether or not this makes sense? >>> Thanks! >>> -Garrett >> >> >> I don't understand. What did you put in tests/legacy? > > The tests from tools/regression. tests/ contains the tests from NetBSD. I do not think adding tests under src/tests/ is a good idea. We have chosen to put tests under the 'tests' subdirectories of the affected components and we should stick to this rule except for very specific cases (see src/tests/sys/). Creating an artificial barrier between tests imported from NetBSD and native tests to FreeBSD is not beneficial (e.g. why should users/developers care where the tests came from?). "Reusing the NetBSD Makefiles" is not a strong enough reason because those are usually trivial -- and if they aren't, that's an indication that they are wrong! We have two options: 1) We treat the NetBSD tests as upstream. In this case, we ought to do the right thing, which is to put them in a vendor branch, merge into contrib, and add reachover Makefiles where necessary. OR 2) We just copy the code (effectively a fork), and when doing so we can do as we wish with the structure. So... Option 1 may sound nice, but it will be problematic long-term without collaboration from NetBSD: I don't think we want to treat NetBSD's tests as upstream if NetBSD themselves don't consider the tests as a "first class project" (and they don't currently). Not to mention that this will require a significant amount of local patching with the corresponding maintenance headaches. Option 2 is definitely easier to handle on our side, but with the obvious disadvantage that we won't be able to easily import tests from NetBSD. How big of a problem is that today? Honestly the fact that our codebases are separate for things like libc is more of a problem than having diverging tests. Option 3 (yes, another one!) would be to start a third-party repository holding "generic Unix tests" (a POSIX test suite?) that both NetBSD and FreeBSD import as a third-party product. Then, each system would just extend these suites with local patches to verify their system-specific behavior. A very nice option in my opinion, but this is a major effort. (And this can be tackled later once we the FreeBSD Test Suite is more complete and we can easily spot how much it overlaps to the NetBSD one.)