Date: Thu, 24 Jul 2014 15:24:07 -0400 From: Julio Merino <jmmv@freebsd.org> To: Garrett Cooper <yanegomi@gmail.com> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Need input on preference on location of 3rd party tests vs FreeBSD tests Message-ID: <CAFY7cWBeJuzQStvf7fyCXB957gDB=o87yKv%2Bgt9fAjwLG1QgzA@mail.gmail.com> In-Reply-To: <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com> References: <B1CE2158-B8C1-4D97-AB7A-031ADF55C435@gmail.com> <E5C29F6D-C632-4B7B-B8C8-0816C93DB7B7@gmail.com> <CAOtMX2izFjpC7r7Vxtk68KynAeuFvaDNNbK5nOEGcdFy7vJE5A@mail.gmail.com> <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com>
index | next in thread | previous in thread | raw e-mail
On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper <yanegomi@gmail.com> wrote: > On Jul 18, 2014, at 7:45 AM, Alan Somers <asomers@freebsd.org> wrote: > >> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper <yaneurabeya@gmail.com> wrote: >>> Hi all, >>> One of the things that I've done on my fork of FreeBSD is I've imported ATF test suites from NetBSD and I have integrated existing test suites from freebsd's tools/regression tree into Kyua as well. Due to the size and difference in test content/coverage, I pulled lib/libc and lib/msun from bother sources and integrated them into Kyua. What I did was I put the netbsd testcases into the tests/ subdirectory and put the FreeBSD test suites into a tests/legacy subdirectory. The goal was that the legacy directory would eventually be converted over to atf testcases and then could be removed once the conversion was complete. >>> I'm not sure if this scheme makes sense though. Does anyone have a preference as to whether or not this makes sense? >>> Thanks! >>> -Garrett >> >> >> I don't understand. What did you put in tests/legacy? > > The tests from tools/regression. tests/ contains the tests from NetBSD. I do not think adding tests under src/tests/ is a good idea. We have chosen to put tests under the 'tests' subdirectories of the affected components and we should stick to this rule except for very specific cases (see src/tests/sys/). Creating an artificial barrier between tests imported from NetBSD and native tests to FreeBSD is not beneficial (e.g. why should users/developers care where the tests came from?). "Reusing the NetBSD Makefiles" is not a strong enough reason because those are usually trivial -- and if they aren't, that's an indication that they are wrong! We have two options: 1) We treat the NetBSD tests as upstream. In this case, we ought to do the right thing, which is to put them in a vendor branch, merge into contrib, and add reachover Makefiles where necessary. OR 2) We just copy the code (effectively a fork), and when doing so we can do as we wish with the structure. So... Option 1 may sound nice, but it will be problematic long-term without collaboration from NetBSD: I don't think we want to treat NetBSD's tests as upstream if NetBSD themselves don't consider the tests as a "first class project" (and they don't currently). Not to mention that this will require a significant amount of local patching with the corresponding maintenance headaches. Option 2 is definitely easier to handle on our side, but with the obvious disadvantage that we won't be able to easily import tests from NetBSD. How big of a problem is that today? Honestly the fact that our codebases are separate for things like libc is more of a problem than having diverging tests. Option 3 (yes, another one!) would be to start a third-party repository holding "generic Unix tests" (a POSIX test suite?) that both NetBSD and FreeBSD import as a third-party product. Then, each system would just extend these suites with local patches to verify their system-specific behavior. A very nice option in my opinion, but this is a major effort. (And this can be tackled later once we the FreeBSD Test Suite is more complete and we can easily spot how much it overlaps to the NetBSD one.)home | help
Want to link to this message? Use this
URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFY7cWBeJuzQStvf7fyCXB957gDB=o87yKv%2Bgt9fAjwLG1QgzA>