Date: Thu, 24 Jul 2014 15:24:07 -0400 From: Julio Merino <jmmv@freebsd.org> To: Garrett Cooper <yanegomi@gmail.com> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Need input on preference on location of 3rd party tests vs FreeBSD tests Message-ID: <CAFY7cWBeJuzQStvf7fyCXB957gDB=o87yKv%2Bgt9fAjwLG1QgzA@mail.gmail.com> In-Reply-To: <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com> References: <B1CE2158-B8C1-4D97-AB7A-031ADF55C435@gmail.com> <E5C29F6D-C632-4B7B-B8C8-0816C93DB7B7@gmail.com> <CAOtMX2izFjpC7r7Vxtk68KynAeuFvaDNNbK5nOEGcdFy7vJE5A@mail.gmail.com> <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper <yanegomi@gmail.com> wrote: > On Jul 18, 2014, at 7:45 AM, Alan Somers <asomers@freebsd.org> wrote: > >> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper <yaneurabeya@gmail.com> = wrote: >>> Hi all, >>> One of the things that I've done on my fork of FreeBSD is I've import= ed ATF test suites from NetBSD and I have integrated existing test suites f= rom freebsd's tools/regression tree into Kyua as well. Due to the size and = difference in test content/coverage, I pulled lib/libc and lib/msun from bo= ther sources and integrated them into Kyua. What I did was I put the netbsd= testcases into the tests/ subdirectory and put the FreeBSD test suites int= o a tests/legacy subdirectory. The goal was that the legacy directory would= eventually be converted over to atf testcases and then could be removed on= ce the conversion was complete. >>> I'm not sure if this scheme makes sense though. Does anyone have a pr= eference as to whether or not this makes sense? >>> Thanks! >>> -Garrett >> >> >> I don't understand. What did you put in tests/legacy? > > The tests from tools/regression. tests/ contains the tests from NetBSD. I do not think adding tests under src/tests/ is a good idea. We have chosen to put tests under the 'tests' subdirectories of the affected components and we should stick to this rule except for very specific cases (see src/tests/sys/). Creating an artificial barrier between tests imported from NetBSD and native tests to FreeBSD is not beneficial (e.g. why should users/developers care where the tests came from?). "Reusing the NetBSD Makefiles" is not a strong enough reason because those are usually trivial -- and if they aren't, that's an indication that they are wrong! We have two options: 1) We treat the NetBSD tests as upstream. In this case, we ought to do the right thing, which is to put them in a vendor branch, merge into contrib, and add reachover Makefiles where necessary. OR 2) We just copy the code (effectively a fork), and when doing so we can do as we wish with the structure. So... Option 1 may sound nice, but it will be problematic long-term without collaboration from NetBSD: I don't think we want to treat NetBSD's tests as upstream if NetBSD themselves don't consider the tests as a "first class project" (and they don't currently). Not to mention that this will require a significant amount of local patching with the corresponding maintenance headaches. Option 2 is definitely easier to handle on our side, but with the obvious disadvantage that we won't be able to easily import tests from NetBSD. How big of a problem is that today? Honestly the fact that our codebases are separate for things like libc is more of a problem than having diverging tests. Option 3 (yes, another one!) would be to start a third-party repository holding "generic Unix tests" (a POSIX test suite?) that both NetBSD and FreeBSD import as a third-party product. Then, each system would just extend these suites with local patches to verify their system-specific behavior. A very nice option in my opinion, but this is a major effort. (And this can be tackled later once we the FreeBSD Test Suite is more complete and we can easily spot how much it overlaps to the NetBSD one.)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFY7cWBeJuzQStvf7fyCXB957gDB=o87yKv%2Bgt9fAjwLG1QgzA>