Date: Thu, 24 Jul 2014 13:39:49 -0700 From: Garrett Cooper <yaneurabeya@gmail.com> To: Julio Merino <jmmv@freebsd.org> Cc: Garrett Cooper <yanegomi@gmail.com>, "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org> Subject: Re: Need input on preference on location of 3rd party tests vs FreeBSD tests Message-ID: <CAGHfRMAnv-cAZCxBDGtgwr3ZcNTpnzps4XT%2B%2By2sUVz02yjMVA@mail.gmail.com> In-Reply-To: <CAFY7cWBeJuzQStvf7fyCXB957gDB=o87yKv%2Bgt9fAjwLG1QgzA@mail.gmail.com> References: <B1CE2158-B8C1-4D97-AB7A-031ADF55C435@gmail.com> <E5C29F6D-C632-4B7B-B8C8-0816C93DB7B7@gmail.com> <CAOtMX2izFjpC7r7Vxtk68KynAeuFvaDNNbK5nOEGcdFy7vJE5A@mail.gmail.com> <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com> <CAFY7cWBeJuzQStvf7fyCXB957gDB=o87yKv%2Bgt9fAjwLG1QgzA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jul 24, 2014 at 12:24 PM, Julio Merino <jmmv@freebsd.org> wrote: > On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper <yanegomi@gmail.com> wrot= e: >> On Jul 18, 2014, at 7:45 AM, Alan Somers <asomers@freebsd.org> wrote: >> >>> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper <yaneurabeya@gmail.com>= wrote: >>>> Hi all, >>>> One of the things that I've done on my fork of FreeBSD is I've impor= ted ATF test suites from NetBSD and I have integrated existing test suites = from freebsd's tools/regression tree into Kyua as well. Due to the size and= difference in test content/coverage, I pulled lib/libc and lib/msun from b= other sources and integrated them into Kyua. What I did was I put the netbs= d testcases into the tests/ subdirectory and put the FreeBSD test suites in= to a tests/legacy subdirectory. The goal was that the legacy directory woul= d eventually be converted over to atf testcases and then could be removed o= nce the conversion was complete. >>>> I'm not sure if this scheme makes sense though. Does anyone have a p= reference as to whether or not this makes sense? >>>> Thanks! >>>> -Garrett >>> >>> >>> I don't understand. What did you put in tests/legacy? >> >> The tests from tools/regression. tests/ contains the tests from NetBSD. > > I do not think adding tests under src/tests/ is a good idea. We have > chosen to put tests under the 'tests' subdirectories of the affected > components and we should stick to this rule except for very specific > cases (see src/tests/sys/). Creating an artificial barrier between > tests imported from NetBSD and native tests to FreeBSD is not > beneficial (e.g. why should users/developers care where the tests came > from?). Sorry if this was a bit misleading in my original email... Basically I'm trying to solve the problem of keeping the FreeBSD testcases isolated from the NetBSD testcases. This only impacted two components (so far) -- libc and msun. What I did was placed the libc testcases from NetBSD in lib/libc/tests and the libc testcases from FreeBSD in lib/libc/tests/legacy (kind of matching the model that we have for TAP/plain scripts with the "legacy" name, but in name only). Example: - https://github.com/yaneurabeya/freebsd/tree/isilon-atf/lib/libc/tests - the NetBSD testcases (mostly) - https://github.com/yaneurabeya/freebsd/tree/isilon-atf/lib/libc/tests/leg= acy - the testcases from tools/regression on FreeBSD head > "Reusing the NetBSD Makefiles" is not a strong enough reason because > those are usually trivial -- and if they aren't, that's an indication > that they are wrong! Well, they're really trivial, yes, and apart from small caveats with bsd.progs.mk in HEAD today, there isn't a reason for the testcase Makefiles to be divergent, except when dealing with FreeBSD vs NetBSD MK_ knobs. > We have two options: > > 1) We treat the NetBSD tests as upstream. In this case, we ought to do > the right thing, which is to put them in a vendor branch, merge into > contrib, and add reachover Makefiles where necessary. This is what I was thinking of doing because I don't have a compelling argument and it doesn't really seem to make sense to fork [at this point]. The modifications that Peter and I have been making to the isilon-atf branch are relatively trivial/documented with #if defined(__FreeBSD__) or #if defined(__NetBSD__) (still not happy with the __arraycount vs nitems divergence, but it's a simple sed one-liner). I've pushed a number of patches out to NetBSD GNATS to fix issues with the testcases (mostly header discrepancies, but also some valid testcase bugs with NetBSD's copy of the tests). > OR > > 2) We just copy the code (effectively a fork), and when doing so we > can do as we wish with the structure. > > So... > > Option 1 may sound nice, but it will be problematic long-term without > collaboration from NetBSD: I don't think we want to treat NetBSD's > tests as upstream if NetBSD themselves don't consider the tests as a > "first class project" (and they don't currently). Not to mention that > this will require a significant amount of local patching with the > corresponding maintenance headaches. > > Option 2 is definitely easier to handle on our side, but with the > obvious disadvantage that we won't be able to easily import tests from > NetBSD. How big of a problem is that today? Honestly the fact that our > codebases are separate for things like libc is more of a problem than > having diverging tests. > > Option 3 (yes, another one!) would be to start a third-party > repository holding "generic Unix tests" (a POSIX test suite?) that > both NetBSD and FreeBSD import as a third-party product. Then, each > system would just extend these suites with local patches to verify > their system-specific behavior. A very nice option in my opinion, but > this is a major effort. (And this can be tackled later once we the > FreeBSD Test Suite is more complete and we can easily spot how much it > overlaps to the NetBSD one.) This is an alternative as well, but the tricky thing is that it moves away from the whole idea of "having the tests in the tree" which was part of the point behind doing things this way (to get developers in the habit of writing tests that match the test code), unless both FreeBSD and NetBSD collaboratively pull test code from the "BSD Test Project". Thanks! -Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGHfRMAnv-cAZCxBDGtgwr3ZcNTpnzps4XT%2B%2By2sUVz02yjMVA>