Date: Tue, 22 Jul 2008 23:26:42 -0700 From: "Garrett Cooper" <yanefbsd@gmail.com> To: "Alexander Leidinger" <Alexander@leidinger.net> Cc: jkoshy@freebsd.org, FreeBSD Hackers <freebsd-hackers@freebsd.org> Subject: Re: TET and other testing framework for FreeBSD Message-ID: <7d6fde3d0807222326m3b906d17w76fdb2943d9c38d5@mail.gmail.com> In-Reply-To: <20080723072122.29891m0id5yoctes@webmail.leidinger.net> References: <7d6fde3d0807212343i69526416h35a3d961fac0c9d9@mail.gmail.com> <20080722111038.21404k434075zx8g@webmail.leidinger.net> <7d6fde3d0807220831u185d6c6dlbb6a92cdbedb6be@mail.gmail.com> <20080723072122.29891m0id5yoctes@webmail.leidinger.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Jul 22, 2008 at 10:21 PM, Alexander Leidinger <Alexander@leidinger.net> wrote: > Quoting "Garrett Cooper" <yanefbsd@gmail.com> (from Tue, 22 Jul 2008 > 08:31:12 -0700): > >> On Tue, Jul 22, 2008 at 2:10 AM, Alexander Leidinger >> <Alexander@leidinger.net> wrote: >>> >>> Quoting "Garrett Cooper" <yanefbsd@gmail.com> (from Mon, 21 Jul 2008 >>> 23:43:11 -0700): > >>>> I'm also working on getting tst_res (1.) dually licensed from >>>> the LTP folks with a BSD / GPL license to allow for better inclusion >>>> into FreeBSD's infrastructure, to ensure that we have a deterministic >>>> means of dealing with testcases and reporting. >>>> Just wondering what, if any, interest would be in adopting and >>>> applying this work to FreeBSD for the good of the community. >>> >>> If this is what is used in the LTP to run some tests, I have to say it >>> does >>> not look mature. I've run the LTP a lot in FreeBSD (native linux >>> binaries, >>> http://wiki.FreeBSD.org/linux-kernel/ltp ... LTP not run and updated the >>> page since a long time) to test the linux compatibility layer, and the >>> are >>> several places where tests fail and no output is produced or even the >>> summary said the test passed. I even opened bug reports at the LTP page >>> on >>> SF, but it seems nobody was interested in those reports. >> >> tst_res consists of the functions: >> >> tst_res >> tst_resm >> tst_brk >> tst_brkm >> tst_brkloop >> tst_brkloopm >> tst_exit >> >> which (thanks to some feedback) has finally made it up onto its own >> online manpage: >> >> http://ltp.sourceforge.net/ltpmantemp.php?file=man3/tst_res.html >> >> This is the simple output and behavior modification API's that produce >> messages like (according to the sample on the manpage): >> >> tsttcs01 1 PASS : Able to create MAXUP processes >> tsttcs01 2 FAIL : Too many processes (MAXUP+1) created >> tsttcs01 3 BROK : tabinfo(PROCTAB, &tbs) failed; errno = >> 13: Permission denied >> tsttcs01 4-10 BROK : Remaining cases broken >> tsttcs01 0 WARN : cleanup(): kill(0, SIGALRM) failed; errno >> = 3: No such process > > And this is where I've seen problems while testing the linux compatibility > layer in FreeBSD. Some tests said PASS when in reality they didn't pass but > died. I don't know if this is fixed now, it's been a while when I reported > this. Yeah, and I've seen tests hang on LTP, but that's because people don't monitor test execution but just claim that since it works, it's fine. I try to keep abreast of these things because test quality's important; otherwise your results are worth no more than a $2 bill! >>> There are also some other ideas, like using the protocol the perl test >>> suites use, to be able to use existing perl stuff to generate reports and >>> overviews out of the generated logs (AFAIR this was the idea behind some >>> changes to the existing regression tests a long time ago). >> >> I have yet to really explore Test::Harness, but while I like Perl and >> I'm sure that Test::Harness is as good as people laud it to be, I find >> dealing with Perl to be cumbersome as they tend to break some stuff >> between releases =(... > > I don't know if I was clear, the tests themself are written in C (or > whatever) and not in perl, the goal as I remember it was to be able to use > the already existing perl tools where we don't care about the development, > as someone else will fix broken things because it is used for perl itself. > > Bye, > Alexander. Hmmm... I've also been forced into working with Python nose lately, and while I might not be a Python officiando, I do find the framework to be expeditious and simple to use. Just another thought, because Python seems to be a better base framework... -Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7d6fde3d0807222326m3b906d17w76fdb2943d9c38d5>