From owner-freebsd-hackers@FreeBSD.ORG Wed Jul 23 06:26:44 2008 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 994FB106564A for ; Wed, 23 Jul 2008 06:26:44 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.156]) by mx1.freebsd.org (Postfix) with ESMTP id 2CC138FC19 for ; Wed, 23 Jul 2008 06:26:43 +0000 (UTC) (envelope-from yanefbsd@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so1372460fgb.35 for ; Tue, 22 Jul 2008 23:26:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=5BXwdTUNPsuWxeD1QZ7vQ2bMALd2K5g89ftd6DBO6A0=; b=sM3gy3o2EpKijEJsvulleack8Ee6HwCPhaVKApmWl6SqbrVYvTGem2AdV74slOv2Ew fyBvCyE2gMaRQuu8PLoh9uKcxuIVRxG3Mn6OEGwEDCiGNnlksdeRUdi5wQU7ThrR1hdo WxVbY4gtzKhNVLXVHvvBYCAGHWQj/Wxm3U95Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=q+1tjuFM38PhQHcKgJB9ltIC9JtV3DAEPAlkO3b+DishZxjomiR+5FS1BuuQI6xm16 7fbB3+Mx8ik3oMLro79aFpWWHxA6wa5dV7BYrvclcgf6moxBbEbZP2l4oCeXhm/SOaFj igz0txfwI6viaGCBlZqjvOiR5WHD7rquw413s= Received: by 10.86.93.19 with SMTP id q19mr4479293fgb.67.1216794402507; Tue, 22 Jul 2008 23:26:42 -0700 (PDT) Received: by 10.86.51.1 with HTTP; Tue, 22 Jul 2008 23:26:42 -0700 (PDT) Message-ID: <7d6fde3d0807222326m3b906d17w76fdb2943d9c38d5@mail.gmail.com> Date: Tue, 22 Jul 2008 23:26:42 -0700 From: "Garrett Cooper" To: "Alexander Leidinger" In-Reply-To: <20080723072122.29891m0id5yoctes@webmail.leidinger.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <7d6fde3d0807212343i69526416h35a3d961fac0c9d9@mail.gmail.com> <20080722111038.21404k434075zx8g@webmail.leidinger.net> <7d6fde3d0807220831u185d6c6dlbb6a92cdbedb6be@mail.gmail.com> <20080723072122.29891m0id5yoctes@webmail.leidinger.net> Cc: jkoshy@freebsd.org, FreeBSD Hackers Subject: Re: TET and other testing framework for FreeBSD X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jul 2008 06:26:44 -0000 On Tue, Jul 22, 2008 at 10:21 PM, Alexander Leidinger wrote: > Quoting "Garrett Cooper" (from Tue, 22 Jul 2008 > 08:31:12 -0700): > >> On Tue, Jul 22, 2008 at 2:10 AM, Alexander Leidinger >> wrote: >>> >>> Quoting "Garrett Cooper" (from Mon, 21 Jul 2008 >>> 23:43:11 -0700): > >>>> I'm also working on getting tst_res (1.) dually licensed from >>>> the LTP folks with a BSD / GPL license to allow for better inclusion >>>> into FreeBSD's infrastructure, to ensure that we have a deterministic >>>> means of dealing with testcases and reporting. >>>> Just wondering what, if any, interest would be in adopting and >>>> applying this work to FreeBSD for the good of the community. >>> >>> If this is what is used in the LTP to run some tests, I have to say it >>> does >>> not look mature. I've run the LTP a lot in FreeBSD (native linux >>> binaries, >>> http://wiki.FreeBSD.org/linux-kernel/ltp ... LTP not run and updated the >>> page since a long time) to test the linux compatibility layer, and the >>> are >>> several places where tests fail and no output is produced or even the >>> summary said the test passed. I even opened bug reports at the LTP page >>> on >>> SF, but it seems nobody was interested in those reports. >> >> tst_res consists of the functions: >> >> tst_res >> tst_resm >> tst_brk >> tst_brkm >> tst_brkloop >> tst_brkloopm >> tst_exit >> >> which (thanks to some feedback) has finally made it up onto its own >> online manpage: >> >> http://ltp.sourceforge.net/ltpmantemp.php?file=man3/tst_res.html >> >> This is the simple output and behavior modification API's that produce >> messages like (according to the sample on the manpage): >> >> tsttcs01 1 PASS : Able to create MAXUP processes >> tsttcs01 2 FAIL : Too many processes (MAXUP+1) created >> tsttcs01 3 BROK : tabinfo(PROCTAB, &tbs) failed; errno = >> 13: Permission denied >> tsttcs01 4-10 BROK : Remaining cases broken >> tsttcs01 0 WARN : cleanup(): kill(0, SIGALRM) failed; errno >> = 3: No such process > > And this is where I've seen problems while testing the linux compatibility > layer in FreeBSD. Some tests said PASS when in reality they didn't pass but > died. I don't know if this is fixed now, it's been a while when I reported > this. Yeah, and I've seen tests hang on LTP, but that's because people don't monitor test execution but just claim that since it works, it's fine. I try to keep abreast of these things because test quality's important; otherwise your results are worth no more than a $2 bill! >>> There are also some other ideas, like using the protocol the perl test >>> suites use, to be able to use existing perl stuff to generate reports and >>> overviews out of the generated logs (AFAIR this was the idea behind some >>> changes to the existing regression tests a long time ago). >> >> I have yet to really explore Test::Harness, but while I like Perl and >> I'm sure that Test::Harness is as good as people laud it to be, I find >> dealing with Perl to be cumbersome as they tend to break some stuff >> between releases =(... > > I don't know if I was clear, the tests themself are written in C (or > whatever) and not in perl, the goal as I remember it was to be able to use > the already existing perl tools where we don't care about the development, > as someone else will fix broken things because it is used for perl itself. > > Bye, > Alexander. Hmmm... I've also been forced into working with Python nose lately, and while I might not be a Python officiando, I do find the framework to be expeditious and simple to use. Just another thought, because Python seems to be a better base framework... -Garrett