Date: Wed, 27 Sep 1995 17:29:17 -0700 (MST) From: Terry Lambert <terry@lambert.org> To: wilko@yedi.iaf.nl (Wilko Bulte) Cc: jkh@time.cdrom.com, terry@lambert.org, kelly@fsl.noaa.gov, gryphon@healer.com, freebsd-hackers@FreeBSD.ORG, freebsd-ports@freebs.org.iaf.nl Subject: Re: ports startup scripts Message-ID: <199509280029.RAA11400@phaeton.artisoft.com> In-Reply-To: <199509272101.WAA01282@yedi.iaf.nl> from "Wilko Bulte" at Sep 27, 95 10:01:17 pm
next in thread | previous in thread | raw e-mail | index | archive | help
> > > The product is QA'd with the validation suite. 2 hours were required for > > > the validation suite, which is interesting, considering POSIX validation > > > takes less time to run. > > > > Sorry, this statement simply leads me to believe that you've never > > worked on anything of significant size or complexity. 2 hours is > > about how long it took to brief the QA team on how to structure the > > run at most large ISVs I've worked at! :-) If you got the results back > > in 3 or 4 days you considered it a rush job. > > I used to do (amongst other things) X/Open validation work. Believe me > that you are going for your friendly colleagues throats when the XPG > suite falls over once more. I've run both the X/Open PVS and NIST/PCTS. Actually ran them on 1.X BSD at one time. Unfortunately I was unable to send the 20 or so patches and didn't work for a certified testing laboratory. > Generally say 4 - 5 runs were needed to get everything thru the > XPG. And that was only XPG/2. XPG > 2 is worse. The SVVS (SVID test) > is a piece of &*^*( This is assuming a failure. This is also assuming that you care about testing anything other than a few interfaces. You use as few system interfaces as possible in portable software -- it's one of the things that makes it portable. The SVVS is useless. UnixWare and Solaris both passed SVVS (I ran them for reference platform designation for the NetWare for UNIX product), but at the time, both were technically not SVID compliant. The test just didn't look for all aspects of conformance. UnixWare and SCO are *still* not SVID compliant (read SVID III, gettimeofday(RT), setitimer(RT), and getitimer(RT), with special emphasis on the phrases "system time" and "system time update frequency"). > And don't talk about bugs in your infallible test suite.. Everyone knows TET and ETET are bug free. 8-) 8-). > And who writes this flawless testsuite BTW? A piece of software called "battlemap". Other software. Mostly *not* human beings. I never claimed it was flawless, only that a human running test cases is inherently slower and provides less code coverage. If you run a fast machine test, throwing a human at it won't result in a better product, only a waste of the humans time. The human will make the same mistakes running from a writen procedure that a test suite coded from the same written procedure would make -- and will probably add data entry and interpretation mistakes as well. Terry Lambert terry@lambert.org --- Any opinions in this posting are my own and not those of my present or previous employers.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199509280029.RAA11400>