From owner-freebsd-ports Tue Aug 28 1:17: 0 2001 Delivered-To: freebsd-ports@freebsd.org Received: from ipcard.iptcom.net (ipcard.iptcom.net [212.9.224.5]) by hub.freebsd.org (Postfix) with ESMTP id 3CEE137B40A for ; Tue, 28 Aug 2001 01:16:49 -0700 (PDT) (envelope-from sobomax@FreeBSD.org) Received: from vega.vega.com (dialup14-51.iptelecom.net.ua [212.9.229.115]) by ipcard.iptcom.net (8.9.3/8.9.3) with ESMTP id LAA03084; Tue, 28 Aug 2001 11:16:32 +0300 (EEST) (envelope-from sobomax@FreeBSD.org) Received: from FreeBSD.org (big_brother.vega.com [192.168.1.1]) by vega.vega.com (8.11.4/8.11.3) with ESMTP id f7S8FBo53333; Tue, 28 Aug 2001 11:15:11 +0300 (EEST) (envelope-from sobomax@FreeBSD.org) Message-ID: <3B8B5354.EE77877D@FreeBSD.org> Date: Tue, 28 Aug 2001 11:16:20 +0300 From: Maxim Sobolev Organization: Vega International Capital X-Mailer: Mozilla 4.78 [en] (Windows NT 5.0; U) X-Accept-Language: en,uk,ru MIME-Version: 1.0 To: mi@aldan.algebra.com Cc: ports@FreeBSD.org Subject: Re: RFC: on the merits of post-build testing References: <200108272211.f7RMB1k22861@misha.privatelabs.com> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 7bit Sender: owner-freebsd-ports@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.org mi@aldan.algebra.com wrote: > Hello! > > Some of the software packages ported to FreeBSD (ported software or > software) come with own suite of post-build tests. Some ports try to use > those tests to check the results of the build prior to installation. > While it seems like a Good Idea(TM) to yours truly, some others > (violently) object to it. In this letter, I'll try to summarize and > illustrate why I like it, why you may dislike it, and why you might be > wrong disliking it :-) > > The benefit is rather obvious: it helps catching bugs. > > Bugs of different nature -- of the ported software, of the port, of the > local configuration (as in "over optimization" requested by the user > through the CFLAGS setting, for example). I look forward for the new > version of the compiler, for another example :-) > > Why some dislike it: > > * it is a waste of the CPU time > > well, so is checking for the result of the malloc() :-) So is not > compiling your kernel with -fomit-frame-pointer. I think, that majority > of those concerned about the CPU time will use the precompiled packages. > Testing time is, usually, only a fraction of the build time. > > * all such detectable errors _could_ be detected prior to > committing the port > > I wish. Take the graphics/lcms port, for example. The lcms author does > development on WinNT x86. The current port compiles on FreeBSD-Alpha, > but does not pass its own tests. If those weren't turned on by the port > maintainer, the author's numerous wrong assumptions about the sizes of > ints and longs would've been mysteriously crashing programs, which use > liblcms on the Alphas... Although I could've tested the thing on beast > prior to committing, how many people actually do such a thing? For > ports, that require other ports, this is not even possible, unless you > happen to have root-access to beast. The tests ran fine for me on > -current and -stable. They also ran fine on bento... > > * all such detectable errors _should_ be detected prior to > committing the port > > Well, sometimes, it is the users' CFLAGS, that the maintainer has no > control of. Sometimes it is some interaction with some other port, > the amount of memory, screen resolution -- there can be plenty of > situations, when the software builds, but does not work. I'm sure > everyone here has a story or two. > > * it is not going to guarantee anything anyway > > No. Neither is FreeBSD (nor any other software I know of) guaranteed to > work -- at all. Yet, some times it works. The testing will catch the > errors, the software's author(s) deemed likely to happen. > > * having to maintain the tests as well is an unneeded burden on > the port maintainer > > Yes, it is a burden, although not unbearable. No, it is needed, IMO. > > * this might be a good idea for some ports, but not the others, > therefore no general decision is possible/desirable > > Personally, I don't understand this, but even if you do, I'm not saying > we make such testing a requirement. IMO, it should be encouraged, > however. > > Assuming, your read it all up to here, here is what I propose: > > . add handling of the ``test'' target if defined in the port's > Makefile by the bsd.ports.mk; > > . make sure the test target is exercised automaticly after the > build and/or before the install; > > . allow the testing to be turned off with something like > NOPORTTESTS=YES > to satisfy (some of) my opponents; just like NOPORTDOCS, the > flag should not be set by default; > > . encourage the existing and new ports to check the ported > software for the bundled tests and create the above described > test-target whenever possible; if some of the tests fail due > to problems with the tests themselves (Tcl and Tk tests are > pretty horrible at that, for example), they should be patched > or taken out of the sequence (by patching the provided > test-scripts). > > Comments anyone? As a fairly fresh committer, I'm not sure what kind of > quorum/consensus is needed for something like this to make it through. That are a very interesting arguments, but the real point is that in 99.9% of cases running tests wouldn't cause anything but useless waste of CPU time. From the remainder 0.09% is associated with people adding unsupported optimisation levels into CFLAGS (they deserve punishment for that anyway) and 0.01% with people running strange hardware (i.e. Alphas and faulty x86). Therefore, I'm still convinced that this micro-optimisation would not bring us anything but additional maintenance headache, thus diverting valuable developers' resources from the really important problems affecting much wider auditory of users. -Maxim To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-ports" in the body of the message