From owner-freebsd-hackers@FreeBSD.ORG Sun Nov 16 17:00:31 2008 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78F431065677 for ; Sun, 16 Nov 2008 17:00:31 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.158]) by mx1.freebsd.org (Postfix) with ESMTP id 02F2C8FC0A for ; Sun, 16 Nov 2008 17:00:30 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: by fg-out-1718.google.com with SMTP id l26so1715648fgb.35 for ; Sun, 16 Nov 2008 09:00:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=VZdVx+AkuwMU722S4ih2ajQmfRV45m3/0TdXW+L8byM=; b=sEroppaGlIuoWkUdIRMPjDiGSFy425cqBHhnofn8fDVyV27zqDE7Zcf45xoqm2SD5J N+m6ZRa54qOhQxVqI+XAhJZfGIQi5DQXy9+nghsP5t2vhFDihh20oq/XnbBJP4IMRzXy cD0xbvVv6tdQXrLn7FwU6N6Ui0F97blTNdji4= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=rN8HgT3UpTMkqm+2UcRXrLu5VURN9aYt5XF8Ux7qDy9zZLNwBqzlIlhNpb801neral HrHMkWpI2cMrEMwChLG4E5ft4R/uMqYNWGIClq3o61sh2oWowk+hXUCTDwb/hQRW+brs I2/GNn1clq752OPa23i0DnI7fWyh4IcFZQCJM= Received: by 10.86.59.18 with SMTP id h18mr1795157fga.31.1226854829577; Sun, 16 Nov 2008 09:00:29 -0800 (PST) Received: by 10.86.76.13 with HTTP; Sun, 16 Nov 2008 09:00:29 -0800 (PST) Message-ID: <364299f40811160900r420cd841xb8b1546692158b33@mail.gmail.com> Date: Sun, 16 Nov 2008 09:00:29 -0800 From: "Garrett Cooper" To: "Alexander Leidinger" In-Reply-To: <20081116085753.10415gcgqo0etm04@webmail.leidinger.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <1155143A-D9F9-4988-B648-F9CDC2A4080C@gmail.com> <20081116085753.10415gcgqo0etm04@webmail.leidinger.net> X-Mailman-Approved-At: Sun, 16 Nov 2008 17:41:33 +0000 Cc: hackers@freebsd.org Subject: Re: Request for individuals interested in reviewing test / python topics X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Nov 2008 17:00:31 -0000 On Sat, Nov 15, 2008 at 11:57 PM, Alexander Leidinger wrote: > Quoting Garrett Cooper (from Sat, 15 Nov 2008 14:08:02 > -0800): > >> Hello Hackers and Porters, >> I'm currently working on a proposal to the FreeBSD foundation to >> use Python Nose as a testing framework for writing tests. If there are > > Are you aware of the history of the current regression tests? > > If not: > > It started without a structure, then some work was done to move to the perl > testing framework style (really only the output of the tests, and the naming > conventions in the directory). This was not completed, and newer tests may > not comply. > > The reason for choosing the perl style was, to be able to use the extensive > perl tools to > - automatically run all the tests > - be able to compare different runs with the perl tools > - be able to generate a lot of different output formats (html/text/...) > > There's also a wiki page about testing, which you may want to check out: > http://wiki.freebsd.org/TetIntegration > > I don't really know python nose. I just looked at it quickly and can not see > any big benefit compared to the perl test protocol outlined above (and the > stuff outlined in the wiki looks even more advanced than that). Would you > please elaborate where you see the benefits of it? > > Note that during release building perl is needed anyway to generate the > index for the ports collection. I don't know if python is required currently > during the release generation. > > Bye, > Alexander. Alexander, Thanks for getting back with me so quickly. Let me help explain a bit... About selecting tests -- Ok. I gave a quick look at Test::Simple, Test::Harness, etc and although Perl's Test:: modules appear to be rather expedited for executing entire tests, it's not helpful when executing selected tests, unless you write wrappers -- which is part of the point where the pain in adding infrastructure comes in. The default selector plugin that comes with nose is smart enough to crack open the files in a directory structure, do some regexed based searching for ^\s+def\s+test, or something along those lines, and from that you can select individual files with testcases, individual functions in files to execute, or classes, or instances of classes, etc. So if you're developing a fix for a feature bugfix and were provided a single regression test to use for finding the issue and were adding that test to a master test file (for suite execution), but instead wanted to execute only one testcase -- nose supports that functionality out of the box with little change for the user with the default selector plugin: nosetests some_file.py:some_func Furthermore, About capture and printout -- Also, nose with plugins gives you the ability to print not just in plaintext output on a console, but also HTML (I tried looking for the example on the site that was available before, but it appears to be missing now). The example probably consisted of approximately 30~40 lines real code, which for the most part had embedded HTML formatting tags, etc. So, a basic output capture plugins with nose that prints out HTML can be written with <= 25 lines. I'm not sure whether or not Python is required for release generation, but it will have to be listed as one of the required tools in the proposal I'm going to write. Thankfully the required dependencies for Python are small (just a base system with the typical components), and if using Python v2.5+, you also have access to ctypes (a means for testing C-API's in libraries), which is incredibly beneficial I've discovered through my work at Cisco. Assembler instructions would potentially need to be added for architectures other than x86. I'll have to look into this further with Python 2.5 (we currently use Python 2.4.5 -- the last maintenance release of Python -- in our group). Hope that helps explain things a bit more, -Garrett