From owner-freebsd-current@FreeBSD.ORG Sun Feb 24 20:02:05 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id EFCC1C64; Sun, 24 Feb 2013 20:02:05 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-vc0-f178.google.com (mail-vc0-f178.google.com [209.85.220.178]) by mx1.freebsd.org (Postfix) with ESMTP id 712AB19C; Sun, 24 Feb 2013 20:02:05 +0000 (UTC) Received: by mail-vc0-f178.google.com with SMTP id m8so1378548vcd.9 for ; Sun, 24 Feb 2013 12:01:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=ACLTqlIYaftlY9WSdipGA8fpXefDpIJCDpkqt0pEsbY=; b=N98eH6HwPgE2GtuJ9eUOwR1x4rPQgsSp39sck2EtY6WxLrymFVme5zvwBD1vSI3yv1 VF/yLJcvR97SBsaGbkU6zAulTvjSGLNUkGJKNgeucjf35XRUW6nK5Vtx5opPMy6z7kAP 0rQqdNdXXqVwzo0yljZnXbfjHnAey4ae36qzc/z3tuGLWn73K2sJ+OFZIHdozh3LIUAE +TQdnouviU1uMqr93dGd88/CiDLkCJ0DAduaBzpBsGlDdKSIFJoXbTMNzeHomcIkmwWp X777VPZ8VToU4MgKQx4rJ9pt3aX6MSEmEZpfUCWR7sl6+nP9CQurvj4t5u8NJheh6FMg z7Qg== MIME-Version: 1.0 X-Received: by 10.52.68.116 with SMTP id v20mr7593691vdt.126.1361736118822; Sun, 24 Feb 2013 12:01:58 -0800 (PST) Received: by 10.58.170.36 with HTTP; Sun, 24 Feb 2013 12:01:58 -0800 (PST) In-Reply-To: <20130224172524.GA24919@saturn> References: <51263782.3020205@freebsd.org> <20130224172524.GA24919@saturn> Date: Sun, 24 Feb 2013 12:01:58 -0800 Message-ID: Subject: Re: FreeBSD Testing Facility From: Mehmet Erol Sanliturk To: Giorgos Keramidas Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-current@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Feb 2013 20:02:06 -0000 On Sun, Feb 24, 2013 at 9:25 AM, Giorgos Keramidas wrote: > On 2013-02-21 07:04, Matthew Jacob wrote: > > On 2/21/2013 5:04 AM, Mehmet Erol Sanliturk wrote: > > >Dear All , > > > > > >During development of FreeBSD , testing is very vital . > > >... > > > > This in general is a good suggestion. Most companies do such > > automated testing as a matter of course. > > > > Note however that this is a volunteer effort. Were you volunteering > > to set up such an automated, possibly testzilla driven, facility? It > > would certainly help the quality, although as others have noted > > snapshots are often likely to be broken. > > To the OP: > > Like Matthew has said, this is a volunteer effort. So if you have > experience with setting up testing automation, and you are willing to > help us set up something like this, please go ahead :-) > > I've worked in places where the following types of tests are used: > > - Presubmit tests that check specific parts of functionality. > > - Commit-related tests that run asynchronously in the background, > and report back later (e.g. through email). > > - Test systems that cache previous results and report a simple 'green' > vs. 'red' status for _every_ single commit. > > - System tests that check for particular functionality, health > criteria, etc. - some times fully automated, some other times > requiring a token amount of manual support. > > So here are two important questions, regarding the tests you mentioned: > > When you speak about 'testing FreeBSD', which type of tests are you > interested in seeing? > > Are you willing to help us set up something that runs the type of tests > that you want to see? > > To another message I replied as follows : http://lists.freebsd.org/pipermail/freebsd-current/2013-February/039984.html I want say that again to manage such a system is not possible for me very much , additionally for lack of facilities . My wish is that let's take this subject to be worked in detail and by a group effort , design , implement and apply such a testing facility . In this work , if anything I can supply , I will never keep myself back . Without reflective responses , if we consider my first message : http://lists.freebsd.org/pipermail/freebsd-current/2013-February/039976.html a very rough outline is supplied . Reason is such a suggestion is to see many failures of *BSD systems that these can be eliminated if such a system is designed , implemented , and applied . In many messages by FreeBSD developers , it is said that it is not possible to establish a testing farm by the FreeBSD project and maintain it , which is quite correct . Assume the following : If any one is downloading and trying to install and run Current development iso files , itis very likely that the main intention is to test and help to its development . By counting number of such testing attempts , it is possible to estimate the number of people which can participate to a more coordinated testing system . My opinion is that such a system ( a distributed testers population ) is already present , and over time , even it may be enlarged . Actually , problem is large enough to establish a working group to attack it . There is a necessity to have committers to generate the necessary ftp sites and manage them . At present , there is a port system . The testing facility will be similar to that system , but different programs will be employed . The first action will be to design what will be tested and how . Since FreeBSD is composed from components ( boot , kernel , each system programs , etc. , ) , every component will be a testing subject . In the svn , there are already testing parts . By combining these , and supplying , developing missing parts will generate a testing framework . At the beginning , it will not be possible to generate a state of the art testing facility . By starting from the existing parts , over time , it will be possible to ad tests one by one . This will be similar to development of ports system : Over time it has been populated one by one . As a an applicable first step , a FreeBSD developer , may establish a wiki page or use an existing one to develop a "Testing Requirements and Design , Implementation and Applications" page . A mailing list may be established to discuss testing problems . An ftp site may be established to apply tests as suggested in my first message . As an example : Assume that a part related to video display cards is under development , such as KMS . The developer(s) will have a limited number of cards available in their computers . A potential people population exists which they use such cards , for example RADEON , or INTEL cards . In the mailing lists , it may be announced that testers are needed for specific video card branch , such as RADEON . People , fills a form to participate in the tests . The developer prepares a file just like a port/package . Sends a message to the subscribers wishing to test this problem . Possible testers , upon receiving that message , may enter a command , such as # test_apply name_of_testing_package just similar to pkg_add command . The test_apply program , downloads the testing package and applies it . At the end , the best action is to generate a structured e-mail , the results may be sent to the testing data base just a like bug tracker system . I emphasize structured messages for their automatic processing possibility . If this is not very feasible , at least a formal message may be requested to evaluate results easily . Otherwise results like "Tests failed" will not be very helpful for the developers . A script in there processes the returned e-mails and generates a report to developer(s) ,. A cycle of such tests continues up to a satisfactory result . Some parts , such as the following , are dependent very much to hardware to be used : - Booting process - Network communications - Video display units - USB attached components - Parts requiring BIOS cooperation , - Parts detecting , managing main board parts ( circuits , ports , etc. ) - Parts detecting , managing attached peripheral devices . All of these may be subject of such distributed testing actions . After testing many of the components and ensuing that they are working , there comes to testing of their combinations , for example : - File systems in local disks, - NFS like systems Then , a whole test : Testing complete install iso files . Here , there will not be a booting or hardware detection failure , but failures related to cooperative working of systems under user applications . These failures may occur between mismatched components during their interaction . Therefore , such a testing facility requires cooperation of many people to function properly . A group of leaders may organize the efforts . Thank you very much .