Date: Sun, 24 Feb 2013 12:01:58 -0800 From: Mehmet Erol Sanliturk <m.e.sanliturk@gmail.com> To: Giorgos Keramidas <keramida@freebsd.org> Cc: freebsd-current@freebsd.org Subject: Re: FreeBSD Testing Facility Message-ID: <CAOgwaMt9qKJYFu-EB-QetRhgdd8CmJbND2HQLKP_YTN6rnZ9tQ@mail.gmail.com> In-Reply-To: <20130224172524.GA24919@saturn> References: <CAOgwaMsbupU0NB7TA6RMJW0erWBGwcsGO16xt3GPLsPuh-rpRA@mail.gmail.com> <51263782.3020205@freebsd.org> <20130224172524.GA24919@saturn>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Feb 24, 2013 at 9:25 AM, Giorgos Keramidas <keramida@freebsd.org>wrote: > On 2013-02-21 07:04, Matthew Jacob <mjacob@freebsd.org> wrote: > > On 2/21/2013 5:04 AM, Mehmet Erol Sanliturk wrote: > > >Dear All , > > > > > >During development of FreeBSD , testing is very vital . > > >... > > > > This in general is a good suggestion. Most companies do such > > automated testing as a matter of course. > > > > Note however that this is a volunteer effort. Were you volunteering > > to set up such an automated, possibly testzilla driven, facility? It > > would certainly help the quality, although as others have noted > > snapshots are often likely to be broken. > > To the OP: > > Like Matthew has said, this is a volunteer effort. So if you have > experience with setting up testing automation, and you are willing to > help us set up something like this, please go ahead :-) > > I've worked in places where the following types of tests are used: > > - Presubmit tests that check specific parts of functionality. > > - Commit-related tests that run asynchronously in the background, > and report back later (e.g. through email). > > - Test systems that cache previous results and report a simple 'green' > vs. 'red' status for _every_ single commit. > > - System tests that check for particular functionality, health > criteria, etc. - some times fully automated, some other times > requiring a token amount of manual support. > > So here are two important questions, regarding the tests you mentioned: > > When you speak about 'testing FreeBSD', which type of tests are you > interested in seeing? > > Are you willing to help us set up something that runs the type of tests > that you want to see? > > To another message I replied as follows : http://lists.freebsd.org/pipermail/freebsd-current/2013-February/039984.html I want say that again to manage such a system is not possible for me very much , additionally for lack of facilities . My wish is that let's take this subject to be worked in detail and by a group effort , design , implement and apply such a testing facility . In this work , if anything I can supply , I will never keep myself back . Without reflective responses , if we consider my first message : http://lists.freebsd.org/pipermail/freebsd-current/2013-February/039976.html a very rough outline is supplied . Reason is such a suggestion is to see many failures of *BSD systems that these can be eliminated if such a system is designed , implemented , and applied . In many messages by FreeBSD developers , it is said that it is not possible to establish a testing farm by the FreeBSD project and maintain it , which is quite correct . Assume the following : If any one is downloading and trying to install and run Current development iso files , itis very likely that the main intention is to test and help to its development . By counting number of such testing attempts , it is possible to estimate the number of people which can participate to a more coordinated testing system . My opinion is that such a system ( a distributed testers population ) is already present , and over time , even it may be enlarged . Actually , problem is large enough to establish a working group to attack it . There is a necessity to have committers to generate the necessary ftp sites and manage them . At present , there is a port system . The testing facility will be similar to that system , but different programs will be employed . The first action will be to design what will be tested and how . Since FreeBSD is composed from components ( boot , kernel , each system programs , etc. , ) , every component will be a testing subject . In the svn , there are already testing parts . By combining these , and supplying , developing missing parts will generate a testing framework . At the beginning , it will not be possible to generate a state of the art testing facility . By starting from the existing parts , over time , it will be possible to ad tests one by one . This will be similar to development of ports system : Over time it has been populated one by one . As a an applicable first step , a FreeBSD developer , may establish a wiki page or use an existing one to develop a "Testing Requirements and Design , Implementation and Applications" page . A mailing list may be established to discuss testing problems . An ftp site may be established to apply tests as suggested in my first message . As an example : Assume that a part related to video display cards is under development , such as KMS . The developer(s) will have a limited number of cards available in their computers . A potential people population exists which they use such cards , for example RADEON , or INTEL cards . In the mailing lists , it may be announced that testers are needed for specific video card branch , such as RADEON . People , fills a form to participate in the tests . The developer prepares a file just like a port/package . Sends a message to the subscribers wishing to test this problem . Possible testers , upon receiving that message , may enter a command , such as # test_apply name_of_testing_package just similar to pkg_add command . The test_apply program , downloads the testing package and applies it . At the end , the best action is to generate a structured e-mail , the results may be sent to the testing data base just a like bug tracker system . I emphasize structured messages for their automatic processing possibility . If this is not very feasible , at least a formal message may be requested to evaluate results easily . Otherwise results like "Tests failed" will not be very helpful for the developers . A script in there processes the returned e-mails and generates a report to developer(s) ,. A cycle of such tests continues up to a satisfactory result . Some parts , such as the following , are dependent very much to hardware to be used : - Booting process - Network communications - Video display units - USB attached components - Parts requiring BIOS cooperation , - Parts detecting , managing main board parts ( circuits , ports , etc. ) - Parts detecting , managing attached peripheral devices . All of these may be subject of such distributed testing actions . After testing many of the components and ensuing that they are working , there comes to testing of their combinations , for example : - File systems in local disks, - NFS like systems Then , a whole test : Testing complete install iso files . Here , there will not be a booting or hardware detection failure , but failures related to cooperative working of systems under user applications . These failures may occur between mismatched components during their interaction . Therefore , such a testing facility requires cooperation of many people to function properly . A group of leaders may organize the efforts . Thank you very much .
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOgwaMt9qKJYFu-EB-QetRhgdd8CmJbND2HQLKP_YTN6rnZ9tQ>