Date: Sun, 24 Feb 2013 14:49:10 -0800 From: Mehmet Erol Sanliturk <m.e.sanliturk@gmail.com> To: Alexander Yerenkow <yerenkow@gmail.com> Cc: freebsd-current@freebsd.org, Giorgos Keramidas <keramida@freebsd.org> Subject: Re: FreeBSD Testing Facility Message-ID: <CAOgwaMvAUhs09%2But3z=fd_O4_mjYfiQVpE%2BcaCh1M5c82KKGVQ@mail.gmail.com> In-Reply-To: <CAPJF9wm1CDBVU4yNuJGuCO5hNxNKwnZgm-Zk%2BUafWcM1HGv-GQ@mail.gmail.com> References: <CAOgwaMsbupU0NB7TA6RMJW0erWBGwcsGO16xt3GPLsPuh-rpRA@mail.gmail.com> <51263782.3020205@freebsd.org> <20130224172524.GA24919@saturn> <CAOgwaMt9qKJYFu-EB-QetRhgdd8CmJbND2HQLKP_YTN6rnZ9tQ@mail.gmail.com> <CAPJF9wm1CDBVU4yNuJGuCO5hNxNKwnZgm-Zk%2BUafWcM1HGv-GQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Feb 24, 2013 at 12:48 PM, Alexander Yerenkow <yerenkow@gmail.com>wrote:
> How about create -testing mail list, and start searching for any kind of
> volunteers?
> I could lend a hand in creating auto-testing some aspects of FreeBSD (at
> leas successful booting/working net/route etc),
> someone else probably could have to say something too.
>
> I think resources are there, they just need to be gathered :)
>
> --
> Regards,
> Alexander Yerenkow
Really , there are ample amount of facilities in FreeBSD already .
The steps may be the following :
- Prepare a testing requirements document :
There are existing testing programs in the svn .List them .
. The developers may use tests that are not in svn . Collect information
about them .
. Collect additional testing point suggestions .
. Design a testing facility :
This may be just an imitation of the packages/ports tree .
For each test , prepare a package .
The new package management project contains very important information to
imitate .
Prepare a test_apply program just like pkg_add .
Generate an ftp site parts for .../testing/Fatal/... ,
.../testing/Dangerous/... , .../testing/Harmless/...
Generate databases for the collected result mails , with respect to
message groups .
For each test , generate an ( xml , or JSON , or just name = value pairs
( like .conf files ) ) results
e-mail structure .
During tests , this file will be created .
When such an e-mail is received into a definite address , with program ,
introduce it into a database .
Generate reports from databases to submit to developers .
Generate a mailing list about testing facility .
Prepare e-mail templates with respect to testing points ( groups ,
subjects ) .
To the subscriber , present a form to fill information which parts he or
she can test , and store these into a data base .
Associate each test package with fields of the subscribers testing
possibilities .
When a testing action is required ( generated by a modification in a
tested area ) , query the subscribers database , and send an e-mail to
possible testers .
Before test_apply program is executed , if it is necessary to perform some
actions , prepare an algorithm to define such steps and attach to e-mail ,
or fetch by test_apply .
Implementation :
The above steps may be implemented mostly by facilities present in the base
system and packages in the ports tree .
Identify required new programs , write-ups , scripts , etc. , to design ,
and implement them .
Application , Testing :
By using a harmless test , apply the above designed system and evaluate
outcome ,
recycle the above steps ,
Start to populate the testing system with packages one by one .
Maintain the testing tree , and when a modification is applied into a part
, if necessary , also
modify the testing package and request a testing step .
Now , there are "Call for Testing" applications in the present mailing
lists .
With the above structure , such requests will be more widely applied and
will produce structured ,
analyzable results .
If one such package is developed , the rest will start to imitate its many
steps , development of new testing packages will be easy .
Some of the test packages may also be installed during a complete FreeBSD
installation and may be used to test the installed system , and the results
may be collected to analyze .
I always emphasize structured messages because other messages may not be
easily analyzed .
Over the studies of this system , there may be some modifications in the
base system parts or packages to facilitate a more effective testing .
These requirements will emerge over time .
Thank you very much .
Mehmet Erol Sanliturk
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOgwaMvAUhs09%2But3z=fd_O4_mjYfiQVpE%2BcaCh1M5c82KKGVQ>