From owner-freebsd-current@FreeBSD.ORG Sun Feb 24 22:49:18 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 70C92AE1; Sun, 24 Feb 2013 22:49:18 +0000 (UTC) (envelope-from m.e.sanliturk@gmail.com) Received: from mail-vb0-f51.google.com (mail-vb0-f51.google.com [209.85.212.51]) by mx1.freebsd.org (Postfix) with ESMTP id EE718A7F; Sun, 24 Feb 2013 22:49:17 +0000 (UTC) Received: by mail-vb0-f51.google.com with SMTP id fq11so1414422vbb.10 for ; Sun, 24 Feb 2013 14:49:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=zxbCDBBY6UQSb+7D4H2+zHR1wgU3+5cSaWbVNByIQnw=; b=elBSMxbZ69y0V2dOaIbuoti0uo5H2gyaMGd3avecyR/kDa+ZzzpWa5RO0vXtxlwJVr JVGvkEcXAONfzcSNUxnw83G0bASeBVPtckdR1pWRZQU/Yg+O/jtEl3mM1AeMrCnzMDmv 7EVSXXlFLJL+s4vOETglKRtL0SHlvxWuNk3Tb+AgF+dNXwOzZfyH1qg7dKExDUIPMSpS nlt5cLk9oI86vE07EIpUIg6a7aQI9M/BrUUTT20i3sxpOSdmMZ8Hv4sIK2ENjlf2dz4e QZ0AYNZ3jrTk7kfuwv71TI9V66JG5MJBGOicFMoxgOWgOmlACtUjCNsHP/1wrkHaUvSe CgVQ== MIME-Version: 1.0 X-Received: by 10.220.242.73 with SMTP id lh9mr6132081vcb.49.1361746150847; Sun, 24 Feb 2013 14:49:10 -0800 (PST) Received: by 10.58.170.36 with HTTP; Sun, 24 Feb 2013 14:49:10 -0800 (PST) In-Reply-To: References: <51263782.3020205@freebsd.org> <20130224172524.GA24919@saturn> Date: Sun, 24 Feb 2013 14:49:10 -0800 Message-ID: Subject: Re: FreeBSD Testing Facility From: Mehmet Erol Sanliturk To: Alexander Yerenkow Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-current@freebsd.org, Giorgos Keramidas X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Feb 2013 22:49:18 -0000 On Sun, Feb 24, 2013 at 12:48 PM, Alexander Yerenkow wrote: > How about create -testing mail list, and start searching for any kind of > volunteers? > I could lend a hand in creating auto-testing some aspects of FreeBSD (at > leas successful booting/working net/route etc), > someone else probably could have to say something too. > > I think resources are there, they just need to be gathered :) > > -- > Regards, > Alexander Yerenkow Really , there are ample amount of facilities in FreeBSD already . The steps may be the following : - Prepare a testing requirements document : There are existing testing programs in the svn .List them . . The developers may use tests that are not in svn . Collect information about them . . Collect additional testing point suggestions . . Design a testing facility : This may be just an imitation of the packages/ports tree . For each test , prepare a package . The new package management project contains very important information to imitate . Prepare a test_apply program just like pkg_add . Generate an ftp site parts for .../testing/Fatal/... , .../testing/Dangerous/... , .../testing/Harmless/... Generate databases for the collected result mails , with respect to message groups . For each test , generate an ( xml , or JSON , or just name = value pairs ( like .conf files ) ) results e-mail structure . During tests , this file will be created . When such an e-mail is received into a definite address , with program , introduce it into a database . Generate reports from databases to submit to developers . Generate a mailing list about testing facility . Prepare e-mail templates with respect to testing points ( groups , subjects ) . To the subscriber , present a form to fill information which parts he or she can test , and store these into a data base . Associate each test package with fields of the subscribers testing possibilities . When a testing action is required ( generated by a modification in a tested area ) , query the subscribers database , and send an e-mail to possible testers . Before test_apply program is executed , if it is necessary to perform some actions , prepare an algorithm to define such steps and attach to e-mail , or fetch by test_apply . Implementation : The above steps may be implemented mostly by facilities present in the base system and packages in the ports tree . Identify required new programs , write-ups , scripts , etc. , to design , and implement them . Application , Testing : By using a harmless test , apply the above designed system and evaluate outcome , recycle the above steps , Start to populate the testing system with packages one by one . Maintain the testing tree , and when a modification is applied into a part , if necessary , also modify the testing package and request a testing step . Now , there are "Call for Testing" applications in the present mailing lists . With the above structure , such requests will be more widely applied and will produce structured , analyzable results . If one such package is developed , the rest will start to imitate its many steps , development of new testing packages will be easy . Some of the test packages may also be installed during a complete FreeBSD installation and may be used to test the installed system , and the results may be collected to analyze . I always emphasize structured messages because other messages may not be easily analyzed . Over the studies of this system , there may be some modifications in the base system parts or packages to facilitate a more effective testing . These requirements will emerge over time . Thank you very much . Mehmet Erol Sanliturk