From owner-freebsd-testing@FreeBSD.ORG Wed Feb 26 04:02:40 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D7FBA987 for ; Wed, 26 Feb 2014 04:02:40 +0000 (UTC) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6750F1633 for ; Wed, 26 Feb 2014 04:02:40 +0000 (UTC) Received: by mail-wi0-f180.google.com with SMTP id hm4so1627393wib.1 for ; Tue, 25 Feb 2014 20:02:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=/0NE6hhJvnYKqnWARun13bUGDcdKlZQ6ifGPdJv16/Y=; b=lz1zwHQeNh1xsQ6pe8eX/j2mq3RCqvYFCaTTOU7DVIuPht87yZVe2B8FxxHvesD1mV s643+KKpVhQ3jPui2ZlyDmcFPvYO79aXkbS93yVos9V+f4N0ZxJIru57abKcC9hv/XH3 EoDkaS9iDhAnljCNyycmEdD3KcNAXLalYQAdOInj1PkHiUSloRkHONHGq0cSuYt3vrbC 8Rl+HE09Xg4O+7s71zrR/R7SJRZ8SbBAL5Wb+MdguRIlXpeBF6kIzutOt77vdCq9ym5g DAf+JlZ92L3dwKzTUpklwIpV+6k0Fv0P1C6OvKq4WmeEjz1VSskNpwu5/cbRUQPlr2w2 pY3g== MIME-Version: 1.0 X-Received: by 10.180.187.237 with SMTP id fv13mr5962363wic.26.1393387358448; Tue, 25 Feb 2014 20:02:38 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.168.197 with HTTP; Tue, 25 Feb 2014 20:02:38 -0800 (PST) In-Reply-To: <20140225183611.GA70774@x2.osted.lan> References: <20140225161129.GA59741@x2.osted.lan> <20140225183611.GA70774@x2.osted.lan> Date: Tue, 25 Feb 2014 21:02:38 -0700 X-Google-Sender-Auth: BrVLojkdbPaL90DNqcwTPdf5QCs Message-ID: Subject: Re: My first ATF test From: Alan Somers To: Peter Holm Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Feb 2014 04:02:41 -0000 On Tue, Feb 25, 2014 at 11:36 AM, Peter Holm wrote: > On Tue, Feb 25, 2014 at 12:12:38PM -0500, Mehmet Erol Sanliturk wrote: >> On Tue, Feb 25, 2014 at 11:47 AM, Alan Somers <[1]asomers@freebsd.org= > >> wrote: >> >> On Tue, Feb 25, 2014 at 9:11 AM, Peter Holm wrote: >> > In order to understand how ATF works I wrote a small test so I ha= d >> > something to work with: >> > [2]http://people.freebsd.org/~pho/kern_descrip_test.diff >> > Did I get it right? >> ATF-wise, it looks good. =C2 However, it's a bad idea to use random >> numbers in test code, except in stress tests. =C2 Random numbers >> result >> in irreproducible tests. =C2 How about replacing the body of >> dup2_r234131 >> with something like this? >> =C2 int fd1, fd2, ret; >> =C2 fd1 =3D =C2 open("/etc/passwd", O_RDONLY); >> =C2 fd2 =3D INT_MAX; >> =C2 ret =3D dup2(fd1, fd2); >> =C2 ATF_CHECK_EQ(-1, ret); >> =C2 ATF_CHECK_EQ(EBADF, errno); >> On a side note, perhaps WARNS should be set in [3]atf.test.mk, so w= e >> won't have to set it in every other Makefile. >> -Alan >> _______________________________________________ >> >> When random numbers are used , it is possible to make the runs >> reproducible in the following way : >> Generate a specified number of random numbers and store them into a >> file . >> During usage , for random numbers , traverse that file . >> This may be repeated any number of times for different other paramete= rs >> . >> All of the runs will use the same random numbers . >> Then the results ( which they are generated from the same distributio= n >> ) may be compared with suitable statistical tests . >> Thank you very much . >> Mehmet Erol Sanliturk >> > > I guess I'm too used to using random values in tests :) > Using random(3) with an initial seed of "1" could have been an > alternative, but ... But in this case it's not necessary. This isn't a stress test; it's a basic functional test. If Peter had been attempting to trigger a race condition or something like that, then random(3) with a fixed seed would indeed be appropriate. -Alan