From owner-freebsd-testing@FreeBSD.ORG Thu Jan 23 00:06:56 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 757ACFA4 for ; Thu, 23 Jan 2014 00:06:56 +0000 (UTC) Received: from mail-qc0-f170.google.com (mail-qc0-f170.google.com [209.85.216.170]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 2C84712DD for ; Thu, 23 Jan 2014 00:06:56 +0000 (UTC) Received: by mail-qc0-f170.google.com with SMTP id e9so1603419qcy.1 for ; Wed, 22 Jan 2014 16:06:55 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=WI+RPJIGYVX/cG0jKkxJPsakucvRbFa37K8Je0zR/Rw=; b=GwxX5b/XxZEipOoK6o41kwvrmcfx7gDoLbrxyJwYLiB/apmB2LUnSSrK36WGtgYIED cEncNlT94uufeeH8r6GAjeahZe5b51gLSUgng/fLuBTGyARVv8eZVm90ffXs00zWk+LS 8alS2WinPCNwEd0UCkKK7e9ztrXI455WwlKJ62RtHlS0VHWsvCbQQP3dGlehYJSXUI47 srJrRtVIFFC6PbMHn2F1Nda0+G9WJflADBoyg671T8oYOUeBd4kiQhzRVha3nlBixuEI DSXHV68tf3cKx42iCjfd2f2HcWCQ6dUrYy0GuPF891N0SqGI1PQcSYoq/IIRePl1m25C yuGA== X-Gm-Message-State: ALoCoQkPgMk0DJCYH8AlVA0K4eY5bKbduv3vCWHhqOS+VYg/bkil2FTbq4GttiS06EFP+ATwAcjM X-Received: by 10.229.71.69 with SMTP id g5mr7182291qcj.6.1390435615073; Wed, 22 Jan 2014 16:06:55 -0800 (PST) Received: from barcelona.nyc.corp.google.com (barcelona.nyc.corp.google.com [172.26.105.74]) by mx.google.com with ESMTPSA id b18sm16780686qaw.0.2014.01.22.16.06.51 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 22 Jan 2014 16:06:52 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.1 \(1827\)) Subject: Re: PATCH: add ATF tests in sys From: Julio Merino In-Reply-To: <7051E061-C054-47BF-A5B2-747B9D3E85A1@gmail.com> Date: Wed, 22 Jan 2014 19:06:49 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <7A53F7F4-349F-480D-802E-F61C9308D009@meroh.net> References: <7051E061-C054-47BF-A5B2-747B9D3E85A1@gmail.com> To: Garrett Cooper X-Mailer: Apple Mail (2.1827) Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Jan 2014 00:06:56 -0000 On Jan 21, 2014, at 19:23, Garrett Cooper wrote: >>=20 >> On Jan 21, 2014, at 16:02, Alan Somers wrote: >>=20 >>> On Tue, Jan 21, 2014 at 4:56 PM, Garrett Cooper = wrote: >>>=20 >>>> On Jan 21, 2014, at 14:45, Alan Somers wrote: >>>>=20 >>>> I rewrote the unix seqpacket tests in ATF. The hard part is adding >>>> them to the build. The linked patch adds them in the directory >>>> sys/kern/tests and builds them as part of buildworld, not = buildkernel. >>>> They get installed to /usr/tests/sys/kern, but the intermediate >>>> objects are stored in /usr/obj/sys/kern/tests. That means that you >>>> can't have different tests associated with different kernel = configs. >>>> I think that this is desirable, because you wouldn't be able to >>>> install tests for different kernel configs anyway, given our chosen >>>> layout of /usr/tests. >>>>=20 >>>> Please comment on the parts of this patch that deal with Makefiles. >>>> Is this the appropriate way to add sys tests to the build? = Shouldn't >>>> I be building them in buildkernel instead of buildworld? I = couldn't >>>> find a good way to do that. >>>=20 >>> I bypassed that for simplicity and placed the tests in = tests/sys/kern/... etc. Shoehorning things into our overly complex = kernel build system just seems like a really bad idea... Just make it = load as a driver, loadable, and dependent on kern.features to run..? >>> Cheers! >>> -Garrett >>=20 >> I tried to place the tests as closely as possible to the source code >> that they're testing, according to >> = https://wiki.freebsd.org/TestSuite/Structure#Makefiles_for_test_programs >> . I don't feel strongly about it either way, but I think Simon and >> jmmv do. Yes in general, but... see below. >>=20 >> FWIW, the tests run as a userland program, not in kernel mode. >=20 > Fwiw I spent months mulling over various details (this was just one = point) with gnn, marcel, and mfleming. The consensus was that that was a = really bad idea because it wasn't terribly clear that tests covered a = particular area, and some sys/ tests could cover multiple areas = (pjdfstest for instance). >=20 > I mirrored the exact structure in the source tree, but avoided = tainting things in sys/... Because integrating sys/ into buildworld and = starting to mix kernel and world build logic together will irritate a = large number of kernel devs and userland devs. This was the path of = least resistance... I think Garrett has a good point here and, if the build within src/sys/ = is significantly different from the build of userspace tools and has = different restrictions, it is probably a good idea to treat the kernel = tests as "special". After all, these tests are userland programs as you well said and they = are a kind of "integration" tests because there is no way currently to = unit-test kernel code. Such programs will just do "stuff" hoping to = validate kernel code, but there are so many levels of indirection that = they cannot be though of as testing things in very fine-grain detail. (If we had rump-like features as NetBSD, things may be a bit different = though because in that case you really are writing unit tests for very = specific kernel code.) Using src/tests/sys/ and replicating the layout of src/sys/ as much as = possible within it may be a reasonable and easy compromise for now.=