From owner-freebsd-arch@FreeBSD.ORG Wed Nov 14 09:28:11 2012 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id CB22FE03; Wed, 14 Nov 2012 09:28:11 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 5932F8FC15; Wed, 14 Nov 2012 09:28:11 +0000 (UTC) Received: by mail-ob0-f182.google.com with SMTP id 16so243983obc.13 for ; Wed, 14 Nov 2012 01:28:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=cBf4a/ykEX9zxKVI3Hkf6tDXQ9jWfOajAZT2AAIArqo=; b=rECjtoteGzTdLb2ip3G1YGPXSHniP44sQffreHPDMCcvwzdDXaHv6pj9zRbEGyzS+k T4Kj+CX0jg/9U0bJv1WiTnFPpN6jmXfwhTU7cIpr7ztY2vHksxq9w5JUrvGZGObLxCJ4 VMGnF6U6dPHiBN/sOa0rAi0uAzjqvUO5XIVCw/ySJnUqD3IoO78hc/vWSQ/YwHKtM/B6 Jf7GTdj5d8S0gfQBdp1xYm8M32KALHuIr37fjU0ed5SRp3HEEvLox0+TpVuMtEuNwRdT X0Cd7OVhIQVqOYLwiWLO5dpprYhUFEllZASPCMLhd/jndo9HE1I+o3aDXf+14WArB0hH 4LIQ== MIME-Version: 1.0 Received: by 10.182.131.100 with SMTP id ol4mr20286539obb.38.1352885284895; Wed, 14 Nov 2012 01:28:04 -0800 (PST) Received: by 10.76.143.33 with HTTP; Wed, 14 Nov 2012 01:28:04 -0800 (PST) Date: Wed, 14 Nov 2012 01:28:04 -0800 Message-ID: Subject: [RFC] test layout/standardization for FreeBSD From: Garrett Cooper To: "freebsd-arch@FreeBSD.org Arch" Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: George Neville-Neil , Marcel Moolenaar , Matthew Fleming X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Nov 2012 09:28:12 -0000 As David O'Brien noted at the BSD Vendor Support, some of the requirements for standardizing the test process and layout in FreeBSD aren't clear today. I asked for feedback from some of the stakeholders of the FreeBSD test automation effort (gnn, marcel, mdf, sjg) I've been working with, and here is a summary of the agreed upon method for organizing tests in FreeBSD. Please provide constructive criticism, feel free to ask questions, etc if any of the points below could be clarified/improved upon; items 1. and 3. are straightforward, but item 2. is still up to some debate, in particular item 2.i and 2.iii. The agreed upon items will be added to the wiki (http://wiki.freebsd.org/TestingFreeBSD ). Thanks! -Garrett 1. Userland tests: - Tests will go into a tests/ subdirectory, e.g. for source directory lib/libc/, there will be a tests directory lib/libc/tests/; for bin/date, there will be a tests directory bin/date/tests/. 2. Kernel tests: i. The user will be able to invoke buildkernel -DWITH_TESTS build all applicable testcases and install them into ${TESTSBASE}/sys/... . ii. Generic kernel tests will go into tests/sys/ and will be included in the standard build process. Example: sys/tests/capsicum/... sys/tests/kern/acct/... sys/tests/kern/read/... sys/tests/kern/write/... iii. Driver specific testcases should live close to the source, e.g. mps(4) testcases should live in sys/dev/mps/tests/, and mps(4) unittest programs/modules should be compiled/linked as part of the standard module build process. Moreover the build logic should live under sys/modules/mps/tests/ [*]. The install path is vendor/developer specific, but should be ${TESTSBASE}/sys/${DEV}/ in order to match FreeBSD's layout. This will require another minor piece of build infrastructure (kern.test.mk) in order to ensure proper versioning, and I will work on drafting up that next item if it's agreed that this is the correct direction for standardizing driver testcases. 3. Jumping into tests/ subdirectories should be handled transparently via bsd.subdir.mk like so: diff --git a/share/mk/bsd.subdir.mk b/share/mk/bsd.subdir.mk index 38a9080..a0e9984 100644 --- a/share/mk/bsd.subdir.mk +++ b/share/mk/bsd.subdir.mk @@ -43,6 +43,11 @@ distribute: .endfor .endif +.if ${MK_TESTS} != "no" && exists(${.CURDIR}/tests/Makefile) +SUBDIR+= tests +SUBDIR_TARGETS+= test +.endif + _SUBDIR: .USE .if defined(SUBDIR) && !empty(SUBDIR) && !defined(NO_SUBDIR) @${_+_}set -e; for entry in ${SUBDIR}; do \ This will require adding an additional knob (MK_TESTS) to bsd.own.mk. [*] How we run and handle driver-specific kernel testcases is outside the scope of this note as we do not have the full NetBSD testing infrastructure (RUMP, et all) to commit to a full-scale plan [yet]. This needs to be worked out sometime in the next 1~2 years and/or should done as a GSoC project.