From owner-freebsd-testing@FreeBSD.ORG Mon Jul 21 19:06:21 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 47A28275 for ; Mon, 21 Jul 2014 19:06:21 +0000 (UTC) Received: from oms-md01.mx.aol.com (oms-md01.mx.aol.com [64.12.81.145]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0859C2885 for ; Mon, 21 Jul 2014 19:06:20 +0000 (UTC) Received: from mtaout-aae02.mx.aol.com (mtaout-aae02.mx.aol.com [172.27.1.98]) by oms-md01.mx.aol.com (AOL Outbound OMS Interface) with ESMTP id D0FD43800121A for ; Mon, 21 Jul 2014 15:06:18 -0400 (EDT) Received: from TOSHIBASATEL-PC (unknown [41.138.185.28]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mtaout-aae02.mx.aol.com (MUA/Third Party Client Interface) with ESMTPSA id A3B6A38000086 for ; Mon, 21 Jul 2014 15:06:17 -0400 (EDT) From: "pamela hill" Subject: Pamela To: freebsd-testing@freebsd.org MIME-Version: 1.0 Reply-To: pamelahilltogo22@yahoo.co.jp Date: Mon, 21 Jul 2014 20:05:52 +0100 Message-ID: <7376783749194@smtp.aol.com> x-aol-global-disposition: S X-SPAM-FLAG: YES DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=mx.aol.com; s=20140625; t=1405969578; bh=Wtp4k8flib0Q/1U27GIDvQaf3TQj7IpjMq/KK+zuFo8=; h=From:To:Subject:Message-ID:Date:MIME-Version:Content-Type; b=hu/mcNuyhSb6CWbJnpQg6rWCFMrJUOrr6FnEtXChZsAtTPk9M1NZNEEu9DEPQEste 8uZe1GlgKhN7782AMHlV0MXUGHYkHVowuhL44p9H29kU46i827G0e56tV87JtpQtBb eYlMl1CR5x2NjR6SvDugi19mos7nNnVuCPCn91ro= X-AOL-REROUTE: YES x-aol-sid: 3039ac1b016253cd64a93b65 X-AOL-IP: 41.138.185.28 Content-Type: text/plain ; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.18 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jul 2014 19:06:21 -0000 Hello!!!!!!!!, I saw your profile at wed Search Results and i really love to know you= if you wish, am looking a long term relationship if it should go in t= hat direction, like most times i do ask my self this :- Obviously you = are going to chose a partner that you are physically attracted to. Apa= rt from that aspect, there are a few important areas you should look a= t. Do they speak very slowly compared to you? Do they understand you w= hen you speak? Can you really talk to them about absolutely anything?.= When you're ready to make that full commitment, you're sometimes left= with questions and a nagging sense of self-doubt. Is this the one? Am= I making the right choice? Does my potential partner feel the same wa= y? In this guide i hope u attempt to answer these questions and more. = The best way to make sure a relationship succeeds is to make sure it's= the right one from the start!. Hope am not too hard? well i hope to h= ear from you again bye for now. I here by paste one of my pix, Am 28 y= ears old and also from the States (USA), female that is all i have to = give now till u reply i will give more. You can send me email so we ge= t to know each other. Pamela. From owner-freebsd-testing@FreeBSD.ORG Tue Jul 22 23:07:57 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5B8D58E0 for ; Tue, 22 Jul 2014 23:07:57 +0000 (UTC) Received: from mail-ie0-x22d.google.com (mail-ie0-x22d.google.com [IPv6:2607:f8b0:4001:c03::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2F79E2209 for ; Tue, 22 Jul 2014 23:07:57 +0000 (UTC) Received: by mail-ie0-f173.google.com with SMTP id tr6so318048ieb.32 for ; Tue, 22 Jul 2014 16:07:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=TojNHiDJu0btrsuBeBNaliWtqZjKMxTisccSY64YUO8=; b=YaClK8v5XVfFgwm2/Pq20Ic/Yyr8xwTp4/xi5yDk0hrb/QjpH1LzoVIcl5QZxNavPN xtPozeilP/rB82ihfznI1oRk8xd67LOnV6YbAiLiDAlICuT69fITRKgkuq0vkgTkhhmy dBT9eJ0L3SjoA5cXJQD4UQyobT7VnF5UlcoVf82OWtKc950Zm7x1twaB/wjRBJl9RA2v DdTimZ9BxhMChEkvROeAQ0dzB1jnSlvyeUmf8+V+BhsiLKNknyxFCImIqDqtc/6hMzoI 47S70/vzomMpK+AcxnclPdLPaJ56sZQLPmFhKrnlGKPB+Q12OBSEPDFT6rIzlHmaZ+81 X+2w== MIME-Version: 1.0 X-Received: by 10.50.66.133 with SMTP id f5mr21281711igt.38.1406070476695; Tue, 22 Jul 2014 16:07:56 -0700 (PDT) Received: by 10.50.213.102 with HTTP; Tue, 22 Jul 2014 16:07:56 -0700 (PDT) Date: Tue, 22 Jul 2014 16:07:56 -0700 Message-ID: Subject: Location of test kernel drivers in tree? From: Garrett Cooper To: "freebsd-testing@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jul 2014 23:07:57 -0000 Hi all, I need to do some KPI verification and hook that into ATF / kyua. Since we don't have RUMP [yet] from NetBSD, I would typically write a simple, one-off test driver, hook it into the kernel build and have a piece of C or shell code that pokes directly at the driver to get access to kernel interfaces. I was wondering if it made sense to put all test drivers into sys/tests//..., e.g. sys/tests/test_memguard/{Makefile,test_memguard.c}, etc. Is there an alternative approach that others use to solve this problem? Thanks! -Garrett From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 01:12:11 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8475C257 for ; Thu, 24 Jul 2014 01:12:11 +0000 (UTC) Received: from mail-qa0-f54.google.com (mail-qa0-f54.google.com [209.85.216.54]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 44CA82FC3 for ; Thu, 24 Jul 2014 01:12:10 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id k15so2186512qaq.41 for ; Wed, 23 Jul 2014 18:12:09 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=8SoxhiMzUQGm8VQmylbBs9BCHInA3X4wa9leUS0mM/U=; b=hR2GYqH1nBbin45nZOy2WUWOBV4+VAHEpVJl4ZFOrSXmoCtb5hBOVPEjm1hzk9xWdM 1myfubYSoNBbtJ5c9PW2c1eHz/eMXPPEUo6oDy/uydNfxqFDIbgjV9dVtxt29FFW9gRf ZGuQ5pA8abLbOjfI6Zj50+ilEqewYtMFlb3nhKbcXpvsl/7OrKEwqAVOBii05MuuNBBL +fOWpg04HYGaDhq7QWXAS2TTsdTzzhoCEyMDGataH26ASrNHWx7k5novkbeaTT2/gHFP HNqw8MbeRLnf6EbZJeGNGc6kgW60Ux+7OAYduJ5k1rgRJFThwBRaQcVYtCcfC+2X9Br5 5Fdg== X-Gm-Message-State: ALoCoQngkKhSI7a2c1QdXc/pHboJ3l6Rtw5onNOJfmCHlzk6lPT2bRaqvUtu/xkorUfeh1gtjFw8 X-Received: by 10.224.42.196 with SMTP id t4mr8655015qae.48.1406164329421; Wed, 23 Jul 2014 18:12:09 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Wed, 23 Jul 2014 18:11:49 -0700 (PDT) X-Originating-IP: [184.153.117.174] In-Reply-To: References: From: Julio Merino Date: Wed, 23 Jul 2014 21:11:49 -0400 X-Google-Sender-Auth: 0hlQm2uROO6MOGSP7TppUkxY5s4 Message-ID: Subject: Re: Location of test kernel drivers in tree? To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 01:12:11 -0000 On Tue, Jul 22, 2014 at 7:07 PM, Garrett Cooper wrote: > Hi all, > I need to do some KPI verification and hook that into ATF / kyua. > Since we don't have RUMP [yet] from NetBSD, I would typically write a > simple, one-off test driver, hook it into the kernel build and have a > piece of C or shell code that pokes directly at the driver to get > access to kernel interfaces. > I was wondering if it made sense to put all test drivers into > sys/tests//..., e.g. > sys/tests/test_memguard/{Makefile,test_memguard.c}, etc. Is there an > alternative approach that others use to solve this problem? I don't have an answer to alternative solutions, but keeping the helper modules inside tests/sys/ (I think you got that backwards in your email?) is a good idea. (Instead of tests/sys//, I'd probably do tests/sys///... though, to keep the helpers next to the only tests that need them. Unless the helper is really generic and usable by many tests.) From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 01:20:41 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2636E2EC; Thu, 24 Jul 2014 01:20:41 +0000 (UTC) Received: from mail-ie0-x22d.google.com (mail-ie0-x22d.google.com [IPv6:2607:f8b0:4001:c03::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E164B2FF7; Thu, 24 Jul 2014 01:20:40 +0000 (UTC) Received: by mail-ie0-f173.google.com with SMTP id tr6so1695225ieb.4 for ; Wed, 23 Jul 2014 18:20:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=nVyFrYgGfl/WFded2jCQRwDF8mLl7vM/MFqmyYmnlis=; b=v6SHKA1Hl/R/ZXoPlcffoKOEtN1IzxpDsOJD6mgUYbm4Po7YrASPmeiRU3xOgaedSM zvGO6yUIso6yQdiah/gUckGUMm3FxwELYH1Ys0uEwj7ofC0FJjSly+Ad1U+dZ9384exK ukSEWuiBxiKcm8GLbHtxn3Qbw7YNH6SuRELox5HZhoo0tzNpU73Joa7afmd1yN5XKbBU ItjYBVAYpfHV8ID/kdol8m4EhQ1MwL77ctnhwdAweJp9tTnGb2lF28YdCRl+Fr3yx0Qp fJEDQCOkrW9nD1qPKKkDSlWwd5yWvLClD5/NltqVWAwdzsMGiyrhN06RUpHd5952VkN4 C6Fw== MIME-Version: 1.0 X-Received: by 10.50.138.72 with SMTP id qo8mr9563764igb.2.1406164840300; Wed, 23 Jul 2014 18:20:40 -0700 (PDT) Received: by 10.50.213.102 with HTTP; Wed, 23 Jul 2014 18:20:40 -0700 (PDT) In-Reply-To: References: Date: Wed, 23 Jul 2014 18:20:40 -0700 Message-ID: Subject: Re: Location of test kernel drivers in tree? From: Garrett Cooper To: Julio Merino Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 01:20:41 -0000 On Wed, Jul 23, 2014 at 6:11 PM, Julio Merino wrote: > On Tue, Jul 22, 2014 at 7:07 PM, Garrett Cooper wrote: >> Hi all, >> I need to do some KPI verification and hook that into ATF / kyua. >> Since we don't have RUMP [yet] from NetBSD, I would typically write a >> simple, one-off test driver, hook it into the kernel build and have a >> piece of C or shell code that pokes directly at the driver to get >> access to kernel interfaces. >> I was wondering if it made sense to put all test drivers into >> sys/tests//..., e.g. >> sys/tests/test_memguard/{Makefile,test_memguard.c}, etc. Is there an >> alternative approach that others use to solve this problem? > > I don't have an answer to alternative solutions, but keeping the > helper modules inside tests/sys/ (I think you got that backwards in > your email?) is a good idea. I was thinking of sys/tests (with associated build machinery under sys/modules/tests) because the kernel and world [*] build vary wildly by design. > (Instead of tests/sys//, I'd probably do > tests/sys///... though, to keep the > helpers next to the only tests that need them. Unless the helper is > really generic and usable by many tests.) Most definitely! Thanks :)!! -Garrett * There's MODULES_WITH_WORLD [see make.conf(5)], which is off by default, so technically modules can be built with world, but I'm not sure how well it works when you have to have to use opt_* headers. From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 01:29:33 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 68354386 for ; Thu, 24 Jul 2014 01:29:33 +0000 (UTC) Received: from mail-qg0-f50.google.com (mail-qg0-f50.google.com [209.85.192.50]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 263FC20C0 for ; Thu, 24 Jul 2014 01:29:32 +0000 (UTC) Received: by mail-qg0-f50.google.com with SMTP id q108so2482504qgd.9 for ; Wed, 23 Jul 2014 18:29:26 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=utS5nJXRr8pyXqFPEEHxyyalS84w/jvKdKz8oaddowY=; b=l2244ccit34zs1N09rqtv1e6BfsdN3pxMIZ0sE67siBs7PptIxyZ+DPc2m/U5tw+k8 RNQs8VfoWueLE0i4quqTHj17ZFaVzSHhK2i4LteB7Jej3UAPsMgIgo6F5tIxENsTR+i2 NJGGoLOOmrj+9BHpbvxjXr00l+L8xEIrqidd5y5wKDBRpKCHvx+Ft+n/XdPWo5bUUbfk hdIswBJgxT8gQM6yioMKOA3bTJhDJAq7ywmUNkVpMQS6kY3Ngt0UxY1QSpbY5Xs4/6gi 6qttG1kDPlAHujcaj5JDvlrDmki+q23VVGGJ0j2dxCQ8D9NMkc3J4qymSNxjfreoMcM+ VOgg== X-Gm-Message-State: ALoCoQnVW+mvwnYDIvE6Mz7M8FiKqsh3bVu7KeJNO8ufNgiJPqIMhDdN09P9cFFC9kGvWj3xgbSy X-Received: by 10.224.120.138 with SMTP id d10mr8853880qar.9.1406165366734; Wed, 23 Jul 2014 18:29:26 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Wed, 23 Jul 2014 18:29:06 -0700 (PDT) X-Originating-IP: [184.153.117.174] In-Reply-To: References: From: Julio Merino Date: Wed, 23 Jul 2014 21:29:06 -0400 X-Google-Sender-Auth: 0VCQIiY8Da0jSaBwpgzih2VlZ44 Message-ID: Subject: Re: Location of test kernel drivers in tree? To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 01:29:33 -0000 On Wed, Jul 23, 2014 at 9:20 PM, Garrett Cooper wrote: > > I was thinking of sys/tests (with associated build machinery under > sys/modules/tests) because the kernel and world [*] build vary wildly > by design. But didn't we decide a while ago that we'd put kernel tests in tests/sys/ precisely to avoid the build system "oddities" of the kernel? That's where the few kernel-level tests that we have today live... Or am I misunderstanding? Because if you originally said that you'd put _only_ the test-related kernel modules in sys/..., then I guess that makes sense! From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 01:45:13 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B3227721; Thu, 24 Jul 2014 01:45:13 +0000 (UTC) Received: from mail-ie0-x234.google.com (mail-ie0-x234.google.com [IPv6:2607:f8b0:4001:c03::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 78D5D2217; Thu, 24 Jul 2014 01:45:13 +0000 (UTC) Received: by mail-ie0-f180.google.com with SMTP id at20so1750150iec.11 for ; Wed, 23 Jul 2014 18:45:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=CwlfpXi+nWumhQzpfMBszkJXdlIhUwnrXhFfeiyh1MA=; b=MXSQffWDFWfxRTYDzpgqDLB2tZ40qOu9rXdFrnZhxZCg5VVwkqxNLqUflrVEiO276A bGw47gV4Yp4PHkLQFNUJdbTBUQ6Rkl10Me8up92qxlDSL5FN3rX3UtT3hU2nrj5a45DG Zkci+zK/5HQ/7zI5KFq3LAkmASEwaEq4B3II87WIXCBHalLWuxR8H841ZXsNl4+z5+Y4 8BUQsqWhlJOQqTNlYl7fvO9iMn+JKu2ELpVNBX7IiRItLay74Wtqhc98AFund56zZAlV HVfiV6E+vOddF6xm3Hw1uYAt/vKPN1a2Pygg4+VEOiaPWhuqRUabIai5OH7SdAiWo+qp tANw== MIME-Version: 1.0 X-Received: by 10.50.152.98 with SMTP id ux2mr34059029igb.27.1406166312688; Wed, 23 Jul 2014 18:45:12 -0700 (PDT) Received: by 10.50.213.102 with HTTP; Wed, 23 Jul 2014 18:45:12 -0700 (PDT) In-Reply-To: References: Date: Wed, 23 Jul 2014 18:45:12 -0700 Message-ID: Subject: Re: Location of test kernel drivers in tree? From: Garrett Cooper To: Julio Merino Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 01:45:13 -0000 On Wed, Jul 23, 2014 at 6:29 PM, Julio Merino wrote: > On Wed, Jul 23, 2014 at 9:20 PM, Garrett Cooper wrote: >> >> I was thinking of sys/tests (with associated build machinery under >> sys/modules/tests) because the kernel and world [*] build vary wildly >> by design. > > But didn't we decide a while ago that we'd put kernel tests in > tests/sys/ precisely to avoid the build system "oddities" of the > kernel? That's where the few kernel-level tests that we have today > live... Correct (and I have more in isilon-atf and there are more at Isilon that need to be upstreamed :)..). > Or am I misunderstanding? Because if you originally said that you'd > put _only_ the test-related kernel modules in sys/..., then I guess > that makes sense! Yup! That's what I was thinking of doing. I was kind of envisioning the following: sys/modules/tests/memguard/Makefile - This will be the makefile that handles compiling the test kernel module(s) (it might need to be nested a bit if there are multiple modules because I don't remember if bsd.kmod.mk can build multiple modules from one Makefile) sys/tests/memguard/ - this will be where the source for the memguard test module(s) lives tests/sys/memguard/ - this will be where the memguard test scripts live This layout does the following: 1. Makes it clear where the test code lives (helps make things more intuitive). 2. Follows the existing model with kernel code (to some degree). 3. Makes it easy to disable/skip the tests with MK_TESTS (ease of maintenance/use). The only catch is that we really need to "grow" some additional infrastructure around ATF/kyua on FreeBSD to detect when modules are present/loaded/etc to set that as requirements for tests to avoid sprinkling bits everywhere in test code to skip tests if the helper programs or modules aren't present. I already started that to some degree (I mentioned that in another freebsd-testing thread), but it needs more work (mostly because what I did works for simple cases but there are some things I potentially missed dealing with how drivers are named, etc): https://github.com/yaneurabeya/freebsd_atf_kyua_test_extensions . Alan's helped out with an initial pass at reviewing the code. Thoughts? Thanks! -Garrett From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 14:07:52 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id CADCA161 for ; Thu, 24 Jul 2014 14:07:52 +0000 (UTC) Received: from mail-qa0-f42.google.com (mail-qa0-f42.google.com [209.85.216.42]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 880AD2390 for ; Thu, 24 Jul 2014 14:07:52 +0000 (UTC) Received: by mail-qa0-f42.google.com with SMTP id j15so3011285qaq.29 for ; Thu, 24 Jul 2014 07:07:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=tgf3WFq+30sWzmEa47WRd4Fyn1+NhAZzbhCf5bF7YmE=; b=VgqX0JwfHc/z7O5PCYKKi4Q3tVnnFedcz6QWS3bH9q5bQmMU+Aumogus/XpSg1nUy5 WA6gjPccHLdLXdP1EGPhJ3iFWX5oE164CtuDqNJp/pyODjHzkPMwBEKg+RukCPxq6u9W Skp2rE/Pa8HLrVyTvrBRxZO5nj+BuU5Ecura4ObjygwnYzsGyvj1wPN3cukCfsVxKZaQ ePgSa1xMRZGDXrtPwfhatIxNUAP69pjvwB03ouwXwSvfDO2RQd+jtylY2Jhjhisiek8Y kfOv7QC+CzE/P5GjAJDH8NJVSJWjezm4bOTqZ3xd9SApbeohpBYdkGbYYUCDGWEg0mxM ycQg== X-Gm-Message-State: ALoCoQnW37BdE8akG0Xmr8UY2Mfl0cl3ukZ1nTT8ccubtSlI07w7LbErdml6IalCzrqrjKoG2D70 X-Received: by 10.140.49.77 with SMTP id p71mr14733942qga.86.1406210558444; Thu, 24 Jul 2014 07:02:38 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Thu, 24 Jul 2014 07:02:16 -0700 (PDT) X-Originating-IP: [2620:0:1003:1007:ac3e:4bce:435e:6e80] In-Reply-To: References: From: Julio Merino Date: Thu, 24 Jul 2014 10:02:16 -0400 X-Google-Sender-Auth: lu1DYaLHPo9YbtFEO8LtOJUtz2k Message-ID: Subject: Re: Location of test kernel drivers in tree? To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 14:07:52 -0000 On Wed, Jul 23, 2014 at 9:45 PM, Garrett Cooper wrote: > Yup! That's what I was thinking of doing. I was kind of envisioning > the following: [...] Ah, OK. Sounds fine to me! From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 19:24:34 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1CFEBF89 for ; Thu, 24 Jul 2014 19:24:34 +0000 (UTC) Received: from mail-qg0-f44.google.com (mail-qg0-f44.google.com [209.85.192.44]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CD79C2F07 for ; Thu, 24 Jul 2014 19:24:33 +0000 (UTC) Received: by mail-qg0-f44.google.com with SMTP id e89so3910699qgf.3 for ; Thu, 24 Jul 2014 12:24:27 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=Wc93h6g/uoVGNh65heinmgA6lvmWGlqPpiKkbl9Dp2I=; b=eXb4wJxR/2zZQufr3XbYoy3UrGH2C78OqrPR6zQSrSRK60kR6TW06Z+Co3t2kdbGxz 7TfKWQa01g8XFvoAfqfwEIfEyYvdC85uqudK5ZWoS+34wBt07DhLBhm5NMF+Q3urNWM0 jNwG0ZB40M0KdvEt73ruPMt5VxdJRzI9Qq3GgQS9WzvtuOsZ6nuZrdjkzzxg6v+diiI5 4sFxY28P58USkTFMbvPzqVZWddh8YsKAWQkbylObyjXfVGfDhwEqK/5611XuZg7qaPJj uTVFdiJAE1Y7FFdP/cINNcU82l/bcn0hFr7pNH2/yDQbtK1QqxBqe1kk4kmwVNjnqz8k FNxQ== X-Gm-Message-State: ALoCoQkAjWdvXp8D3eeLI9aXZO4gnRAbpFKnMhzNfsrxcEbs+e+E2+6J/4jLD+7vT5AlpN3oX23N X-Received: by 10.140.109.118 with SMTP id k109mr18080907qgf.98.1406229867260; Thu, 24 Jul 2014 12:24:27 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Thu, 24 Jul 2014 12:24:07 -0700 (PDT) X-Originating-IP: [2620:0:1003:1007:ac3e:4bce:435e:6e80] In-Reply-To: <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com> References: <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com> From: Julio Merino Date: Thu, 24 Jul 2014 15:24:07 -0400 X-Google-Sender-Auth: 1qssWOGXL8oaxzYNDjnrfXnuHRY Message-ID: Subject: Re: Need input on preference on location of 3rd party tests vs FreeBSD tests To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 19:24:34 -0000 On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper wrote: > On Jul 18, 2014, at 7:45 AM, Alan Somers wrote: > >> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper = wrote: >>> Hi all, >>> One of the things that I've done on my fork of FreeBSD is I've import= ed ATF test suites from NetBSD and I have integrated existing test suites f= rom freebsd's tools/regression tree into Kyua as well. Due to the size and = difference in test content/coverage, I pulled lib/libc and lib/msun from bo= ther sources and integrated them into Kyua. What I did was I put the netbsd= testcases into the tests/ subdirectory and put the FreeBSD test suites int= o a tests/legacy subdirectory. The goal was that the legacy directory would= eventually be converted over to atf testcases and then could be removed on= ce the conversion was complete. >>> I'm not sure if this scheme makes sense though. Does anyone have a pr= eference as to whether or not this makes sense? >>> Thanks! >>> -Garrett >> >> >> I don't understand. What did you put in tests/legacy? > > The tests from tools/regression. tests/ contains the tests from NetBSD. I do not think adding tests under src/tests/ is a good idea. We have chosen to put tests under the 'tests' subdirectories of the affected components and we should stick to this rule except for very specific cases (see src/tests/sys/). Creating an artificial barrier between tests imported from NetBSD and native tests to FreeBSD is not beneficial (e.g. why should users/developers care where the tests came from?). "Reusing the NetBSD Makefiles" is not a strong enough reason because those are usually trivial -- and if they aren't, that's an indication that they are wrong! We have two options: 1) We treat the NetBSD tests as upstream. In this case, we ought to do the right thing, which is to put them in a vendor branch, merge into contrib, and add reachover Makefiles where necessary. OR 2) We just copy the code (effectively a fork), and when doing so we can do as we wish with the structure. So... Option 1 may sound nice, but it will be problematic long-term without collaboration from NetBSD: I don't think we want to treat NetBSD's tests as upstream if NetBSD themselves don't consider the tests as a "first class project" (and they don't currently). Not to mention that this will require a significant amount of local patching with the corresponding maintenance headaches. Option 2 is definitely easier to handle on our side, but with the obvious disadvantage that we won't be able to easily import tests from NetBSD. How big of a problem is that today? Honestly the fact that our codebases are separate for things like libc is more of a problem than having diverging tests. Option 3 (yes, another one!) would be to start a third-party repository holding "generic Unix tests" (a POSIX test suite?) that both NetBSD and FreeBSD import as a third-party product. Then, each system would just extend these suites with local patches to verify their system-specific behavior. A very nice option in my opinion, but this is a major effort. (And this can be tackled later once we the FreeBSD Test Suite is more complete and we can easily spot how much it overlaps to the NetBSD one.) From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 19:41:55 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 776D5904 for ; Thu, 24 Jul 2014 19:41:55 +0000 (UTC) Received: from mail-qa0-f45.google.com (mail-qa0-f45.google.com [209.85.216.45]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1759720CD for ; Thu, 24 Jul 2014 19:41:54 +0000 (UTC) Received: by mail-qa0-f45.google.com with SMTP id cm18so3462203qab.32 for ; Thu, 24 Jul 2014 12:41:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=FBKgilNLZoB7mRstax60QoWWeyN+v2+OWDoyEwvHuMg=; b=OA/fBeKiAdMeJq7TqF71Xwbhk8siuXRqBgKn3KXDIw7As54DT2/0qV+EX8zZpt7zzl nyiYvTJtODh36PknNp5QvWyVs1+WS4iFHMRejJCvuqhQoRMGSFebQDcSa9Mrc8YubvGj dvANMy6XoRuKapZxWj2fQ5Poeq+sVQ2hcBw44o8hN9PS9S7si815I4OvdM6T6dV8YmH8 MetzFYLqj7t7CGBlf166FnMeununRo1iocgv0+ldy/6QSp+eOJjlYO1+eIiCZwl8QdBB nYPill1dXQpRVdb/URYztktUypsyGsq8VH33/DYSWYa0LgAC2vlIyE9wZ/es+HFGuxWw FFoA== X-Gm-Message-State: ALoCoQmMpD5jnA2k/0ba8Y7DL2Rz8E942wGJ5yLlbmOX0P7TZgrWqHCy2uCLvAxiQTf8cKc+fPPd X-Received: by 10.224.160.134 with SMTP id n6mr4680724qax.84.1406230907828; Thu, 24 Jul 2014 12:41:47 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Thu, 24 Jul 2014 12:41:27 -0700 (PDT) X-Originating-IP: [2620:0:1003:1007:ac3e:4bce:435e:6e80] In-Reply-To: References: From: Julio Merino Date: Thu, 24 Jul 2014 15:41:27 -0400 X-Google-Sender-Auth: jzF6ZMA85N_AkJTJqYWgbuRldX8 Message-ID: Subject: Re: How do I extend Kyua requirements checking for FreeBSD? To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 19:41:55 -0000 On Thu, Jun 26, 2014 at 3:54 PM, Garrett Cooper wrote: >> >> I did it just be creating a library of shell functions that I call >> from test bodies. For example, something like this >> function require_module() { >> kldstat $1 || atf_skip "Module $1 is not loaded" >> } > > This is sort of what I'm doing, but this only works for the ATF test > interface :/. I used kldstat -m $1 by the way for detecting statically > compiled kernel modules, like aio: Yeah, creating a library of helper functions is the easiest way now with the disadvantage that this won't work for non-ATF test programs as you say. >> But that's not useful for C and C++ tests. And it would be way cooler >> if it were more tightly integrated into Kyua. Would it be possible to >> load site-specific Lua code that would handle this sort of >> functionality, so the test could simply say "atf_set require.modules >> aio"? > > Exactly! Hmm, Lua extensions. That's actually a very nice idea! From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 20:02:37 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C353A369 for ; Thu, 24 Jul 2014 20:02:37 +0000 (UTC) Received: from mail-qg0-f49.google.com (mail-qg0-f49.google.com [209.85.192.49]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 82F802292 for ; Thu, 24 Jul 2014 20:02:36 +0000 (UTC) Received: by mail-qg0-f49.google.com with SMTP id j107so3894998qga.8 for ; Thu, 24 Jul 2014 13:02:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=I/nCw7jG7oeTARqSDaTIdO8JtTPQCuePS1C6R+F9nW8=; b=H84LIKzuiNmzDuzmzaIQ9sChTfLtp8qky4nThRHrHvq06pAoh6pyYbpW9GwumkjYpJ gn+HbYDT9nURvtyEQkbz6AX6W2L2QBi1IOcImdG5pWosmbwTGxLfwTP164uvApBNbleF zmCu+0ZhNSj1DC1ZHXoqOW4ng1GXQ6XGNGiBwF6hzeFHjMkKvBRwseChQUTKNm0+hwaS Fn4rvkzLUXLV1dWAKHvAW3YE1+mVi4alp2INMVlVjnbDx3ff2IErbTBYe1X6arh0OgSw /T0hkuQAcJVpqEgkmzXmcLBg9C4er24tepjoIXuUUzGgyJy2iq8wcHXInawR4/mtz+aJ 6d/g== X-Gm-Message-State: ALoCoQlQQL6DIFpWyU7Cq8/+0+e7Djovg5+cB5wQYH7cn87prxGwQP0IehhYAGIwlRo6FjuXkIS4 X-Received: by 10.140.24.140 with SMTP id 12mr3791121qgr.11.1406232155665; Thu, 24 Jul 2014 13:02:35 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.96.83.99 with HTTP; Thu, 24 Jul 2014 13:02:14 -0700 (PDT) X-Originating-IP: [2620:0:1003:1007:ac3e:4bce:435e:6e80] In-Reply-To: References: From: Julio Merino Date: Thu, 24 Jul 2014 16:02:14 -0400 X-Google-Sender-Auth: yl0A756JrdUjZcemovSjk4Tfmhk Message-ID: Subject: Re: Best place to learn how to write tests which work under kyua? To: Craig Rodrigues Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 20:02:37 -0000 On Tue, Jun 10, 2014 at 6:06 PM, Craig Rodrigues wrote: > Hi, > > Where is the best place to learn about starting from > a clean slate, writing new tests which execute under kyua, > and how to structure the Kyuafile? > Specifically, I want to use kyua in my own project, which is > not part of the FreeBSD source tree. Then... this question belongs in kyua-discuss, doesn't it? ;-P > I looked at these links: > > http://wiki.netbsd.org/kyua/ > https://wiki.freebsd.org/TestingFreeBSD > https://wiki.freebsd.org/TestSuite > http://www.bsdcan.org/2014/schedule/events/462.en.html > > While those links gave good general overviews of kyua, > and also how to use it within the structure of /usr/tests on > FreeBSD, I did not get a clear idea of how to use kyua > in my own project. I thought I had something in the Kyua wiki pages, but I guess not. Meh, I'll have to write some other kind of documentation or introductory article... documentation is currently not very good, I agree. What kind of text would help you most? A step-by-step tutorial maybe? Big parts of the process may be tied to the build system you use though, and I only know the BSD build systems and Automake/autoconf... From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 20:26:12 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5E7D0D3D; Thu, 24 Jul 2014 20:26:12 +0000 (UTC) Received: from mail-ie0-x22b.google.com (mail-ie0-x22b.google.com [IPv6:2607:f8b0:4001:c03::22b]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1B59024AD; Thu, 24 Jul 2014 20:26:12 +0000 (UTC) Received: by mail-ie0-f171.google.com with SMTP id at1so2798605iec.2 for ; Thu, 24 Jul 2014 13:26:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=vsSq/suEZTjPnYaS+KndGn5EiZ7yQLyJc3uxWEpuj0k=; b=AS/vw9qIYVV/th3S14EPaLyCTrrx4S9hJB3rWj9QlSM/PQpdkXzgDwfDCAaVI5VeNe 9hEQwGEr0lNFFAg+ASVezrrCtwL01OKiSL+QsGkpr+WpnE96YIF02YoFhr7ZKs6hNipA FqMCN9gLgjm/VxkeyA6lHlMiS0qBrMq9kdzqGgOxKtxUUF4uwNK3Z8R1rs4ARNHk7Vd8 zU439W1GIQTlqZaIw/ejlr7NCssLGu9GBXATl+KoUfo9P0Fdb6MXyhZz8ePYgfBBiDSS vwEbloYY3u4VNUH6j8rIfvM+PkJpLUVQezsM6D8/u5wI8PXJ0rF6eCUQtQPEPbhF0Kw6 A5Hw== MIME-Version: 1.0 X-Received: by 10.50.66.133 with SMTP id f5mr42230289igt.38.1406233571395; Thu, 24 Jul 2014 13:26:11 -0700 (PDT) Received: by 10.50.213.102 with HTTP; Thu, 24 Jul 2014 13:26:11 -0700 (PDT) In-Reply-To: References: Date: Thu, 24 Jul 2014 13:26:11 -0700 Message-ID: Subject: Re: Best place to learn how to write tests which work under kyua? From: Garrett Cooper To: Julio Merino Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 20:26:12 -0000 On Thu, Jul 24, 2014 at 1:02 PM, Julio Merino wrote: > On Tue, Jun 10, 2014 at 6:06 PM, Craig Rodrigues wrote: >> Hi, >> >> Where is the best place to learn about starting from >> a clean slate, writing new tests which execute under kyua, >> and how to structure the Kyuafile? >> Specifically, I want to use kyua in my own project, which is >> not part of the FreeBSD source tree. > > Then... this question belongs in kyua-discuss, doesn't it? ;-P > >> I looked at these links: >> >> http://wiki.netbsd.org/kyua/ >> https://wiki.freebsd.org/TestingFreeBSD >> https://wiki.freebsd.org/TestSuite >> http://www.bsdcan.org/2014/schedule/events/462.en.html >> >> While those links gave good general overviews of kyua, >> and also how to use it within the structure of /usr/tests on >> FreeBSD, I did not get a clear idea of how to use kyua >> in my own project. > > I thought I had something in the Kyua wiki pages, but I guess not. > Meh, I'll have to write some other kind of documentation or > introductory article... documentation is currently not very good, I > agree. A topic like this is generally done in the handbook. > What kind of text would help you most? A step-by-step tutorial maybe? > Big parts of the process may be tied to the build system you use > though, and I only know the BSD build systems and Automake/autoconf... I would probably focus on examples of how to integrate things in FreeBSD in the FreeBSD handbook. Thanks! -Garrett From owner-freebsd-testing@FreeBSD.ORG Thu Jul 24 20:39:50 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A285B228; Thu, 24 Jul 2014 20:39:50 +0000 (UTC) Received: from mail-ie0-x229.google.com (mail-ie0-x229.google.com [IPv6:2607:f8b0:4001:c03::229]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 64B8F25A1; Thu, 24 Jul 2014 20:39:50 +0000 (UTC) Received: by mail-ie0-f169.google.com with SMTP id rd18so2797835iec.0 for ; Thu, 24 Jul 2014 13:39:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=L9CCJ10uDRW+q2wlpvqXiWnL2chh8AfBMCR8/ejGFPM=; b=Lt9cgWkfmJ8tgG6aEsrfa4v2Mw9bCf2scO4ir3AkeYc5zN+3c40sez/DK5BoGNPrlO 476IXK9cM+0fP3//G7ANkxXd/XhdlJQR/s5akHvpO02nAuLpnroJHtS2E8AesHlC5AAz b9MQCQ56UQVwft758oWpNOskOjQPH34f8mspCvrf/MUlt6ekvFmdv8DephJ2yJ6DM4Xh tUGlMK3UHAJ02KyYP5P7NC6BoWT3QX10wDdByn1gs8xfFfLJGMutwo7UXzOYodCaxpUz 4vnT8071jFO1wh+8oxGPM76lqyaGaOlu9QP4xavM08GVPAFhniIRAuyVfvdDLV3xq/P+ 7P/g== MIME-Version: 1.0 X-Received: by 10.42.198.75 with SMTP id en11mr15304278icb.7.1406234389824; Thu, 24 Jul 2014 13:39:49 -0700 (PDT) Received: by 10.50.213.102 with HTTP; Thu, 24 Jul 2014 13:39:49 -0700 (PDT) In-Reply-To: References: <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com> Date: Thu, 24 Jul 2014 13:39:49 -0700 Message-ID: Subject: Re: Need input on preference on location of 3rd party tests vs FreeBSD tests From: Garrett Cooper To: Julio Merino Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Garrett Cooper , "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 24 Jul 2014 20:39:50 -0000 On Thu, Jul 24, 2014 at 12:24 PM, Julio Merino wrote: > On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper wrot= e: >> On Jul 18, 2014, at 7:45 AM, Alan Somers wrote: >> >>> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper = wrote: >>>> Hi all, >>>> One of the things that I've done on my fork of FreeBSD is I've impor= ted ATF test suites from NetBSD and I have integrated existing test suites = from freebsd's tools/regression tree into Kyua as well. Due to the size and= difference in test content/coverage, I pulled lib/libc and lib/msun from b= other sources and integrated them into Kyua. What I did was I put the netbs= d testcases into the tests/ subdirectory and put the FreeBSD test suites in= to a tests/legacy subdirectory. The goal was that the legacy directory woul= d eventually be converted over to atf testcases and then could be removed o= nce the conversion was complete. >>>> I'm not sure if this scheme makes sense though. Does anyone have a p= reference as to whether or not this makes sense? >>>> Thanks! >>>> -Garrett >>> >>> >>> I don't understand. What did you put in tests/legacy? >> >> The tests from tools/regression. tests/ contains the tests from NetBSD. > > I do not think adding tests under src/tests/ is a good idea. We have > chosen to put tests under the 'tests' subdirectories of the affected > components and we should stick to this rule except for very specific > cases (see src/tests/sys/). Creating an artificial barrier between > tests imported from NetBSD and native tests to FreeBSD is not > beneficial (e.g. why should users/developers care where the tests came > from?). Sorry if this was a bit misleading in my original email... Basically I'm trying to solve the problem of keeping the FreeBSD testcases isolated from the NetBSD testcases. This only impacted two components (so far) -- libc and msun. What I did was placed the libc testcases from NetBSD in lib/libc/tests and the libc testcases from FreeBSD in lib/libc/tests/legacy (kind of matching the model that we have for TAP/plain scripts with the "legacy" name, but in name only). Example: - https://github.com/yaneurabeya/freebsd/tree/isilon-atf/lib/libc/tests - the NetBSD testcases (mostly) - https://github.com/yaneurabeya/freebsd/tree/isilon-atf/lib/libc/tests/leg= acy - the testcases from tools/regression on FreeBSD head > "Reusing the NetBSD Makefiles" is not a strong enough reason because > those are usually trivial -- and if they aren't, that's an indication > that they are wrong! Well, they're really trivial, yes, and apart from small caveats with bsd.progs.mk in HEAD today, there isn't a reason for the testcase Makefiles to be divergent, except when dealing with FreeBSD vs NetBSD MK_ knobs. > We have two options: > > 1) We treat the NetBSD tests as upstream. In this case, we ought to do > the right thing, which is to put them in a vendor branch, merge into > contrib, and add reachover Makefiles where necessary. This is what I was thinking of doing because I don't have a compelling argument and it doesn't really seem to make sense to fork [at this point]. The modifications that Peter and I have been making to the isilon-atf branch are relatively trivial/documented with #if defined(__FreeBSD__) or #if defined(__NetBSD__) (still not happy with the __arraycount vs nitems divergence, but it's a simple sed one-liner). I've pushed a number of patches out to NetBSD GNATS to fix issues with the testcases (mostly header discrepancies, but also some valid testcase bugs with NetBSD's copy of the tests). > OR > > 2) We just copy the code (effectively a fork), and when doing so we > can do as we wish with the structure. > > So... > > Option 1 may sound nice, but it will be problematic long-term without > collaboration from NetBSD: I don't think we want to treat NetBSD's > tests as upstream if NetBSD themselves don't consider the tests as a > "first class project" (and they don't currently). Not to mention that > this will require a significant amount of local patching with the > corresponding maintenance headaches. > > Option 2 is definitely easier to handle on our side, but with the > obvious disadvantage that we won't be able to easily import tests from > NetBSD. How big of a problem is that today? Honestly the fact that our > codebases are separate for things like libc is more of a problem than > having diverging tests. > > Option 3 (yes, another one!) would be to start a third-party > repository holding "generic Unix tests" (a POSIX test suite?) that > both NetBSD and FreeBSD import as a third-party product. Then, each > system would just extend these suites with local patches to verify > their system-specific behavior. A very nice option in my opinion, but > this is a major effort. (And this can be tackled later once we the > FreeBSD Test Suite is more complete and we can easily spot how much it > overlaps to the NetBSD one.) This is an alternative as well, but the tricky thing is that it moves away from the whole idea of "having the tests in the tree" which was part of the point behind doing things this way (to get developers in the habit of writing tests that match the test code), unless both FreeBSD and NetBSD collaboratively pull test code from the "BSD Test Project". Thanks! -Garrett From owner-freebsd-testing@FreeBSD.ORG Fri Jul 25 01:23:05 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 380A0D63 for ; Fri, 25 Jul 2014 01:23:05 +0000 (UTC) Received: from mail-wg0-f52.google.com (mail-wg0-f52.google.com [74.125.82.52]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BEF812F42 for ; Fri, 25 Jul 2014 01:23:03 +0000 (UTC) Received: by mail-wg0-f52.google.com with SMTP id a1so3519482wgh.35 for ; Thu, 24 Jul 2014 18:23:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type :content-transfer-encoding; bh=6yG2wZitMZpeWMj/fEQrunzBDJnjpfazmwPu/HStw38=; b=U612X4NXC9WLxjwW4TqPUV6KslADJfny/JK+9ZjJiMcFAVs4MCZ4qLx2VTEAH3WCx8 dwJep7Bf3Fg7PEaEXl6/JsuUprSopvpehSwzSEE6nKpey/h+Mqd6mZfSG+FmFMCSuoS3 +ipHqxIlK2/r8HoL7Cr7i9d1/H+9MGPYNsIXFTftBFV3Gif+tQ4xHyJhfzsCQ0kra2+w b1m+i4eseMRjU6xo5aqQyBLPYHZR/oUG5d7W5BGvUGZ5NFi0AdBfIC40HYc4o8eDuyt2 XqIuVn+Hcss11pzB0aieKd9NgU/NyuPKO5Tl1yYcRXA/YsiT8WNbBW0UCY6U0X8fBu8H OR/Q== X-Gm-Message-State: ALoCoQmGre59t5UCEMA3EwA/qG6LrCbLgIj0x/AjvKlpJQ3J8XAy6CI2CKFlYXI1sWD865wJCiZ7 X-Received: by 10.194.63.37 with SMTP id d5mr17465047wjs.92.1406250902663; Thu, 24 Jul 2014 18:15:02 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.194.203.3 with HTTP; Thu, 24 Jul 2014 18:14:42 -0700 (PDT) X-Originating-IP: [184.153.117.174] In-Reply-To: References: <4D9EB4FA-672A-47AC-8F6E-19D2B3FAB3F5@gmail.com> From: Julio Merino Date: Thu, 24 Jul 2014 21:14:42 -0400 X-Google-Sender-Auth: DRY63jysvCAT5SoyHlH99yWUmOk Message-ID: Subject: Re: Need input on preference on location of 3rd party tests vs FreeBSD tests To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: Garrett Cooper , "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 01:23:05 -0000 On Thu, Jul 24, 2014 at 4:39 PM, Garrett Cooper wro= te: > On Thu, Jul 24, 2014 at 12:24 PM, Julio Merino wrote: >> On Fri, Jul 18, 2014 at 5:52 PM, Garrett Cooper wro= te: >>> On Jul 18, 2014, at 7:45 AM, Alan Somers wrote: >>> >>>> On Fri, Jul 18, 2014 at 1:11 AM, Garrett Cooper wrote: >>>>> Hi all, >>>>> One of the things that I've done on my fork of FreeBSD is I've impo= rted ATF test suites from NetBSD and I have integrated existing test suites= from freebsd's tools/regression tree into Kyua as well. Due to the size an= d difference in test content/coverage, I pulled lib/libc and lib/msun from = bother sources and integrated them into Kyua. What I did was I put the netb= sd testcases into the tests/ subdirectory and put the FreeBSD test suites i= nto a tests/legacy subdirectory. The goal was that the legacy directory wou= ld eventually be converted over to atf testcases and then could be removed = once the conversion was complete. >>>>> I'm not sure if this scheme makes sense though. Does anyone have a = preference as to whether or not this makes sense? >>>>> Thanks! >>>>> -Garrett >>>> >>>> >>>> I don't understand. What did you put in tests/legacy? >>> >>> The tests from tools/regression. tests/ contains the tests from NetBSD. >> >> I do not think adding tests under src/tests/ is a good idea. We have >> chosen to put tests under the 'tests' subdirectories of the affected >> components and we should stick to this rule except for very specific >> cases (see src/tests/sys/). Creating an artificial barrier between >> tests imported from NetBSD and native tests to FreeBSD is not >> beneficial (e.g. why should users/developers care where the tests came >> from?). > > Sorry if this was a bit misleading in my original email... > > Basically I'm trying to solve the problem of keeping the FreeBSD > testcases isolated from the NetBSD testcases. This only impacted two > components (so far) -- libc and msun. What I did was placed the libc > testcases from NetBSD in lib/libc/tests and the libc testcases from > FreeBSD in lib/libc/tests/legacy (kind of matching the model that we > have for TAP/plain scripts with the "legacy" name, but in name only). Right... but I'm not sure if that's what FreeBSD should be doing. I can see how, in the context of Isilon, this was the best thing to do: keep yourself out of the business of writing tests for the base system where possible and just reuse what NetBSD had already done. However, we are now working towards the test suite that ships with FreeBSD itself, and in this case we really should be fully on-board with maintaining the tests themselves within the project. Compare this to the development of libc itself, for example. My biggest fear is that we will perform the initial import of tests from NetBSD (as a bunch of files littered throughout the tree, with no links to where they came from), then hack on the code to make it work for FreeBSD, then add a bunch of new tests by various users... and then never import updates from NetBSD again. There are plenty of examples of code that was shared across projects and that never suffered further reimports due to the difficulty of it; see the truly obsolete smbfs in NetBSD, the pf in FreeBSD, or the way tmpfs has gone different paths in NetBSD and FreeBSD. This is why I wouldn't introduce any intentional divergence between NetBSD and FreeBSD tests in the tree, in particular to prevent any internal fears about "this code is not ours so better not touch it much". Just import the code and, if we want to pull in new tests from NetBSD, do it on a case-by-case basis. (Note that I don't object to keeping the legacy tests in a subdirectory though as that sounds reasonable. But I feel that this is orthogonal to whether we (re)import tests from NetBSD or we write our shiny new test programs!) From owner-freebsd-testing@FreeBSD.ORG Fri Jul 25 01:39:16 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3B7A72A7 for ; Fri, 25 Jul 2014 01:39:16 +0000 (UTC) Received: from mail-wg0-f52.google.com (mail-wg0-f52.google.com [74.125.82.52]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C7378204F for ; Fri, 25 Jul 2014 01:39:15 +0000 (UTC) Received: by mail-wg0-f52.google.com with SMTP id a1so3422895wgh.23 for ; Thu, 24 Jul 2014 18:39:14 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-type; bh=XGF4IHYOFAV6CkcuegkQFqA3ZFCHeaXlvImSW7eevEA=; b=LASmoJLFP7mLHa8UXGlxvp8TMt9sezXuduIlGNGT/L+PNIaj2bSyrR6qx4KTue3xPI o8Y6I+1E+pIfjyGfRtG+KrezPKXszvuqxWUyYsIstsJb/1yVmbX5lhDyNLrKY3+4CWBQ /dg5hrleuL7b+b361qZ0ozaP/XN3JuIci20T5rQLdrOON0WRGHaGneRrgxgTfboryx30 YlJF5UvlRpAvY2LsBiDMEIZkPJBD/BpUupXVHCffW/MTq87fvnNvc+7BsMMxDW9tH6/a QFrxM133BLCt4/VQim72ljQ2y0vMdYP8K0gTFyP0v4trM0//6VSVjloykF6BY7ET6+hx LNKg== X-Gm-Message-State: ALoCoQky3Qaiy/elYcIcjpo86GGIYaLdsVJurD1yAbTazNeTI29oLv7fUSv/jZKIfdG+AYJkAbJN X-Received: by 10.180.38.51 with SMTP id d19mr1035683wik.10.1406250970448; Thu, 24 Jul 2014 18:16:10 -0700 (PDT) MIME-Version: 1.0 Sender: jmmv@meroh.net Received: by 10.194.203.3 with HTTP; Thu, 24 Jul 2014 18:15:50 -0700 (PDT) X-Originating-IP: [184.153.117.174] In-Reply-To: References: From: Julio Merino Date: Thu, 24 Jul 2014 21:15:50 -0400 X-Google-Sender-Auth: zqk5MRaFvhjL5VbVnQWiOP1Ptwk Message-ID: Subject: Re: Best place to learn how to write tests which work under kyua? To: Garrett Cooper Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 01:39:16 -0000 On Thu, Jul 24, 2014 at 4:26 PM, Garrett Cooper wrote: > On Thu, Jul 24, 2014 at 1:02 PM, Julio Merino wrote: >> I thought I had something in the Kyua wiki pages, but I guess not. >> Meh, I'll have to write some other kind of documentation or >> introductory article... documentation is currently not very good, I >> agree. > > A topic like this is generally done in the handbook. Not that Craig was asking about general documentation on writing tests _not_ for FreeBSD. That cannot go in the handbook ;-) From owner-freebsd-testing@FreeBSD.ORG Fri Jul 25 18:19:49 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3F107D77; Fri, 25 Jul 2014 18:19:49 +0000 (UTC) Received: from mail-lb0-x236.google.com (mail-lb0-x236.google.com [IPv6:2a00:1450:4010:c04::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8EA552E01; Fri, 25 Jul 2014 18:19:48 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id z11so3751627lbi.27 for ; Fri, 25 Jul 2014 11:19:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=AW61YW3Sh75OqLO7AtgyzmlF9F8rXa3Ml4z3yhfgUrQ=; b=07XwyARG08qDgFdDDpsfVuOhtMFa/M73tUd+KIU9QQg9pbhnaLwI6RDj0JBnC7ZKJg OAZRqmBLuryNuweOtHF+kNzvUXgEVLOsvIY8Ftr+hbIAdfy2ikA6v/Uj5xwRtBR0WI6I HOBP3cNsE7+z00dDNLr/2XAKIl2JVVv39kOzv4jM7b5bCHiGCKrsyv3x0hAgpGspbDfg xdLY6zBOQbR04RJkf2foCQRfBp3SqAkpjq5bdok2f9EWY4A7SU4JazG55L+ZU+vefvG0 FDp6+OQeZzv1fjzbwuh0rwYpaS3DYtftSv9FZrbv5Q9heF+zhF+7baogCJ2/JpyI8+Cj flhg== MIME-Version: 1.0 X-Received: by 10.152.36.135 with SMTP id q7mr13938857laj.42.1406312386507; Fri, 25 Jul 2014 11:19:46 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.112.225.34 with HTTP; Fri, 25 Jul 2014 11:19:46 -0700 (PDT) In-Reply-To: References: Date: Fri, 25 Jul 2014 11:19:46 -0700 X-Google-Sender-Auth: lgoCrPEUBVwe-LYDVp0KW0p4zmQ Message-ID: Subject: Re: Best place to learn how to write tests which work under kyua? From: Craig Rodrigues To: Julio Merino Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.18 Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 18:19:49 -0000 On Thu, Jul 24, 2014 at 1:02 PM, Julio Merino wrote: > I thought I had something in the Kyua wiki pages, but I guess not. > Meh, I'll have to write some other kind of documentation or > introductory article... documentation is currently not very good, I > agree. > > What kind of text would help you most? A step-by-step tutorial maybe? > Big parts of the process may be tied to the build system you use > though, and I only know the BSD build systems and Automake/autoconf... > A step-by-step tutorial would be a good start. The wiki pages that are available now, do not give a good idea of even how to get started with writing Kyua-compatible tests. For example, if I was working in a company where I had to point a QA team to some docs so that they could write Kyua-compatible tests for a software product, there is nothing I can point them to now. It's great that there are examples in the FreeBSD src tree (which is what I started looking at to understand how things work), but Kyua should be able to stand on its own, without FreeBSD. I'm neutral on whether the docs should live in the FreeBSD handbook or elsewhere. There are pros and cons for each approach. -- Craig From owner-freebsd-testing@FreeBSD.ORG Fri Jul 25 18:56:07 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DF3EB56A for ; Fri, 25 Jul 2014 18:56:07 +0000 (UTC) Received: from smtprelay02.ispgateway.de (smtprelay02.ispgateway.de [80.67.31.40]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 772482188 for ; Fri, 25 Jul 2014 18:56:06 +0000 (UTC) Received: from [78.35.187.212] (helo=fabiankeil.de) by smtprelay02.ispgateway.de with esmtpsa (SSLv3:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1XAker-0004E9-Lm for freebsd-testing@freebsd.org; Fri, 25 Jul 2014 20:55:57 +0200 Date: Fri, 25 Jul 2014 20:55:59 +0200 From: Fabian Keil To: Subject: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details Message-ID: <5ee51891.12b049e9@fabiankeil.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/9OaCUxQFeVDQVhxqf1DcW4i"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 18:56:07 -0000 --Sig_/9OaCUxQFeVDQVhxqf1DcW4i Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable I'm currently evaluating using kyua to test a FreeBSD-specific shell script, however I'm having trouble analysing failing tests. For example, using: | #!/usr/bin/env atf-sh | | atf_test_case failing_test | failing_test_body() { | atf_check -s exit:1 -o empty -e empty 'true' | } | | atf_init_test_cases() { | atf_add_test_case failing_test | } I get: | fk@r500 ~/git/zogftw/tests $kyua --loglevel debug test -k Kyuafile exampl= e:failing_test | example:failing_test -> failed: atf-check failed; see the output of the= test for details [0.150s] | | 0/1 passed (1 failed) | Committed action 21 The test failure obviously is expected, but I'm missing the referenced "output of the test". The relevant log excerpt: ######## 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO test_cases (test_program_id, name, metadata_id) VALUES (:test_p= rogram_id, :name, :metadata_id) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: example:failing_te= st -> =20 20140725-183911 D 71935 utils/process/child.cpp:350: Spawned process 71950:= stdout=3D/tmp/kyua.7ESBAU/stdout.txt, stderr=3D/tmp/kyua.7ESBAU/stderr.txt 20140725-183911 D 71935 utils/process/child.cpp:167: Executing /usr/local/l= ibexec/kyua-atf-tester -t300 test -vhas.cleanup=3Dfalse -vunprivileged-user= =3Dtests /home/fk/git/zogftw/tests/example failing_test /tmp/kyua.7ESBAU/re= sult.txt 20140725-183911 D 71935 utils/process/child.cpp:145: Waiting for pid=3D71950 20140725-183911 D 71935 store/transaction.cpp:896: Storing __STDOUT__ (/tmp= /kyua.7ESBAU/stdout.txt) of test case 42 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO files (contents) VALUES (:contents) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO test_case_files (test_case_id, file_name, file_id) VALUES (:tes= t_case_id, :file_name, :file_id) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 D 71935 store/transaction.cpp:896: Storing __STDERR__ (/tmp= /kyua.7ESBAU/stderr.txt) of test case 42 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO files (contents) VALUES (:contents) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO test_case_files (test_case_id, file_name, file_id) VALUES (:tes= t_case_id, :file_name, :file_id) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement: = INSERT INTO test_results (test_case_id, result_type, = result_reason, start_time, end_time) VALUES (= :test_case_id, :result_type, :result_reason, :start_time, :end_time) 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; no = more rows 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: failed: atf-check = failed; see the output of the test for details [0.150s] 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout:=20 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: 0/1 passed (1 fail= ed) 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: Committed action 21 20140725-183911 I 71935 cli/main.cpp:326: Clean exit with code 1 ######## I'm looking for something like this: | fk@r500 ~ $/usr/local/libexec/atf-check -s exit:1 -o empty -e empty 'true' | Executing command [ true ] | Fail: incorrect exit status: 0, expected: 1 | stdout: | | stderr: | I didn't read all the docs available yet and I may be missing something, but seeing test failures is something I'd expect to work out of the box. Can someone familiar with kyua confirm that this is working as expected? I'm currently using: fk@r500 ~ $uname -a FreeBSD r500.local 11.0-CURRENT FreeBSD 11.0-CURRENT #651 r269009+6d12a21(f= k): Wed Jul 23 17:49:07 CEST 2014 fk@r500.local:/usr/obj/usr/src/sys/ZO= EY amd64 fk@r500 ~ $pkg info kyua\* kyua-20140215,1 kyua-atf-compat-0.2_2 kyua-cli-0.8_3 kyua-testers-0.2_3 Fabian --Sig_/9OaCUxQFeVDQVhxqf1DcW4i Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlPSqD8ACgkQBYqIVf93VJ2hrQCgphDcCWsi0L2w/J/mybeEHYae CtgAn3hVPRzcSDRIGi99+00ZRc3t12tG =tKn1 -----END PGP SIGNATURE----- --Sig_/9OaCUxQFeVDQVhxqf1DcW4i-- From owner-freebsd-testing@FreeBSD.ORG Fri Jul 25 19:16:55 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 28B2BF5A for ; Fri, 25 Jul 2014 19:16:55 +0000 (UTC) Received: from mail-wi0-x22c.google.com (mail-wi0-x22c.google.com [IPv6:2a00:1450:400c:c05::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B5C89239F for ; Fri, 25 Jul 2014 19:16:54 +0000 (UTC) Received: by mail-wi0-f172.google.com with SMTP id n3so1492875wiv.5 for ; Fri, 25 Jul 2014 12:16:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=vqncrcYJSiuNqyfdSSxT6G8CaR9jax3LuGVUaiGqa48=; b=aH9BN79JXirsY+PhF+N4x+T+zNCSkkbXitSObGjQsDRk3TsvvcHtfjpCZN6BmBamtG WJvR7KypxKhnnEtFxoGoGJTApNF2Qw/bqNArF+n7atMmpeBhMQnO8vdpaqBrgufR/+Tq XGXCQipn4rVsEG68XxDuw42dzRwWhVJW3T6hMK2SCDGa5aNslVJBL/XaRO6lUI8b+Yfz EdJERm/qo/9kheiEw4N3AnaTIVmrMCKoaBRWkh/ANejpySZNZ3cMi5f/vrJsN65aAo/u D1stOEUa0U3Jbo6Iju+l192sO/aRRvMFMxB57azgkZsvQ9Xw6z5yS1c1NhybGWOCP6Tg /Etg== MIME-Version: 1.0 X-Received: by 10.180.39.33 with SMTP id m1mr7976653wik.82.1406315812454; Fri, 25 Jul 2014 12:16:52 -0700 (PDT) Sender: asomers@gmail.com Received: by 10.194.203.134 with HTTP; Fri, 25 Jul 2014 12:16:52 -0700 (PDT) In-Reply-To: <5ee51891.12b049e9@fabiankeil.de> References: <5ee51891.12b049e9@fabiankeil.de> Date: Fri, 25 Jul 2014 13:16:52 -0600 X-Google-Sender-Auth: 7KsEStRZ7rTibBJSrVOib7XVwKs Message-ID: Subject: Re: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details From: Alan Somers To: Fabian Keil Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 Jul 2014 19:16:55 -0000 On Fri, Jul 25, 2014 at 12:55 PM, Fabian Keil wrote: > I'm currently evaluating using kyua to test a FreeBSD-specific > shell script, however I'm having trouble analysing failing tests. > > For example, using: > > | #!/usr/bin/env atf-sh > | > | atf_test_case failing_test > | failing_test_body() { > | atf_check -s exit:1 -o empty -e empty 'true' > | } > | > | atf_init_test_cases() { > | atf_add_test_case failing_test > | } > > I get: > > | fk@r500 ~/git/zogftw/tests $kyua --loglevel debug test -k Kyuafile exam= ple:failing_test > | example:failing_test -> failed: atf-check failed; see the output of t= he test for details [0.150s] > | > | 0/1 passed (1 failed) > | Committed action 21 > > The test failure obviously is expected, but I'm missing the referenced > "output of the test". The relevant log excerpt: > > ######## > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO test_cases (test_program_id, name, metadata_id) VALUES (:test= _program_id, :name, :metadata_id) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: example:failing_= test -> > 20140725-183911 D 71935 utils/process/child.cpp:350: Spawned process 7195= 0: stdout=3D/tmp/kyua.7ESBAU/stdout.txt, stderr=3D/tmp/kyua.7ESBAU/stderr.t= xt > 20140725-183911 D 71935 utils/process/child.cpp:167: Executing /usr/local= /libexec/kyua-atf-tester -t300 test -vhas.cleanup=3Dfalse -vunprivileged-us= er=3Dtests /home/fk/git/zogftw/tests/example failing_test /tmp/kyua.7ESBAU/= result.txt > 20140725-183911 D 71935 utils/process/child.cpp:145: Waiting for pid=3D71= 950 > 20140725-183911 D 71935 store/transaction.cpp:896: Storing __STDOUT__ (/t= mp/kyua.7ESBAU/stdout.txt) of test case 42 > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO files (contents) VALUES (:contents) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO test_case_files (test_case_id, file_name, file_id) VALUES (:t= est_case_id, :file_name, :file_id) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 D 71935 store/transaction.cpp:896: Storing __STDERR__ (/t= mp/kyua.7ESBAU/stderr.txt) of test case 42 > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO files (contents) VALUES (:contents) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO test_case_files (test_case_id, file_name, file_id) VALUES (:t= est_case_id, :file_name, :file_id) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 D 71935 utils/sqlite/database.cpp:275: Creating statement= : INSERT INTO test_results (test_case_id, result_type, = result_reason, start_time, end_time) VALUES= (:test_case_id, :result_type, :result_reason, :start_time, :end_ti= me) > 20140725-183911 D 71935 utils/sqlite/statement.cpp:192: Step statement; n= o more rows > 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: failed: atf-chec= k failed; see the output of the test for details [0.150s] > 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: > 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: 0/1 passed (1 fa= iled) > 20140725-183911 I 71935 utils/cmdline/ui.cpp:94: stdout: Committed action= 21 > 20140725-183911 I 71935 cli/main.cpp:326: Clean exit with code 1 > ######## > > I'm looking for something like this: > > | fk@r500 ~ $/usr/local/libexec/atf-check -s exit:1 -o empty -e empty 'tr= ue' > | Executing command [ true ] > | Fail: incorrect exit status: 0, expected: 1 > | stdout: > | > | stderr: > | > > I didn't read all the docs available yet and I may be missing something, > but seeing test failures is something I'd expect to work out of the box. > Can someone familiar with kyua confirm that this is working as expected? > > I'm currently using: > > fk@r500 ~ $uname -a > FreeBSD r500.local 11.0-CURRENT FreeBSD 11.0-CURRENT #651 r269009+6d12a21= (fk): Wed Jul 23 17:49:07 CEST 2014 fk@r500.local:/usr/obj/usr/src/sys/= ZOEY amd64 > fk@r500 ~ $pkg info kyua\* > kyua-20140215,1 > kyua-atf-compat-0.2_2 > kyua-cli-0.8_3 > kyua-testers-0.2_3 > > Fabian The test output is stored in Kyua's sqlite database. There are two ways to see it: 1) Run " kyua report-html" 2) Sometimes its easier just to rerun the testcase in debug mode. Do "kyua debug example:failing_test" -Alan From owner-freebsd-testing@FreeBSD.ORG Sat Jul 26 08:58:48 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 63B113BD; Sat, 26 Jul 2014 08:58:48 +0000 (UTC) Received: from smtprelay04.ispgateway.de (smtprelay04.ispgateway.de [80.67.31.31]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2137C2640; Sat, 26 Jul 2014 08:58:47 +0000 (UTC) Received: from [87.79.198.227] (helo=fabiankeil.de) by smtprelay04.ispgateway.de with esmtpsa (SSLv3:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1XAxoM-0003bW-G3; Sat, 26 Jul 2014 10:58:38 +0200 Date: Sat, 26 Jul 2014 10:58:41 +0200 From: Fabian Keil To: Subject: Re: Missing atf-check output: example:failing_test -> failed: atf-check failed; see the output of the test for details Message-ID: <450da50a.02ce9897@fabiankeil.de> In-Reply-To: References: <5ee51891.12b049e9@fabiankeil.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/AWhD6yk+bxIyBTkCdiIncNH"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Jul 2014 08:58:48 -0000 --Sig_/AWhD6yk+bxIyBTkCdiIncNH Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Alan Somers wrote: > On Fri, Jul 25, 2014 at 12:55 PM, Fabian Keil > wrote: > > I'm currently evaluating using kyua to test a FreeBSD-specific > > shell script, however I'm having trouble analysing failing tests. [...] > > I get: > > > > | fk@r500 ~/git/zogftw/tests $kyua --loglevel debug test -k Kyuafile ex= ample:failing_test > > | example:failing_test -> failed: atf-check failed; see the output of= the test for details [0.150s] > > | > > | 0/1 passed (1 failed) > > | Committed action 21 > > > > The test failure obviously is expected, but I'm missing the referenced > > "output of the test". The relevant log excerpt: [...] > > I'm looking for something like this: > > > > | fk@r500 ~ $/usr/local/libexec/atf-check -s exit:1 -o empty -e empty '= true' > > | Executing command [ true ] > > | Fail: incorrect exit status: 0, expected: 1 > > | stdout: > > | > > | stderr: > > | > > > > I didn't read all the docs available yet and I may be missing something, > > but seeing test failures is something I'd expect to work out of the box. > > Can someone familiar with kyua confirm that this is working as expected? > The test output is stored in Kyua's sqlite database. There are two > ways to see it: > 1) Run " kyua report-html" > 2) Sometimes its easier just to rerun the testcase in debug mode. Do > "kyua debug example:failing_test" Thanks a lot. Not showing test failures on the cli by default seems like a strange design decision to me, but I guess I'll get used to it or write a wrapper script that calls kyua debug for all tests. Fabian --Sig_/AWhD6yk+bxIyBTkCdiIncNH Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlPTbcEACgkQBYqIVf93VJ0RJQCglUy3nhQHeMLYPongYIgze9xk V7wAnAvNpRLqUrd9QrfQtUhjbdW8RgG/ =2/jp -----END PGP SIGNATURE----- --Sig_/AWhD6yk+bxIyBTkCdiIncNH-- From owner-freebsd-testing@FreeBSD.ORG Sat Jul 26 11:30:27 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 449B4307 for ; Sat, 26 Jul 2014 11:30:27 +0000 (UTC) Received: from smtprelay06.ispgateway.de (smtprelay06.ispgateway.de [80.67.31.103]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 06AA0224F for ; Sat, 26 Jul 2014 11:30:26 +0000 (UTC) Received: from [87.79.198.227] (helo=fabiankeil.de) by smtprelay06.ispgateway.de with esmtpsa (SSLv3:AES128-SHA:128) (Exim 4.68) (envelope-from ) id 1XB0B6-0004O4-4y for freebsd-testing@freebsd.org; Sat, 26 Jul 2014 13:30:16 +0200 Date: Sat, 26 Jul 2014 13:30:19 +0200 From: Fabian Keil To: Subject: atf_check() equivalent to test individual shell functions? Message-ID: <41d8a2fb.75b47028@fabiankeil.de> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/RA443hGBEO=kGyPObZgnDDi"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 26 Jul 2014 11:30:27 -0000 --Sig_/RA443hGBEO=kGyPObZgnDDi Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Here's another zogftw-related problem. Putting something like this in a test case works as expected: | atf_check -s exit:0 -o empty -e empty $ZOGFTW cmd zogftw_string_ends_with= "abc" "c" It would be preferably to source zogftw once at the beginning and call the shell function zogftw_string_ends_with() directly afterwards to reduce the overhead, though. The following sort of does this, but the stdout and stderr output isn't verified, it's tedious to write and if nothing fails, the "kyua debug" output isn't particular useful: | . $ZOGFTW source || atf_fail "Failed to source zogftw" | | if ! zogftw_string_ends_with "abc" "c"; then | atf_fail "Failed to find 'c' at the end of 'abc'" | fi The following also sort of works, but in case of errors the output isn't helpful (because atf_check_equal() doesn't see the actual function) and the stderr output isn't verified either: | . $ZOGFTW source || atf_fail "Failed to source zogftw" | | atf_check_equal $(zogftw_string_ends_with "abc" "c"; echo $?) "0" What I'm looking for is something like this: | atf_check_shell_function -s return:0 -o empty -e empty zogftw_string_ends= _with "abc" "c" atf-sh-api(3) doesn't seem to mention anything like it, so I'm wondering how other people are testing individual shell functions? Fabian --Sig_/RA443hGBEO=kGyPObZgnDDi Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlPTkUsACgkQBYqIVf93VJ0OgwCfbyxvQkkb85gHu4P/JexXBh2f kpUAoJN0FD32/2QTWPGNUoTw8gYtvprB =qBaI -----END PGP SIGNATURE----- --Sig_/RA443hGBEO=kGyPObZgnDDi--