From owner-freebsd-hackers@freebsd.org Mon Sep 14 18:52:17 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D1C8D3DF3F0 for ; Mon, 14 Sep 2020 18:52:17 +0000 (UTC) (envelope-from sega01@go-beyond.org) Received: from haggis.mythic-beasts.com (haggis.mythic-beasts.com [IPv6:2a00:1098:0:86:1000:0:2:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4BqwTD5wphz4BNx for ; Mon, 14 Sep 2020 18:52:16 +0000 (UTC) (envelope-from sega01@go-beyond.org) Received: from [174.126.78.63] (port=22650 helo=localhost) by haggis.mythic-beasts.com with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92.3) (envelope-from ) id 1kHtaC-0001vB-Ny; Mon, 14 Sep 2020 19:52:09 +0100 Date: Mon, 14 Sep 2020 18:51:59 +0000 From: Teran McKinney To: Lawrence Greening Cc: "freebsd-hackers@freebsd.org" Subject: Re: Developers' Handbook: Chapter on testing Message-ID: <20200914185159.GA25758@daemon> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-BlackCat-Spam-Score: 4 X-Spam-Status: No, score=0.4 X-Rspamd-Queue-Id: 4BqwTD5wphz4BNx X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of sega01@go-beyond.org designates 2a00:1098:0:86:1000:0:2:1 as permitted sender) smtp.mailfrom=sega01@go-beyond.org X-Spamd-Result: default: False [-2.45 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1098::86:1000:0:2:1]; NEURAL_HAM_LONG(-1.05)[-1.049]; MIME_GOOD(-0.10)[text/plain]; MID_RHS_NOT_FQDN(0.50)[]; ARC_NA(0.00)[]; DMARC_NA(0.00)[go-beyond.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[2a00:1098:0:86:1000:0:2:1:from]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.41)[-0.410]; NEURAL_HAM_MEDIUM(-0.99)[-0.988]; FREEMAIL_TO(0.00)[outlook.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:44684, ipnet:2a00:1098::/32, country:GB]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Sep 2020 18:52:17 -0000 On 2020-09-14 08-25-21 , Lawrence Greening wrote: > Hi. > ??? > A variation of the following was posted to freebsd-testing. I would generally avoid cross-posting but I'm reposting here because of recent interest in testing on this list... > > I am writing a chapter on testing for the FreeBSD Developers' Handbook. > > The following topics are expected to be covered: > > * FreeBSD Test Suite > * Kyua > * Automated Testing Framework (ATF) > * Organization of current FreeBSD tests. > * Using automation and virtualization to reduce the overhead in setting up the test infrastructure. > * Examples of different types of testing. > * Reproducibility in building and testing. > > This list of topics is just a starting point. The scope and organization will evolve as the chapter is written. > > After reading the chapter, the reader should be able to do at least the following: > > * Install and configure the necessary testing tools, test suites, etc. for the purposes of testing FreeBSD. > * Run tests. > * Write tests. > * Submit tests. > > The chapter will include walkthroughs and guided or cumulative examples. > > At this preliminary stage, I am interested in the following feedback: > > (1) What material do you want to see covered? > (2) What material do you want to see prioritized? > (3) What general approach(es) do you want to see taken towards the subject matter? > (4) Are there good example tests or areas of the code base that deserve focus, either because of their importance, usefulness as examples, history of regressions, etc.? > (5) Anything else? > > Cheers, > Lawrence > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" Hi Lawrence, I personally like this idea a lot. I guess it might be a bit too late for 12.2-RELEASE but maybe could land in time for 13.0-RELEASE? I also wonder if there could be a single burn-in style command to run all of the tests (after making sure the kyua package is installed) once a box is installed. Or maybe an installer option to run the tests on first boot. I think though that some are not compatible with all of the security options, so may be some conflicts there. One angle I'm curious about is vulnerability-based testing, where a vulnerability comes out and a version of the exploit becomes a test case that can be ran. Not sure how much that is used with FreeBSD though or how relevant it is, just thought I'd mention it. Thank you, Teran