Date: Wed, 16 Sep 2020 00:16:00 -0400 From: Yoshihiro Ota <ota@j.email.ne.jp> To: Lawrence Greening <lg49152@outlook.com> Cc: ota@j.email.ne.jp, "freebsd-hackers@freebsd.org" <freebsd-hackers@freebsd.org> Subject: Re: Developers' Handbook: Chapter on testing Message-ID: <20200916001600.9741e3230466be3a0e1ad098@j.email.ne.jp> In-Reply-To: <DM6PR04MB54847A47689C03D86619C3E5B2230@DM6PR04MB5484.namprd04.prod.outlook.com> References: <DM6PR04MB54847A47689C03D86619C3E5B2230@DM6PR04MB5484.namprd04.prod.outlook.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi, My responses are embedded below. By the way, is it okay to continue here or should I join freebst-testing? On Mon, 14 Sep 2020 08:25:21 +0000 Lawrence Greening <lg49152@outlook.com> wrote: > Hi. > 〓 > A variation of the following was posted to freebsd-testing. I would generally avoid cross-posting > but I'm reposting here because of recent interest in testing on this list... > > I am writing a chapter on testing for the FreeBSD Developers' Handbook. > > The following topics are expected to be covered: > > * FreeBSD Test Suite > * Kyua > * Automated Testing Framework (ATF) > * Organization of current FreeBSD tests. > * Using automation and virtualization to reduce the overhead in setting up the test > infrastructure. > * Examples of different types of testing. > * Reproducibility in building and testing. > > This list of topics is just a starting point. The scope and organization will evolve as the > chapter is written. > > After reading the chapter, the reader should be able to do at least the following: > > * Install and configure the necessary testing tools, test suites, etc. for the purposes of > testing FreeBSD. > * Run tests. > * Write tests. > * Submit tests. > > The chapter will include walkthroughs and guided or cumulative examples. That sounds a good starting point and what I started looking around. I found some extra documents below after posting my previous question. Mark and Warner's responses were also helpful to understand further. https://wiki.freebsd.org/TestSuite https://wiki.freebsd.org/TestSuite/DeveloperHowTo https://wiki.freebsd.org/TestSuite/Structure These wiki covers some of topics above. > At this preliminary stage, I am interested in the following feedback: > > (1) What material do you want to see covered? In addition to the list you have, topics on kernel code will be great. > (2) What material do you want to see prioritized? My personal interest is step by step guide to setup a test environment and one or few examples of how to start writing additional test cases. > (3) What general approach(es) do you want to see taken towards the subject matter? We need a good test coverage and start writing more test cases. My suggestion is lowering bar with good documents with examples to encourage people writing more test cases. Then, I'd like to have code-coverage tool run together. Then, I can check how/if my new code reduced test coverage and my new test cases increased test coverage. It will be even nicer if all tests are run by Jenkins and phabricator bounces back ones with broken test cases. > (4) Are there good example tests or areas of the code base that deserve focus, either because of > their importance, usefulness as examples, history of regressions, etc.? Not familiar enough on FreeBSD test case but I recently realized some of user land programs actively adding more test cases from svn mailing list. That's a good trend. > (5) Anything else? In my experiences, I like unit test cases at C/C++ API level. These allow me to test same scenarios over and over again each time I make a change. These also detect un-intended behavior changes. Unit test cases also pays off very quickly. I also think integration tests are important but some programs like client/server have extra difficulties. > Cheers, > Lawrence Regards, Hiro
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200916001600.9741e3230466be3a0e1ad098>