Date: Fri, 11 Sep 2020 15:28:08 -0400 From: Mark Johnston <markj@freebsd.org> To: Yoshihiro Ota <ota@j.email.ne.jp> Cc: freebsd-hackers@freebsd.org Subject: Re: Integration Test, Unit Test, and head/tests in FreeBSD Message-ID: <20200911190238.GA39530@raichu> In-Reply-To: <20200908201352.e4e78f7aea8a2b081adfb57e@j.email.ne.jp> References: <20200908201352.e4e78f7aea8a2b081adfb57e@j.email.ne.jp>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Sep 08, 2020 at 08:13:52PM -0400, Yoshihiro Ota wrote: > Hi Hackers, > > How are FreeBSD unit tests, integration test, and CI pipe lines are connected? We do not have a rigid separation between types of tests. Most tests for executable programs will use shell scripts to invoke the programs and verify that they behave as expected, i.e., no unit testing. Most tests for libraries are C code that link against the library and test its entry points. Some library tests will compile standalone helper programs that link against the library and test it that way. It depends a lot on the nature of the code being tested. Individual source files may be linked into a test program to provide some type of unit testing, but we do not have any framework for mocking, and most of the code in the base system is not very amenable to being unit-tested. sbin/dhclient/tests/option-domain-search.c links some code from dhclient, for example. The CI pipeline periodically builds the head branch, installs it into a VM image, and runs tests in a VM. There is currently no way to ask it to run tests against a patch. You would need to run the tests manually, by installing tests and running "kyua test" from whichever subdirectory of /usr/tests you are interested in. Running it from /usr/tests will run all tests. > I found some documents from "man tests", tests/README, and few others. > > I also wasn't clear how unit tests and integration tests are organized in the tree well. Are all integration tests in shell script, for example? In principle tests can be written in any language. We make use of a test framework called ATF which supports sh, C and C++ (and maybe some others, I'm not sure). The test runner is called kyua and can consume output from multiple test frameworks like TAP or ATF. Tests can be written in other languages, they just need to declare their dependencies in the Makefile. Some of our kernel tests are written in python for instance; DTrace tests are written in ksh and perl. The CI system installs some packages to ensure that these tests still get run. Tests for individual programs or libraries usually live next to the sources, in a directory called "tests". Tests for the kernel live in tests/sys/. > For example, I want to make a change to ggated/ggatec under sbin/ggate and would like to couple with unit tests. Will I create sbin/ggate/tests or sbin/ggate/ggated/tests and sbin/ggate/ggatec/tests or all of these 3 directories based on APIs? I would suggest putting them in sbin/ggate/tests. Tests for the client and server components could be placed in separate files if that ends up being useful. I don't see any real advantage in splitting up client and server tests into separate directories given that ggate is relatively simple, and it seems likely that they will share some code. > Where should I put it and how do I set up so that review/phab can verify as well? For the purpose of review, you can submit a patch through phabricator just as you would for code changes. As I noted there is currently no mechanism to automatically run tests on your behalf; you'll have to run the tests yourself until they are committed, at which point they will be run regularly as part of the CI system.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20200911190238.GA39530>