From owner-freebsd-hackers@freebsd.org Fri Sep 11 19:28:19 2020 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 804103E3ECB for ; Fri, 11 Sep 2020 19:28:19 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qt1-x82f.google.com (mail-qt1-x82f.google.com [IPv6:2607:f8b0:4864:20::82f]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Bp5QB2D4kz3VxZ for ; Fri, 11 Sep 2020 19:28:18 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qt1-x82f.google.com with SMTP id k25so8801134qtu.4 for ; Fri, 11 Sep 2020 12:28:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=Db+PtTcBIpzJB84avVvcvPK1FL2qJ+FyoYYD7oX0L60=; b=THxj4HQJv/jqFWh7aFzj3z7oYnlW5PAcPoL7mwtRk3cZF9sBe9z9zRVVfMVU89510a 7k7tdXReE9Y5LihYgIouQ/NKnr2d2VwssxnUZMQ7msU7aPJlvkGut/pTMfVdJD0THhkj d+4xArbm4b8P4AGz+VfOE92IT5lI0tYs0hsko2/tXP+P8nqFfQ1QGT2Gwi6giatYFoiO i/+tMzAyIX/rV+FSzceEahzp6FHh0GCNow5jyBfVA1XDCW1/qFEojM3zN5ZR2QU3rDni 6bIrjqFGgQHq7LhRaN4fI4MGz0/DG8ibhmPj0fan3AuqZNqooda0jGf9KBg2tbDb/B7F 7wQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=Db+PtTcBIpzJB84avVvcvPK1FL2qJ+FyoYYD7oX0L60=; b=C2t750VkENJNGi/aQ13yjjbxxRd1/ThYhI+4gbZzZhKG8tqpcXG6ENApAtMQdJ+yIP 2wNu9Grcrl9JSBTmqlpDUYbgdf4Kp9AXoTsvCN9r6AFzYQM+h8ZB1xULE5bPJIcppTpa 8i0EJ+fKISyfrpsdYYl28SZzly6IbD0kRGH5Oicg8684bsIlRXvCeSBMA56nGKJFWT8j mpr15fjN9J4iSCJUwgJ+FuJYhOGcruP7mdO6NnMtaiR7ur9pPTrypa9kJ3nAvoKTYspN 4dAfXFCnjeSMQsuS+D/+sBmYECAcm9EF6bjqwFEBOQJ2/NxiTgZcBpPRWQ4ABLwqkfSj HYwA== X-Gm-Message-State: AOAM530s5ZRgeaOtDuX6S3W2buU5+GeyxTuukggDfsAJkinkhwxgDOPh OpGikAe1OWKCjPrbCfV1aGEwtJ3m5L1RWg== X-Google-Smtp-Source: ABdhPJzceFidSPrh9KzkZwGQR8oBKFnXeujtL/jB+j9Vx4v4VcnMALJcVkmNUYDIG38JEKgkzhquvw== X-Received: by 2002:ac8:5d14:: with SMTP id f20mr3494414qtx.104.1599852497113; Fri, 11 Sep 2020 12:28:17 -0700 (PDT) Received: from raichu (toroon0560w-lp130-08-67-71-176-35.dsl.bell.ca. [67.71.176.35]) by smtp.gmail.com with ESMTPSA id v131sm4167147qka.97.2020.09.11.12.28.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 11 Sep 2020 12:28:16 -0700 (PDT) Sender: Mark Johnston Date: Fri, 11 Sep 2020 15:28:08 -0400 From: Mark Johnston To: Yoshihiro Ota Cc: freebsd-hackers@freebsd.org Subject: Re: Integration Test, Unit Test, and head/tests in FreeBSD Message-ID: <20200911190238.GA39530@raichu> References: <20200908201352.e4e78f7aea8a2b081adfb57e@j.email.ne.jp> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200908201352.e4e78f7aea8a2b081adfb57e@j.email.ne.jp> X-Rspamd-Queue-Id: 4Bp5QB2D4kz3VxZ X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=THxj4HQJ; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::82f as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-1.28 / 15.00]; RCVD_TLS_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-0.996]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; NEURAL_SPAM_SHORT(0.40)[0.405]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::82f:from]; NEURAL_HAM_MEDIUM(-0.98)[-0.984]; MID_RHS_NOT_FQDN(0.50)[]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; MAILMAN_DEST(0.00)[freebsd-hackers] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 11 Sep 2020 19:28:19 -0000 On Tue, Sep 08, 2020 at 08:13:52PM -0400, Yoshihiro Ota wrote: > Hi Hackers, > > How are FreeBSD unit tests, integration test, and CI pipe lines are connected? We do not have a rigid separation between types of tests. Most tests for executable programs will use shell scripts to invoke the programs and verify that they behave as expected, i.e., no unit testing. Most tests for libraries are C code that link against the library and test its entry points. Some library tests will compile standalone helper programs that link against the library and test it that way. It depends a lot on the nature of the code being tested. Individual source files may be linked into a test program to provide some type of unit testing, but we do not have any framework for mocking, and most of the code in the base system is not very amenable to being unit-tested. sbin/dhclient/tests/option-domain-search.c links some code from dhclient, for example. The CI pipeline periodically builds the head branch, installs it into a VM image, and runs tests in a VM. There is currently no way to ask it to run tests against a patch. You would need to run the tests manually, by installing tests and running "kyua test" from whichever subdirectory of /usr/tests you are interested in. Running it from /usr/tests will run all tests. > I found some documents from "man tests", tests/README, and few others. > > I also wasn't clear how unit tests and integration tests are organized in the tree well. Are all integration tests in shell script, for example? In principle tests can be written in any language. We make use of a test framework called ATF which supports sh, C and C++ (and maybe some others, I'm not sure). The test runner is called kyua and can consume output from multiple test frameworks like TAP or ATF. Tests can be written in other languages, they just need to declare their dependencies in the Makefile. Some of our kernel tests are written in python for instance; DTrace tests are written in ksh and perl. The CI system installs some packages to ensure that these tests still get run. Tests for individual programs or libraries usually live next to the sources, in a directory called "tests". Tests for the kernel live in tests/sys/. > For example, I want to make a change to ggated/ggatec under sbin/ggate and would like to couple with unit tests. Will I create sbin/ggate/tests or sbin/ggate/ggated/tests and sbin/ggate/ggatec/tests or all of these 3 directories based on APIs? I would suggest putting them in sbin/ggate/tests. Tests for the client and server components could be placed in separate files if that ends up being useful. I don't see any real advantage in splitting up client and server tests into separate directories given that ggate is relatively simple, and it seems likely that they will share some code. > Where should I put it and how do I set up so that review/phab can verify as well? For the purpose of review, you can submit a patch through phabricator just as you would for code changes. As I noted there is currently no mechanism to automatically run tests on your behalf; you'll have to run the tests yourself until they are committed, at which point they will be run regularly as part of the CI system.