From owner-svn-src-all@freebsd.org Wed Oct 9 08:01:41 2019 Return-Path: Delivered-To: svn-src-all@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 12E97144B57; Wed, 9 Oct 2019 08:01:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46p69w0SZwz45kg; Wed, 9 Oct 2019 08:01:39 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x9981O7x055871 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Wed, 9 Oct 2019 11:01:27 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x9981O7x055871 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x9981Ois055870; Wed, 9 Oct 2019 11:01:24 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 9 Oct 2019 11:01:24 +0300 From: Konstantin Belousov To: "Enji Cooper (yaneurabeya)" Cc: Eric van Gyzen , src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r353305 - head/tests/sys/kern Message-ID: <20191009080124.GI44691@kib.kiev.ua> References: <201910081343.x98Dh5bW006905@repo.freebsd.org> <20191008150438.GE44691@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.12.2 (2019-09-21) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 46p69w0SZwz45kg X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; RCPT_COUNT_FIVE(0.00)[5]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(0.00)[ip: (-2.69), ipnet: 2001:470::/32(-4.57), asn: 6939(-3.40), country: US(-0.05)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 09 Oct 2019 08:01:41 -0000 On Tue, Oct 08, 2019 at 11:22:48PM -0700, Enji Cooper (yaneurabeya) wrote: > > > On Oct 8, 2019, at 08:04, Konstantin Belousov wrote: > > > > On Tue, Oct 08, 2019 at 01:43:05PM +0000, Eric van Gyzen wrote: > >> Author: vangyzen > >> Date: Tue Oct 8 13:43:05 2019 > >> New Revision: 353305 > >> URL: https://svnweb.freebsd.org/changeset/base/353305 > >> > >> Log: > >> Fix problems in the kern_maxfiles__increase test > >> > >> ATF functions such as ATF_REQUIRE do not work correctly in child processes. > >> Use plain C functions to report errors instead. > > There are much more tests that fork and use ATF_ in children. > > Look e.g. at most ptrace(2) tests. > > I beg to disagree: > > 86 /* > 87 * A variant of ATF_REQUIRE that is suitable for use in child > 88 * processes. This only works if the parent process is tripped up by > 89 * the early exit and fails some requirement itself. > 90 */ > 91 #define CHILD_REQUIRE(exp) do { \ > 92 if (!(exp)) \ > 93 child_fail_require(__FILE__, __LINE__, \ > 94 #exp " not met"); \ > 95 } while (0) Disagree to what ? How this citation is relevant to my statement that there are tests that use fork and ATF_ in children, I know about ptrace(2) tests at least. > > The issue, in this particular case, and the item that evangyzen was fixing, was the fact that failures in children can result in very confusing failures from a parent level. In particular, ATF_CHECK does not trickle up to parents and ATF_REQUIRE* gets thrown up to parents as abort()’ed processes. > > The best way to handle this within the atf-c/atf-c++ framework (with less boilerplate) is to use these APIs: atf_utils_fork(..)/atf_utils_wait(..). You will still need to use `_exit` (instead of exit(..)/assert(..)/ATF_CHECK(..)/ATF_REQUIRE(..), but it’s a lot less boilerplate than writing it yourself. This is not a way at all. Fork/wait is needed to create the child in controlled matter, sometimes we must even know precisely which syscalls are executed both by parent and child on their path. We need exactly the bare-bone syscalls to create the testing situation. > > Again, this is why I was driving towards GoogleTest. Despite the fact that it’s a C++ framework, there’s a lot less confusing boilerplate involved and context, since things are executed in relatively the same context, i.e., setup -> test -> teardown, and they’re easier to follow. > > The best way to move forward usability-wise with this (I think) is to probably alias the ATF_* macros to something more sensible when forking processes. However, given the amount of complaints I’ve heard about ATF, I think it’s best not to build upon an unstable foundation, but instead encourage the use of something more widely-accepted across the open source community/straightforward use wise. In this case, googletest. Can we have some testing framework where tests can be conveniently debugged, for the start ?