From owner-freebsd-testing@freebsd.org Wed Feb 27 20:43:01 2019 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3F2391504BCE for ; Wed, 27 Feb 2019 20:43:01 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf1-f46.google.com (mail-lf1-f46.google.com [209.85.167.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 85A0E8F0C4; Wed, 27 Feb 2019 20:42:59 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf1-f46.google.com with SMTP id g12so13451046lfb.13; Wed, 27 Feb 2019 12:42:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=mrfczeiX4hHclqckP+gAuBXZOG8eZdoWvIc1O8TA+mw=; b=oRhIiP3tCxEkb4nr0YztfwpZ2n0WF1MdUhhd8hIU5J91JvpiJ/DCD/UNtSB2hpUSmT zIdPvHLjqeDTNPPugY6f1A6wVOrkELuk75jBZvnv4ayNIrrBJOwSVpBEjRyEoY12UFKJ fj06ZoHe3yyxeseCqyQ9UTmMsrhEi8VdH/6re7aTTVhwEfCkdNm5RpYhqoaVAgHtbD4F c3foKkQlv+gLzBtQgDzOHuOTxco72G1Yt04f3AO64aMemwcRy2Eax74hKdi4TU+z/SLE yB+/ST5oNb0+I3P+KDMvySJToC84PuVj9/rrhkf+UtN7sqwbXCRQPCbChdOKimbpylZY mQkg== X-Gm-Message-State: AHQUAuaoDOYde5xmpdat6fnWJblLHiif66AK+8h5rzEfjGCaD6oMX45Y JftpHwLvEYL5erpyAlfXE77XHM2K+RvGpeuCuI4= X-Google-Smtp-Source: AHgI3Ib8JotaBlbyXPgDf1GJ2iV0ASEyH7J3YwK9bm0KUvjpgnrQaEauQnaQAqQ+yhRZOzh5bxOAfib4KTqZONYWzs4= X-Received: by 2002:ac2:415a:: with SMTP id c26mr2129485lfi.62.1551300170885; Wed, 27 Feb 2019 12:42:50 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Alan Somers Date: Wed, 27 Feb 2019 13:42:38 -0700 Message-ID: Subject: Re: atf-c++ vs GoogleTest vs share/mk To: Enji Cooper Cc: "freebsd-testing@freebsd.org" , Ed Maste , Julio Merino , "Jonathan T. Looney" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 85A0E8F0C4 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of asomers@gmail.com designates 209.85.167.46 as permitted sender) smtp.mailfrom=asomers@gmail.com X-Spamd-Result: default: False [-3.37 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.18)[-0.177,0]; RCVD_IN_DNSWL_NONE(0.00)[46.167.85.209.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; FORGED_SENDER(0.30)[asomers@freebsd.org,asomers@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[asomers@freebsd.org,asomers@gmail.com]; IP_SCORE(-1.18)[ipnet: 209.85.128.0/17(-3.81), asn: 15169(-2.02), country: US(-0.07)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Feb 2019 20:43:01 -0000 On Wed, Feb 27, 2019 at 12:46 PM Enji Cooper wrote: > > > > On Feb 27, 2019, at 9:59 AM, Alan Somers wrote: > > > > So it turns out to be impossible to use GoogleMock with atf-c++. The > > problem is that atf-c++'s only way to report a test failure is > > ATF_FAIL, which immediately terminates the program on failure. That > > conflicts with the way that GoogleMock uses pthreads, which is to > > report a failure while a pthread mutex is locked. atf-c++ has many > > other shortcomings, too. It lacks the ATF_CHECK_* macros, and its > > syntax is surprisingly inconsistent with atf-c=E2=80=99s. > > Yes, I=E2=80=99ve brought that up before in the past. This is part of the= reason why I want to abandon/kill off atf-c++ in favor of something more f= unctionally complete, i.e., googletest > > > So I tried writing a C++ program that uses atf-c instead. But the > > Makefiles in share/mk make that a frustrating proposition. They don't > > want C++ programs to link to atf-c, and they don't want atf-c programs > > to be built in C++ mode. > > Well=E2=80=A6 yeah. This makes sense. > > > Googletest would probably work fine, but I would sorely miss ATF's > > test case isolation features. > > In what ways? Using plain tests builds in some level of isolation, but it= =E2=80=99s not perfect, and this is the direction I=E2=80=99m taking Google= test for a first iteration. Having $PWD be a disposable tmpdir is convenient, but I'm even more concerned about what happens if a googletest segfaults. In that case, its cleanup routines won't run. That could potentially be catastrophic for the entire test run if the test program does something like alter network configuration. Or do you have a plan for this? > > > So what should I do? Should I fix atf-c++? That would entail > > basically copying over everything from atf-c, which would be a lot of > > work. Or should I hack atf.test.mk to allow C++ programs to use > > atf-c? That would be ugly, but easier. Or should I just switch to > > Googletest, and live with its fragile cleanup handlers? > > Using Googletest with Googlemock is the way that I intend to have things = work. Adding ATF into the mix as another layer seems like the ultimate path= to pain, since ATF=E2=80=99s libraries require full stack management in or= der to do things like handle sandboxing, etc, and Googlemock does some clev= er things with C++ that seems like they would conflict with ATF=E2=80=99s g= oals. > > I=E2=80=99ll work on closing out the first iteration of googletest.test.m= k by the end of the week and work on getting the code committed to ^/head. = In parallel, I=E2=80=99ll focus on adding googletest driver support to kyua= (I have part of it done, but there are still some missing pieces around te= st listing and results parsing), which is the next required phase in order = to make googletest function more flexibly with kyua. Cool. Feel free to add me as a reviewer. > > Thanks, > -Enji > > PS I appreciate the reminder about documenting my work and my goals on th= e wiki. I=E2=80=99ll do that now.