From owner-freebsd-testing@FreeBSD.ORG Tue Feb 25 03:35:37 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 48E8213A for ; Tue, 25 Feb 2014 03:35:37 +0000 (UTC) Received: from sanddollar.geekisp.com (sanddollar.geekisp.com [216.168.135.167]) by mx1.freebsd.org (Postfix) with SMTP id F25A619CA for ; Tue, 25 Feb 2014 03:35:36 +0000 (UTC) Received: (qmail 15705 invoked by uid 1003); 25 Feb 2014 03:35:30 -0000 Received: from unknown (HELO kiwi) (tyler@monkeypox.org@173.239.70.2) by mail.geekisp.com with (DHE-RSA-AES256-SHA encrypted) SMTP; 25 Feb 2014 03:35:29 -0000 Date: Mon, 24 Feb 2014 19:35:26 -0800 From: "R. Tyler Croy" To: Alan Somers Subject: Re: Generating Kyua test result output in JUnit XML format for Jenkins integration? Message-ID: <20140225033526.GN85115@kiwi> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="zBPbvmIlJjvpbu6L" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.22 (2013-10-16) Cc: "freebsd-testing@freebsd.org" , jenkins-admin@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Feb 2014 03:35:37 -0000 --zBPbvmIlJjvpbu6L Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable snipped some context, reply still inline On Mon, 24 Feb 2014, Alan Somers wrote: > On Mon, Feb 24, 2014 at 5:58 PM, Craig Rodrigues wr= ote: > >> This is a very awesome idea. Kyua's report-html command is only 286 > >> SLOC long, so this probably won't be very hard. Do you have a link to > >> the format specification? I'm afraid that I can't do the work, > >> however. I have extremely limited free time ATM. > > > > > > The closest thing I have to a format specification are links to these X= SD > > files: > > > > https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-= model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd > > (simple) > > http://windyroad.com.au/dl/Open%20Source/JUnit.xsd (more complicated) >=20 > On second thought, I think that Junit-XML output is a good idea in the > short term, but not in the long term. In the long term, displaying > Kyua results in Jenkins is nonideal because: >=20 > 1) A Kyua run could easily have thousands of test cases, which are > impractical to display in a flat view like Jenkins does. Some sort of > hierarchical view will be necessary. Jenkins already solves for this, if the Kyua generated JUnit-XML uses dot-delimited (foo.bar) "class names" then Jenkins will organize things int= o a tree-like hierarchy. Here's an example of the display with Jenkins itself and its 4000+ test cas= es: > 2) I don't see any "expected failure" results in the links that you > provided. That's an absolute requirement for Kyua. Treating expected > failures like ordinary failures will greatly reduce the utility of > Kyua, by reducing its ability to spot regressions. JUnit allows for the concept of "skipped" tests, which Jenkins' parsing code respects. IIRC this is just a "skipped=3Dtrue" on a , I believe t= hat would help solve for this case. > 3) The Kyua DB format is designed to support viewing a test case's > success history over time, even though the tools don't do it yet. > That's going to be a killer feature when its implemented, but I doubt > that Jenkins has support for that kind of thing. I think that it will > be difficult if not impossible to add it via a Jenkins plugin, but > perhaps I just lack imagination. Boy howdy does Jenkins definitely support this! Unfortunately it's somewhat limited to a Job itself, but you can get pretty good test history down to a test-case level; > In the long term, I think we'll need to write a native webapp for > viewing Kyua results. But in the short term, it will definitely be > easier to get Jenkins integration, so I think that it's worthwhile to > pursue both. It sounds like we'll probably have to read the Jenkins > source to figure out exactly what it thinks JUnit-XML format is. Depending on your level of fear/loathing for Java, I actually disagree that Kyua should spend time building a custom web app for viewing results. The Jenkins project supports a *very* rich set of plugin APIs and extension poi= nts to where writing a custom Kyua plugin would be relatively easy. There's plenty of non-Jenkins examples of generating JUnit XML for what it's worth, in the Ruby community, the ci_reporter gem generates XML, Cucumber d= oes natively as well, Python's nosetest-xml package has plenty of examples as w= ell. I took a look at the C++ in Kyua and I'm afraid I'm not capable enough to w= rite the reporter, but if somebody wants to ping me on IRC (rtyler) I can probab= ly help them a bit if they have trouble with the format. - R. Tyler Croy ------------------------------------------------------ Code: Chatter: % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F ------------------------------------------------------ --zBPbvmIlJjvpbu6L Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iEYEARECAAYFAlMMD34ACgkQFCbH3D9R4W9HSQCfbDAZBPEv1ZYkYeaBNfnpvaPi Pl8AnjkGrAKg6jWQ+A1NgwkKh9tJp+YI =0k2Z -----END PGP SIGNATURE----- --zBPbvmIlJjvpbu6L--