Date: Mon, 24 Feb 2014 19:35:26 -0800 From: "R. Tyler Croy" <tyler@monkeypox.org> To: Alan Somers <asomers@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, jenkins-admin@freebsd.org Subject: Re: Generating Kyua test result output in JUnit XML format for Jenkins integration? Message-ID: <20140225033526.GN85115@kiwi> In-Reply-To: <CAOtMX2ix1oKVUs9Vue=1RVxyxaZrGzb=3JLN-_LOUmxW=iL_qQ@mail.gmail.com> References: <CAG=rPVd_32sXvZnx_btys=GgjNrxTmC8NAnkwd4U-hRu7eCx=A@mail.gmail.com> <CAOtMX2jU3DbxeX3CDJzmWkBAL4CsQK=zs5Vmp8uCFYFze1g%2Bnw@mail.gmail.com> <CAG=rPVfuz_K38KtvtCCkCNMADAc4d0xg_fSibbwZ%2BYy3NOF=PQ@mail.gmail.com> <CAOtMX2ix1oKVUs9Vue=1RVxyxaZrGzb=3JLN-_LOUmxW=iL_qQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--zBPbvmIlJjvpbu6L Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable snipped some context, reply still inline On Mon, 24 Feb 2014, Alan Somers wrote: > On Mon, Feb 24, 2014 at 5:58 PM, Craig Rodrigues <rodrigc@freebsd.org> wr= ote: > >> This is a very awesome idea. Kyua's report-html command is only 286 > >> SLOC long, so this probably won't be very hard. Do you have a link to > >> the format specification? I'm afraid that I can't do the work, > >> however. I have extremely limited free time ATM. > > > > > > The closest thing I have to a format specification are links to these X= SD > > files: > > > > https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-= model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd > > (simple) > > http://windyroad.com.au/dl/Open%20Source/JUnit.xsd (more complicated) >=20 > On second thought, I think that Junit-XML output is a good idea in the > short term, but not in the long term. In the long term, displaying > Kyua results in Jenkins is nonideal because: >=20 > 1) A Kyua run could easily have thousands of test cases, which are > impractical to display in a flat view like Jenkins does. Some sort of > hierarchical view will be necessary. Jenkins already solves for this, if the Kyua generated JUnit-XML uses dot-delimited (foo.bar) "class names" then Jenkins will organize things int= o a tree-like hierarchy. Here's an example of the display with Jenkins itself and its 4000+ test cas= es: <https://ci.jenkins-ci.org/job/jenkins_main_trunk/3192/testReport/>; > 2) I don't see any "expected failure" results in the links that you > provided. That's an absolute requirement for Kyua. Treating expected > failures like ordinary failures will greatly reduce the utility of > Kyua, by reducing its ability to spot regressions. JUnit allows for the concept of "skipped" tests, which Jenkins' parsing code respects. IIRC this is just a "skipped=3Dtrue" on a <testcase>, I believe t= hat would help solve for this case. > 3) The Kyua DB format is designed to support viewing a test case's > success history over time, even though the tools don't do it yet. > That's going to be a killer feature when its implemented, but I doubt > that Jenkins has support for that kind of thing. I think that it will > be difficult if not impossible to add it via a Jenkins plugin, but > perhaps I just lack imagination. Boy howdy does Jenkins definitely support this! Unfortunately it's somewhat limited to a Job itself, but you can get pretty good test history down to a test-case level; <https://ci.jenkins-ci.org/job/jenkins_main_trunk/3192/testReport/hudso= n.tasks.test/TestObjectTest/testSafe/history/> > In the long term, I think we'll need to write a native webapp for > viewing Kyua results. But in the short term, it will definitely be > easier to get Jenkins integration, so I think that it's worthwhile to > pursue both. It sounds like we'll probably have to read the Jenkins > source to figure out exactly what it thinks JUnit-XML format is. Depending on your level of fear/loathing for Java, I actually disagree that Kyua should spend time building a custom web app for viewing results. The Jenkins project supports a *very* rich set of plugin APIs and extension poi= nts to where writing a custom Kyua plugin would be relatively easy. There's plenty of non-Jenkins examples of generating JUnit XML for what it's worth, in the Ruby community, the ci_reporter gem generates XML, Cucumber d= oes natively as well, Python's nosetest-xml package has plenty of examples as w= ell. I took a look at the C++ in Kyua and I'm afraid I'm not capable enough to w= rite the reporter, but if somebody wants to ping me on IRC (rtyler) I can probab= ly help them a bit if they have trouble with the format. - R. Tyler Croy ------------------------------------------------------ Code: <https://github.com/rtyler>; Chatter: <https://twitter.com/agentdero>; % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F ------------------------------------------------------ --zBPbvmIlJjvpbu6L Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iEYEARECAAYFAlMMD34ACgkQFCbH3D9R4W9HSQCfbDAZBPEv1ZYkYeaBNfnpvaPi Pl8AnjkGrAKg6jWQ+A1NgwkKh9tJp+YI =0k2Z -----END PGP SIGNATURE----- --zBPbvmIlJjvpbu6L--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140225033526.GN85115>