Date: Mon, 24 Feb 2014 20:06:01 -0700 From: Alan Somers <asomers@freebsd.org> To: Craig Rodrigues <rodrigc@freebsd.org> Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>, jenkins-admin@freebsd.org Subject: Re: Generating Kyua test result output in JUnit XML format for Jenkins integration? Message-ID: <CAOtMX2ix1oKVUs9Vue=1RVxyxaZrGzb=3JLN-_LOUmxW=iL_qQ@mail.gmail.com> In-Reply-To: <CAG=rPVfuz_K38KtvtCCkCNMADAc4d0xg_fSibbwZ%2BYy3NOF=PQ@mail.gmail.com> References: <CAG=rPVd_32sXvZnx_btys=GgjNrxTmC8NAnkwd4U-hRu7eCx=A@mail.gmail.com> <CAOtMX2jU3DbxeX3CDJzmWkBAL4CsQK=zs5Vmp8uCFYFze1g%2Bnw@mail.gmail.com> <CAG=rPVfuz_K38KtvtCCkCNMADAc4d0xg_fSibbwZ%2BYy3NOF=PQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Feb 24, 2014 at 5:58 PM, Craig Rodrigues <rodrigc@freebsd.org> wrote: > > > > On Mon, Feb 24, 2014 at 3:07 PM, Alan Somers <asomers@freebsd.org> wrote: >> >> On Mon, Feb 24, 2014 at 2:11 PM, Craig Rodrigues <rodrigc@freebsd.org> >> wrote: >> > Hi, >> > >> > Some Continuous Integration systems like Jenkins >> > and Atlassian Bamboo can natively parse >> > a test result format called JUnit XML. >> > >> > How hard would it be to add some code >> > to Kyua to optionally be able to generate test result output >> > in JUnit XML? >> > >> > Here is some example Junit XML output: >> > >> > http://nose2.readthedocs.org/en/latest/plugins/junitxml.html >> > >> > It would be super cool if by BSDCan we could demo: >> > >> > * start Kyua tests in Jenkins >> > * have Kyua run the tests and generate the JUnit XML test output >> > * have the tests results visible in the Jenkins web UI, something like: >> > http://zbyhoo.files.wordpress.com/2011/05/jenkins-tests.png >> > >> > >> > How hard would this be to do? >> > >> > Is there anyone available who is interested in working on this? >> >> This is a very awesome idea. Kyua's report-html command is only 286 >> SLOC long, so this probably won't be very hard. Do you have a link to >> the format specification? I'm afraid that I can't do the work, >> however. I have extremely limited free time ATM. > > > > > The closest thing I have to a format specification are links to these XSD > files: > > https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd > (simple) > http://windyroad.com.au/dl/Open%20Source/JUnit.xsd (more complicated) On second thought, I think that Junit-XML output is a good idea in the short term, but not in the long term. In the long term, displaying Kyua results in Jenkins is nonideal because: 1) A Kyua run could easily have thousands of test cases, which are impractical to display in a flat view like Jenkins does. Some sort of hierarchical view will be necessary. 2) I don't see any "expected failure" results in the links that you provided. That's an absolute requirement for Kyua. Treating expected failures like ordinary failures will greatly reduce the utility of Kyua, by reducing its ability to spot regressions. 3) The Kyua DB format is designed to support viewing a test case's success history over time, even though the tools don't do it yet. That's going to be a killer feature when its implemented, but I doubt that Jenkins has support for that kind of thing. I think that it will be difficult if not impossible to add it via a Jenkins plugin, but perhaps I just lack imagination. In the long term, I think we'll need to write a native webapp for viewing Kyua results. But in the short term, it will definitely be easier to get Jenkins integration, so I think that it's worthwhile to pursue both. It sounds like we'll probably have to read the Jenkins source to figure out exactly what it thinks JUnit-XML format is. -Alan
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAOtMX2ix1oKVUs9Vue=1RVxyxaZrGzb=3JLN-_LOUmxW=iL_qQ>