From owner-freebsd-testing@FreeBSD.ORG Tue Feb 25 03:06:03 2014 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B9317910; Tue, 25 Feb 2014 03:06:03 +0000 (UTC) Received: from mail-wi0-x232.google.com (mail-wi0-x232.google.com [IPv6:2a00:1450:400c:c05::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id F40911728; Tue, 25 Feb 2014 03:06:02 +0000 (UTC) Received: by mail-wi0-f178.google.com with SMTP id cc10so113293wib.11 for ; Mon, 24 Feb 2014 19:06:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=CPdF84RlqPHErU4QQNyjmm8qjnZT42FweB9sAGyGEEQ=; b=g9Qq/g0j82EetMGmj/madGuP1goJ1hduapx0hPO9AttEjN8lp1wvc370qXCwgKutnp h5OeuA+P5Q3mSht5amXvgoWASYenwJ+jByo9eK8mD09LyA5joeCNxPRmR47g7h4q3aDr KiRnDjRne7f0rG+YwrCxiqXPP6A9/F8QI6/InDbhfzVwdXMpfCgAQev2hXqQF7YXlH6A z/qimn6UGoDWuq4p2C8KNMJNMrx94j2OG/zMx+eKfn9GDAlGhnE3WUJUpbIeSpwAE7Yh mWR4FQp4fvZu00xuhdvOT5GjSSyFrtc+gVJACIjMVp7VjEJukxVvp0T5vJ+b7WrjYlWv Qq1A== MIME-Version: 1.0 X-Received: by 10.194.87.195 with SMTP id ba3mr25954wjb.53.1393297561406; Mon, 24 Feb 2014 19:06:01 -0800 (PST) Sender: asomers@gmail.com Received: by 10.194.168.197 with HTTP; Mon, 24 Feb 2014 19:06:01 -0800 (PST) In-Reply-To: References: Date: Mon, 24 Feb 2014 20:06:01 -0700 X-Google-Sender-Auth: RIJAB87bqpPERCLWIbqFlNClxE0 Message-ID: Subject: Re: Generating Kyua test result output in JUnit XML format for Jenkins integration? From: Alan Somers To: Craig Rodrigues Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-testing@freebsd.org" , jenkins-admin@freebsd.org X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Feb 2014 03:06:03 -0000 On Mon, Feb 24, 2014 at 5:58 PM, Craig Rodrigues wrote: > > > > On Mon, Feb 24, 2014 at 3:07 PM, Alan Somers wrote: >> >> On Mon, Feb 24, 2014 at 2:11 PM, Craig Rodrigues >> wrote: >> > Hi, >> > >> > Some Continuous Integration systems like Jenkins >> > and Atlassian Bamboo can natively parse >> > a test result format called JUnit XML. >> > >> > How hard would it be to add some code >> > to Kyua to optionally be able to generate test result output >> > in JUnit XML? >> > >> > Here is some example Junit XML output: >> > >> > http://nose2.readthedocs.org/en/latest/plugins/junitxml.html >> > >> > It would be super cool if by BSDCan we could demo: >> > >> > * start Kyua tests in Jenkins >> > * have Kyua run the tests and generate the JUnit XML test output >> > * have the tests results visible in the Jenkins web UI, something like: >> > http://zbyhoo.files.wordpress.com/2011/05/jenkins-tests.png >> > >> > >> > How hard would this be to do? >> > >> > Is there anyone available who is interested in working on this? >> >> This is a very awesome idea. Kyua's report-html command is only 286 >> SLOC long, so this probably won't be very hard. Do you have a link to >> the format specification? I'm afraid that I can't do the work, >> however. I have extremely limited free time ATM. > > > > > The closest thing I have to a format specification are links to these XSD > files: > > https://svn.jenkins-ci.org/trunk/hudson/dtkit/dtkit-format/dtkit-junit-model/src/main/resources/com/thalesgroup/dtkit/junit/model/xsd/junit-4.xsd > (simple) > http://windyroad.com.au/dl/Open%20Source/JUnit.xsd (more complicated) On second thought, I think that Junit-XML output is a good idea in the short term, but not in the long term. In the long term, displaying Kyua results in Jenkins is nonideal because: 1) A Kyua run could easily have thousands of test cases, which are impractical to display in a flat view like Jenkins does. Some sort of hierarchical view will be necessary. 2) I don't see any "expected failure" results in the links that you provided. That's an absolute requirement for Kyua. Treating expected failures like ordinary failures will greatly reduce the utility of Kyua, by reducing its ability to spot regressions. 3) The Kyua DB format is designed to support viewing a test case's success history over time, even though the tools don't do it yet. That's going to be a killer feature when its implemented, but I doubt that Jenkins has support for that kind of thing. I think that it will be difficult if not impossible to add it via a Jenkins plugin, but perhaps I just lack imagination. In the long term, I think we'll need to write a native webapp for viewing Kyua results. But in the short term, it will definitely be easier to get Jenkins integration, so I think that it's worthwhile to pursue both. It sounds like we'll probably have to read the Jenkins source to figure out exactly what it thinks JUnit-XML format is. -Alan