From owner-freebsd-testing@FreeBSD.ORG  Sun Mar  2 20:47:02 2014
Return-Path: <owner-freebsd-testing@FreeBSD.ORG>
Delivered-To: freebsd-testing@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id E501EF12
 for <freebsd-testing@freebsd.org>; Sun,  2 Mar 2014 20:47:02 +0000 (UTC)
Received: from mail-qa0-f46.google.com (mail-qa0-f46.google.com
 [209.85.216.46])
 (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id 9F27A1720
 for <freebsd-testing@freebsd.org>; Sun,  2 Mar 2014 20:47:01 +0000 (UTC)
Received: by mail-qa0-f46.google.com with SMTP id i13so2768143qae.33
 for <freebsd-testing@freebsd.org>; Sun, 02 Mar 2014 12:47:01 -0800 (PST)
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20130820;
 h=x-gm-message-state:mime-version:sender:in-reply-to:references:from
 :date:message-id:subject:to:cc:content-type;
 bh=pby86351a54jMXmAsyHgmFC6ZJuXNCvMGlRAZq0TwDA=;
 b=Pt477uRRMlWfYnz8rPwVyBTAZ1gbrlgGiZtpMGCPhYImyIb6r4Eb28geN2+1WaL0K6
 QFUPbzTRne9qBQ/CUTzqkCS5ti4qLhAPK2HBZAgt7/Our6bUncl2CdOK7p5iiJO/Pc1b
 Ctcs7kDv+8S2pJWZ2Os8BJuJuyDuSJVIEt1aEqxw8Odj9E44r48uUfAYdWVYW6Yfv7Lw
 3AnuJ0KPuI861tJVgdHz9jjeIKqA3tvHHhMX8YUAh6h1OQtcMBkQxG7O9dluTVx93YVm
 MOkhmDiGHGeuX1rFTOWreSHGlop3ByeDiq97pIyPvEajF5GmCiLd5pYh9XLNhUTpWVif
 kbgQ==
X-Gm-Message-State: ALoCoQkW9ZeC0kDBN2Txvr5m1VmGhfn0k5g6Tmp/y6XXG/9OHdvm4J3aIz9u5jFlDNQtAPh/t8IR
X-Received: by 10.140.39.50 with SMTP id u47mr18337408qgu.46.1393793221272;
 Sun, 02 Mar 2014 12:47:01 -0800 (PST)
MIME-Version: 1.0
Sender: jmmv@meroh.net
Received: by 10.96.83.102 with HTTP; Sun, 2 Mar 2014 12:46:41 -0800 (PST)
X-Originating-IP: [108.176.158.82]
In-Reply-To: <20140225033526.GN85115@kiwi>
References: <CAG=rPVd_32sXvZnx_btys=GgjNrxTmC8NAnkwd4U-hRu7eCx=A@mail.gmail.com>
 <CAOtMX2jU3DbxeX3CDJzmWkBAL4CsQK=zs5Vmp8uCFYFze1g+nw@mail.gmail.com>
 <CAG=rPVfuz_K38KtvtCCkCNMADAc4d0xg_fSibbwZ+Yy3NOF=PQ@mail.gmail.com>
 <CAOtMX2ix1oKVUs9Vue=1RVxyxaZrGzb=3JLN-_LOUmxW=iL_qQ@mail.gmail.com>
 <20140225033526.GN85115@kiwi>
From: Julio Merino <jmmv@freebsd.org>
Date: Sun, 2 Mar 2014 15:46:41 -0500
X-Google-Sender-Auth: azjH0LJ4fveJkeQNScYjQN4gwio
Message-ID: <CAFY7cWCMjPw+pO36bcyAarTMOFF7CW+LRtunztHYivaR9BNEAw@mail.gmail.com>
Subject: Re: Generating Kyua test result output in JUnit XML format for
 Jenkins integration?
To: "R. Tyler Croy" <tyler@monkeypox.org>
Content-Type: text/plain; charset=ISO-8859-1
Cc: "freebsd-testing@freebsd.org" <freebsd-testing@freebsd.org>,
 jenkins-admin@freebsd.org
X-BeenThere: freebsd-testing@freebsd.org
X-Mailman-Version: 2.1.17
Precedence: list
List-Id: Testing on FreeBSD <freebsd-testing.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-testing>, 
 <mailto:freebsd-testing-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-testing/>
List-Post: <mailto:freebsd-testing@freebsd.org>
List-Help: <mailto:freebsd-testing-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-testing>, 
 <mailto:freebsd-testing-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sun, 02 Mar 2014 20:47:03 -0000

On Mon, Feb 24, 2014 at 10:35 PM, R. Tyler Croy <tyler@monkeypox.org> wrote:
>> 2) I don't see any "expected failure" results in the links that you
>> provided.  That's an absolute requirement for Kyua.  Treating expected
>> failures like ordinary failures will greatly reduce the utility of
>> Kyua, by reducing its ability to spot regressions.
>
>
> JUnit allows for the concept of "skipped" tests, which Jenkins' parsing code
> respects. IIRC this is just a "skipped=true" on a <testcase>, I believe that
> would help solve for this case.

No, it's not.  An expected failure is not the same as skipped.
Possibly the best approximation is to consider an expected failure as
a pass.