From owner-freebsd-current@FreeBSD.ORG Tue Oct 7 23:40:26 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 77802330; Tue, 7 Oct 2014 23:40:26 +0000 (UTC) Received: from mail-ig0-x235.google.com (mail-ig0-x235.google.com [IPv6:2607:f8b0:4001:c05::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 318CA1A3; Tue, 7 Oct 2014 23:40:26 +0000 (UTC) Received: by mail-ig0-f181.google.com with SMTP id r10so6317297igi.2 for ; Tue, 07 Oct 2014 16:40:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ZC08pIDu8LN9t+lXzGaNZa4mAf2ziHzqPYd7GDQUbbM=; b=dTuZUwkl/80pcL3+XozJClxK/0HUNwum0UOlQ7Q2MfbNPWjZ+gAipSt5pCtTxQUbNx nya+0AqY+csKJ0zyVNzBS/FlKtUCic0hYsqsZhzJZr/NwcC8aCWp9BBZPpYqCV/xqq9I 7tXw5pOAlG/vN5ZNRspWJmCHAQHaLK4/T8i8Sza3s4fBEyP3dC7U4h/dpybHsuP0ZNpp 1/TyWmoyHqcokJ/+B3a1WQXVfNtCXcAA0DOmP2vJV1qXzSX+nqXYQSl303is7QcsPjaC yEVQjDGgMY8ks4tdOzFGiR/7UzwRRCvguFaIsuCnUPWTG/j7jVhPJzLbhJYAQ5i2A4vw EIXw== MIME-Version: 1.0 X-Received: by 10.50.70.10 with SMTP id i10mr10046850igu.7.1412725225594; Tue, 07 Oct 2014 16:40:25 -0700 (PDT) Received: by 10.50.227.42 with HTTP; Tue, 7 Oct 2014 16:40:25 -0700 (PDT) In-Reply-To: References: Date: Tue, 7 Oct 2014 16:40:25 -0700 Message-ID: Subject: Re: Need help fixing tests in CURRENT From: NGie Cooper To: Craig Rodrigues Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-testing@freebsd.org" , freebsd-current Current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 07 Oct 2014 23:40:26 -0000 On Tue, Oct 7, 2014 at 4:38 PM, Craig Rodrigues wrote: > > On Tue, Oct 7, 2014 at 4:30 PM, NGie Cooper wrote: >> >> On Tue, Oct 7, 2014 at 2:04 PM, Craig Rodrigues >> wrote: >> > Hi, >> > >> > After committing this fix to yacc in HEAD ( >> > https://lists.freebsd.org/pipermail/svn-src-all/2014-October/092553.html >> > ) >> > I've been able to do: "cd /usr/tests; kyua test" >> > >> > and get 0 test failures: >> > >> > >> > https://jenkins.freebsd.org/jenkins/job/FreeBSD_HEAD-tests2/16/testReport/ >> >> Awesome -- hopefully this will be sustained for several runs (this >> is part of the reason why I haven't integrated all of the NetBSD tests >> yet). >> Thank you for all your work in getting us to green! >> > > For any type of test framework, if the tests aren't run regularly > and reported in an easy to understand manner, > then the tests will bitrot and people will ignore them. > > I've seen this happen in multiple companies I have worked at. > > Hopefully we can do better with FreeBSD, and foster a community > of people who are actively updating the tests, and taking test failures > seriously. > > If we don't do this, then people will just ignore the tests. Agreed. The problem with the NetBSD testcases and some of the bits in tools/regression is that there are a certain batch which are non-deterministic and blur the lines between functional and unittest. Thanks!