From owner-freebsd-testing@freebsd.org Fri Apr 1 10:21:02 2016 Return-Path: Delivered-To: freebsd-testing@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 37E88AEB9CB for ; Fri, 1 Apr 2016 10:21:02 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from smtprelay03.ispgateway.de (smtprelay03.ispgateway.de [80.67.31.37]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0038F1DA1 for ; Fri, 1 Apr 2016 10:21:01 +0000 (UTC) (envelope-from freebsd-listen@fabiankeil.de) Received: from [78.35.178.145] (helo=fabiankeil.de) by smtprelay03.ispgateway.de with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.84) (envelope-from ) id 1alvwi-00057i-Fu; Fri, 01 Apr 2016 12:04:52 +0200 Date: Fri, 1 Apr 2016 12:03:10 +0200 From: Fabian Keil To: Alex Denisov <1101.debian@gmail.com> Cc: freebsd-testing@freebsd.org Subject: Re: Available Tests Message-ID: <20160401120310.3be9446a@fabiankeil.de> In-Reply-To: References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; boundary="Sig_/cEFAIs2MDhGqM+6AZ.X4SJ_"; protocol="application/pgp-signature" X-Df-Sender: Nzc1MDY3 X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Apr 2016 10:21:02 -0000 --Sig_/cEFAIs2MDhGqM+6AZ.X4SJ_ Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Alex Denisov <1101.debian@gmail.com> wrote: > I just built FreeBSD 10 using =E2=80=9CWITH_TESTS=3D1=E2=80=9D, and run t= ests from =E2=80=98/usr/tests=E2=80=99 using =E2=80=98root' user. >=20 > I have found that there are ~3,5k test results, which makes me curious: a= re there other tests? Yes. If you take a closer look at /usr/src you'll find various other tests that aren't integrated into the kyua/atf tool chain (yet). I also believe that your ~3,5k test results represent more that ~3,5k individual tests due to suboptimal integration of other test suites. Various external test frameworks are also potentially relevant for FreeBSD, but have too much dependencies to integrate. As an example, a TCP/IP regression was found through Privoxy-Regression-Tes= t: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D173309 Obviously the regression could have been found with a FreeBSD-specific test as well, but somebody would have to spent the time to write such a test first. > I=E2=80=99m asking because, for instance, LLVM test suite has ~17k tests > (unit, regression, and performance tests). Some of these tests are probably also run on FreeBSD systems to test the clang integration ... > FreeBSD seems to be bigger, but has less tests. Unlike LLVM, FreeBSD contains a lot of old code from a time where automated testing was less common. Comparing it with other operating systems might make more sense. Fabian --Sig_/cEFAIs2MDhGqM+6AZ.X4SJ_ Content-Type: application/pgp-signature Content-Description: OpenPGP digital signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlb+R18ACgkQBYqIVf93VJ1b+QCfekCLHuzlGVJvcZqYCWIBQczI MgoAoKmbdX59jTqZFJNngl0HxrECGWEN =eFle -----END PGP SIGNATURE----- --Sig_/cEFAIs2MDhGqM+6AZ.X4SJ_--