From owner-freebsd-testing@FreeBSD.ORG Fri Feb 27 03:24:57 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 162D6607; Fri, 27 Feb 2015 03:24:57 +0000 (UTC) Received: from mail-pd0-x22a.google.com (mail-pd0-x22a.google.com [IPv6:2607:f8b0:400e:c02::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CCEA676C; Fri, 27 Feb 2015 03:24:56 +0000 (UTC) Received: by pdjz10 with SMTP id z10so17911757pdj.0; Thu, 26 Feb 2015 19:24:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :message-id:references:to; bh=gtCFxPbm6dOIlR3EUH6iBpQktuuuso34JlNqEh72U00=; b=TKlg16+9je6NWtJNNTefagzV/qA0E93T7Ct5zGEceQ8jMxvMhIHJ9CTW+b209F/dtI OTV2fBwYgGg19U0URiuJekJvnFeBlkUUlKLx47gHhPqJKhBhghcz3zAKDunh3nYS0hsb Ssg4z8bRSHQFrBdMqAVlKSMM5Qwm6hWYZdRpx5q6Gl4hcwczOz7bbB7m8YZJRGE2yJYM VVm/K1Js6F3firZQhKn1O1lc8aooFyoaUbGjWDRFbATp90THqr/inQCbHoGXsEWqIm+P UreZkWKeiCnugN7dDMgj8erCZUQXB5uAHifbSVUhI5sepYyFQO3Dw0QoWIEgCsJU5Y+6 +fnQ== X-Received: by 10.70.90.9 with SMTP id bs9mr8627091pdb.137.1425007496359; Thu, 26 Feb 2015 19:24:56 -0800 (PST) Received: from ?IPv6:2601:8:ab80:7d6:15c9:2275:95cb:45c0? ([2601:8:ab80:7d6:15c9:2275:95cb:45c0]) by mx.google.com with ESMTPSA id r7sm2270354pdo.90.2015.02.26.19.24.55 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Thu, 26 Feb 2015 19:24:55 -0800 (PST) Content-Type: multipart/signed; boundary="Apple-Mail=_B5E27DFE-3A9E-4B01-BB52-8A02313317B3"; protocol="application/pgp-signature"; micalg=pgp-sha512 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Running tests as a developer prior to commit From: Garrett Cooper In-Reply-To: Date: Thu, 26 Feb 2015 19:24:53 -0800 Message-Id: <8E237BF1-E8DB-44D6-BA9F-89E386D0F3C0@gmail.com> References: To: Craig Rodrigues X-Mailer: Apple Mail (2.1878.6) Cc: "freebsd-testing@freebsd.org" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Feb 2015 03:24:57 -0000 --Apple-Mail=_B5E27DFE-3A9E-4B01-BB52-8A02313317B3 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Feb 26, 2015, at 19:17, Craig Rodrigues wrote: > On Mon, Feb 23, 2015 at 7:20 PM, Ed Maste wrote: >=20 >>=20 >> This is precisely why I'd like to find a way to make it easy to run a >> (rather small) subset of tests, if a developer desires to do so. The >> alternative is that they decide it's too awkward to run the tests, = and >> just don't bother. Even in that case we're still better off today >> (since the Jenkins task will catch it) than we were some time ago >> before consistent test infrastructure existed. But it seems we ought >> to avoid the latency, mailing list churn, and masking effect of a >> broken test run if we can facilitate the developer running a set of >> tests that they've identified. >>=20 >=20 > If I could do: > cd /usr/src/bin/ls/ > make test >=20 > and have it run the unit tests associated with /bin/ls, that would be = a > good start. >=20 > It wouldn't be perfect, because if a change to /bin/ls, affects some = other > test > in /usr/src/usr.bin/foobar/ , this wouldn't catch it. You would have = to do > something > more complicated by building an expert system which can compute the > dependencies > in the tree and run the appropriate tests. But having that is better = than > nothing. >=20 > As far as I understand things, the way the unit tests are integrated = in the > Makefile > infrastructure, the tests need to be staged to /usr/tests/ in order to = be > run. > As part of the staging process, Kyuafiles are generated which are used = as > input > to the "kyua" binary. I don't have any make-fu to know how to = simplify all > of this stuff. >=20 > Well, on a positive note, at least we *are* running the unit tests = every > day as people > commit code, and reporting the results. That's a huge deal. > I've seen large companies who don't do as good a job with this stuff, = so we > are not doing too badly in FreeBSD. >=20 > Over time, I am sure we will improve things with better automation. >=20 > Julio Merino accomplished a huge milestone by getting *any* type of = unit > tests into the FreeBSD source > tree, and integrated with the build. Regularly running the tests = under > automation was the next milestone. >=20 > The next milestone will be making it easier for developers to use. I = think > that the solution > which will be realistically achievable, which people will actually use = and > not rebel over > is some type of test automation hooked into Github pull requests = against > the FreeBSD src tree. Hi, This hook (running tests from a subdir) already exists, but is = incomplete: $ make test *** Using this test does not preclude you from running the tests *** installed in /usr/tests. This test run may raise false *** positives and/or false negatives. *** WARNING: make test is experimental *** *** Using this test does not preclude you from running the tests *** installed in /usr/tests. This test run may raise false *** positives and/or false negatives. kyua: E: Load of '/usr/tests/bin/pkill/Kyuafile' failed: File = '/usr/tests/bin/pkill/Kyuafile' not found. *** Error code 2 Stop. make: stopped in /usr/src.svn/bin/pkill/tests I have code on github which expands make test to install everything to = ${.OBJDIR}, then run the tests. make testworld doesn=92t exist either, so I=92ll work on plumbing that = into the build system, because having to cd into every directory to run = tests is silly. Thanks for the reminder :). --Apple-Mail=_B5E27DFE-3A9E-4B01-BB52-8A02313317B3 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJU7+OFAAoJEMZr5QU6S73e8S0H/3TbaLzJbQf7QrcnrFwW0aLR h4RNxQgQQp2DO/OluAZikb6H1WfdxJ074AVRisA+4cFm8kgRKKQ/41jf04fciWXz 2zpMEMyPojD/n++aSiU1kRyaNNk2mAH/hS/6w/1Z9X/0C7ah9/yOWTlJqtATGKOT yXdFchMjqTz1GFLGOJGtLPxPFuuLrnl1JXOFmYpsrZLpkcJhx8LokkPHgcLxrvXm hGRbzOoKtI0dNPy3Z+bgbyCjc95zb/FFN2FBjDFVGtaHv6WthJYIe61LQ8jErWJq wHQ2sEWWr2S4GBspi5eqyC8eq1O1xCs42+5xnWMtSU4x2bx3ZkaAWgaEhn9WV9A= =Bhnx -----END PGP SIGNATURE----- --Apple-Mail=_B5E27DFE-3A9E-4B01-BB52-8A02313317B3--