From owner-freebsd-testing@FreeBSD.ORG Tue Feb 24 04:00:29 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 87BA7827 for ; Tue, 24 Feb 2015 04:00:29 +0000 (UTC) Received: from mx1.scaleengine.net (beauharnois2.bhs1.scaleengine.net [142.4.218.15]) by mx1.freebsd.org (Postfix) with ESMTP id 45BCB1EC for ; Tue, 24 Feb 2015 04:00:28 +0000 (UTC) Received: from [192.168.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id D3C15942A4 for ; Tue, 24 Feb 2015 04:00:26 +0000 (UTC) Message-ID: <54EBF759.1060002@freebsd.org> Date: Mon, 23 Feb 2015 23:00:25 -0500 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-testing@freebsd.org Subject: Re: Running tests as a developer prior to commit References: In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="JTVHIspjfKiFMUDuIE5nVIp4t3dxEMCuK" X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Feb 2015 04:00:29 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --JTVHIspjfKiFMUDuIE5nVIp4t3dxEMCuK Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2015-02-23 21:37, Craig Rodrigues wrote: > On Mon, Feb 23, 2015 at 2:19 PM, Ed Maste wrote: >=20 >> >> It'd be even better if we can facilitate simple runs of a >> suitable subset of tests prior to a commit. It looks like "make test" >> is close, despite the warnings it emits about being experimental. Is >> there a path we can take to support it, for at least a common case of >> developing and testing on a FreeBSD-CURRENT install? >> >=20 > On the surface, your request seems simple. > However, I have seen the same request asked in multiple companies > I have worked at, and it is not so simple. >=20 > If someone touches an arbitrary piece of code in FreeBSD, > what are the subset of tests that need to be run to validate the change= ? > Some changes are very simple, and it is quite easy to extrapolate which= > tests need to be run. >=20 > However, some changes are more subtle, and it is not so obvious which > tests are affected. >=20 > Having knowledge about what changes affect what unit tests requires alm= ost > an expert system to be built. >=20 > The other suggestion I have seen in other companies is to require devel= opers > to run all unit tests for every change before they commit. This works = to > some > extent, because in a company you can force this behavior. >=20 > For FreeBSD, I think this will fail. We already have over 3000 tests u= nder > /usr/tests. > It's not that hard to run them with "cd /usr/tests && kyua test". > However, requiring FreeBSD developers to run these tests > for every change they do is a big change in how they submit code to Fre= eBSD. > This will probably result in rebellion. >=20 > I would like to see a compromise solution. I would like an optional > workflow, > where someone does a pull request on the FreeBSD repository in GitHub, > which then > kicks off automation which builds an image, runs the kyua tests, and > reports the results. >=20 > After seeing these test results, the developer would still be responsib= le > for manually > committing the change to SVN. >=20 > People who are OK with using GitHub can follow this workflow, and opt-i= n to > the tests. > People who hate git and GitHub can ignore it, and continue to commit > directly to SVN > as they do today. >=20 > There are a whole lot of automation libraries built on top of Jenkins a= nd > GitHub > which can facilitate this type of workflow. Projects like Mac Homebrew= > have very > sophisticated workflows which kick off tests in response to pull reques= ts. >=20 > Someone in FreeBSD would need to build out this test infrastructure. I= t's > doable, but > a lot of work. It might be nice to make a Google Summer of Code projec= t > out of this. >=20 > -- > Craig > _______________________________________________ > freebsd-testing@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-testing > To unsubscribe, send any mail to "freebsd-testing-unsubscribe@freebsd.o= rg" >=20 Doesn't phabricator have some hooks to launch tests of a "proposed" patch? This could help catch the errors before they are committed --=20 Allan Jude --JTVHIspjfKiFMUDuIE5nVIp4t3dxEMCuK Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJU6/dcAAoJEJrBFpNRJZKf1R4P/iRdhxFXpOlNtQY1sBsaO4Ab AfX7P8GmYnPul82+Ee+8TcS3onvr71yassXCSu7v3IB6IDz7NXuRjfrpYxe14w8u vAEdqlAJ40dn6EqjwJEKerBYxO74Ke3C3FKWJdTpx7c8ej7C6uDev3QzAxRnDJjK IQg7Pivzoi7fGClPg2QqvbCZHsF5OIy5qA+bKgr4SoY41F1lJDPBRzjn2gOnpyU6 gLLqJAr42IbtmcWsqWsxRaHWyR/fNpXfRmgj7X3jKaxhgqBWdhluaoJaMi5HaPVq 2Zqy1stwCnfMNd6pKIr0yIHBb1gj2gWqNh/R2nS04PyllRf81oFPGvEQnWboyMSB b/d6Ib+z2Vz6cSlINznTB67OKnGg1zdvfPqYUBxJ7O2B/CUzQcdVM3CFH3Z8YCWV TF7hjVoO77bv93L1tGyZ0EhAupe5WMkvgs+dwy6CKiZe5Jc3H/rov6lPHipBDfda gkIptT0BpJmL4HVvhEGRpeoO8PsFNNq02ogjNhNFUxzqFdMNbButwll/KjmfvxIM raCAEh4AsDrKehgIt6hBBqNMlGoHl4Ygu8QYfV+FfKR1MdwR5ATTwzp+acfu+7f+ iURZnNI9Gc1YvFZK617LT1Nhw/2UQ/NXgq0W5T7aliJE/EYRLPT3Z6NtB3RrA6Xp WalRzzynHs1KxsV/FS6F =55X9 -----END PGP SIGNATURE----- --JTVHIspjfKiFMUDuIE5nVIp4t3dxEMCuK--