Date: Mon, 23 Feb 2015 23:00:25 -0500 From: Allan Jude <allanjude@freebsd.org> To: freebsd-testing@freebsd.org Subject: Re: Running tests as a developer prior to commit Message-ID: <54EBF759.1060002@freebsd.org> In-Reply-To: <CAG=rPVfvjVor-rt1S1TprOKcfvQL93m8zC4UNq%2BpdqDRcfxqjA@mail.gmail.com> References: <CAPyFy2DKMktGV2yp=u2aesoiGH=s_VMksO_7HkPwmwi6MU3S4Q@mail.gmail.com> <CAG=rPVfvjVor-rt1S1TprOKcfvQL93m8zC4UNq%2BpdqDRcfxqjA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --JTVHIspjfKiFMUDuIE5nVIp4t3dxEMCuK Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2015-02-23 21:37, Craig Rodrigues wrote: > On Mon, Feb 23, 2015 at 2:19 PM, Ed Maste <emaste@freebsd.org> wrote: >=20 >> >> It'd be even better if we can facilitate simple runs of a >> suitable subset of tests prior to a commit. It looks like "make test" >> is close, despite the warnings it emits about being experimental. Is >> there a path we can take to support it, for at least a common case of >> developing and testing on a FreeBSD-CURRENT install? >> >=20 > On the surface, your request seems simple. > However, I have seen the same request asked in multiple companies > I have worked at, and it is not so simple. >=20 > If someone touches an arbitrary piece of code in FreeBSD, > what are the subset of tests that need to be run to validate the change= ? > Some changes are very simple, and it is quite easy to extrapolate which= > tests need to be run. >=20 > However, some changes are more subtle, and it is not so obvious which > tests are affected. >=20 > Having knowledge about what changes affect what unit tests requires alm= ost > an expert system to be built. >=20 > The other suggestion I have seen in other companies is to require devel= opers > to run all unit tests for every change before they commit. This works = to > some > extent, because in a company you can force this behavior. >=20 > For FreeBSD, I think this will fail. We already have over 3000 tests u= nder > /usr/tests. > It's not that hard to run them with "cd /usr/tests && kyua test". > However, requiring FreeBSD developers to run these tests > for every change they do is a big change in how they submit code to Fre= eBSD. > This will probably result in rebellion. >=20 > I would like to see a compromise solution. I would like an optional > workflow, > where someone does a pull request on the FreeBSD repository in GitHub, > which then > kicks off automation which builds an image, runs the kyua tests, and > reports the results. >=20 > After seeing these test results, the developer would still be responsib= le > for manually > committing the change to SVN. >=20 > People who are OK with using GitHub can follow this workflow, and opt-i= n to > the tests. > People who hate git and GitHub can ignore it, and continue to commit > directly to SVN > as they do today. >=20 > There are a whole lot of automation libraries built on top of Jenkins a= nd > GitHub > which can facilitate this type of workflow. Projects like Mac Homebrew= > have very > sophisticated workflows which kick off tests in response to pull reques= ts. >=20 > Someone in FreeBSD would need to build out this test infrastructure. I= t's > doable, but > a lot of work. It might be nice to make a Google Summer of Code projec= t > out of this. >=20 > -- > Craig > _______________________________________________ > freebsd-testing@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-testing > To unsubscribe, send any mail to "freebsd-testing-unsubscribe@freebsd.o= rg" >=20 Doesn't phabricator have some hooks to launch tests of a "proposed" patch? This could help catch the errors before they are committed --=20 Allan Jude --JTVHIspjfKiFMUDuIE5nVIp4t3dxEMCuK Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJU6/dcAAoJEJrBFpNRJZKf1R4P/iRdhxFXpOlNtQY1sBsaO4Ab AfX7P8GmYnPul82+Ee+8TcS3onvr71yassXCSu7v3IB6IDz7NXuRjfrpYxe14w8u vAEdqlAJ40dn6EqjwJEKerBYxO74Ke3C3FKWJdTpx7c8ej7C6uDev3QzAxRnDJjK IQg7Pivzoi7fGClPg2QqvbCZHsF5OIy5qA+bKgr4SoY41F1lJDPBRzjn2gOnpyU6 gLLqJAr42IbtmcWsqWsxRaHWyR/fNpXfRmgj7X3jKaxhgqBWdhluaoJaMi5HaPVq 2Zqy1stwCnfMNd6pKIr0yIHBb1gj2gWqNh/R2nS04PyllRf81oFPGvEQnWboyMSB b/d6Ib+z2Vz6cSlINznTB67OKnGg1zdvfPqYUBxJ7O2B/CUzQcdVM3CFH3Z8YCWV TF7hjVoO77bv93L1tGyZ0EhAupe5WMkvgs+dwy6CKiZe5Jc3H/rov6lPHipBDfda gkIptT0BpJmL4HVvhEGRpeoO8PsFNNq02ogjNhNFUxzqFdMNbButwll/KjmfvxIM raCAEh4AsDrKehgIt6hBBqNMlGoHl4Ygu8QYfV+FfKR1MdwR5ATTwzp+acfu+7f+ iURZnNI9Gc1YvFZK617LT1Nhw/2UQ/NXgq0W5T7aliJE/EYRLPT3Z6NtB3RrA6Xp WalRzzynHs1KxsV/FS6F =55X9 -----END PGP SIGNATURE----- --JTVHIspjfKiFMUDuIE5nVIp4t3dxEMCuK--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54EBF759.1060002>