Date: Wed, 9 Jun 2010 23:12:52 -0700 From: Garrett Cooper <yaneurabeya@gmail.com> To: Pawel Worach <pawel.worach@gmail.com> Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org Subject: Re: svn commit: r208954 - in head/contrib/llvm: . docs test tools/clang tools/clang/docs tools/clang/test website Message-ID: <B3CB5DF6-4B3E-4B91-BFDC-0E0C0715632E@gmail.com> In-Reply-To: <82F59C7F-7984-449A-BA04-796F0A17DC3E@gmail.com> References: <201006091759.o59HxrJw096803@svn.freebsd.org> <AANLkTinHk6hZrCxBuyCiIiFSkCuAnN2ZnlmIsrhChQp-@mail.gmail.com> <20100609190445.GE56080@hoeg.nl> <AANLkTinlzRhxSk84MfFXHWwTdnV6MC_tqw1uimkwU99_@mail.gmail.com> <82F59C7F-7984-449A-BA04-796F0A17DC3E@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jun 9, 2010, at 3:20 PM, Pawel Worach wrote: > On Jun 9, 2010, at 23:58, Garrett Cooper wrote: >=20 >> On Wed, Jun 9, 2010 at 12:04 PM, Ed Schouten <ed@80386.nl> wrote: >>> Hi Garrett, >>>=20 >>> * Garrett Cooper <yanefbsd@gmail.com> wrote: >>>> Why strip test? This might actually be helpful for folks trying to >>>> evaluate whether or not they should upgrade to newer versions of >>>> clang. >>>=20 >>> The testsuite can be checked out separately from the vendor space. = It >>> will account for about 50 MB of additional disk space usage. >>=20 >> Yes, but the tests can change at any point in time and might not >> reflect the code stored in the repository. >>=20 >> Could a note at least be provided, or a port maintained to help = others >> (say vendors) who have to qualify FreeBSD that this is the particular >> version that needs to be picked up from <blah> in order to test our >> shiny new compiler? That would make things considerably easier to = work >> with, as I've worked as QA in a Linux shop in the past, where a lot = of >> Linux provided packages that we had in the tree did not have the >> associated test code, and as QA that made my job a pain to deal with. >>=20 >> Just a small file that stated where and/or how to obtain things would >> make this a lot easier. >>=20 >> I would think that developers would also like the same to see whether >> or not there are any particular known issues with clang/llvm. >>=20 >=20 > We have a buildbot running here[1] that has the goal of serving as a = qualification system. >=20 > It does a nightly build of llvm/clang trunk in a self-host config, = runs the test suite in both stages and then builds freebsd trunk = world+kernel and boots the result. Yes, but how much is tested, and what code paths are tested in the = compiled kernel? Do you have all of the devices required to test all of = the drivers and extensive tests to exercise everything? Probably not; it = sounds like what you have setup wise is a smoke test box -- which is = good but it's not comprehensive enough to catch all issues... That also doesn't test out i386, or powerpc. Ideally the project should have a cluster of dedicated machines for = regression testing available in a colo somewhere where someone can load = images on the machine via netboot, or install, then execute a set of = functional and/or performance tests to gauge whether or not the code has = functional or performance regressions. Don't get me wrong, what you guys have is good -- but it's just not 100% = convincing for someone making a switchover to a newer toolchain, because = of all of the different variables that might be out there with all of = the code in FreeBSD, as well as interactions with compiled code, and 3rd = party software as well. But this is just the beginning of clang on FreeBSD and the project in = and of itself has a long way to go. Adding the bits so that other groups = could help evaluate, add tests, etc to ensure that clang/llvm works as = expected is what we should be focusing on, not just runtime tests and = `works for me' scenarios. Thanks, -Garrett=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B3CB5DF6-4B3E-4B91-BFDC-0E0C0715632E>