From owner-svn-src-all@FreeBSD.ORG Thu Jun 10 06:39:55 2010 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F06F1065672; Thu, 10 Jun 2010 06:39:55 +0000 (UTC) (envelope-from yaneurabeya@gmail.com) Received: from mail-pv0-f182.google.com (mail-pv0-f182.google.com [74.125.83.182]) by mx1.freebsd.org (Postfix) with ESMTP id D23928FC17; Thu, 10 Jun 2010 06:39:54 +0000 (UTC) Received: by pvb32 with SMTP id 32so959977pvb.13 for ; Wed, 09 Jun 2010 23:39:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:subject:mime-version :content-type:from:in-reply-to:date:cc:content-transfer-encoding :message-id:references:to:x-mailer; bh=4AO/qAGuucOcfkoJIelOS9vENJ9ttsaVdXWVlRXX7zk=; b=ET3egQnnqhn/qaEJ0ssRdMABVE8etnufjzHYPDbH+KlnyjZZ4zICLU6et+ueIHchgP lquayx+3qftUenzBNyX6PvDBzD+ZEpGSSI/SAlay5pA6ESXR5hSo1pQJM6Gj8GCgMXSz iAnem1SUvJiXPnf6dMlRYpjd/GDOqNCPeSn18= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=txZZZbmoJaU8T4GB9+aP0uvEnc6K4ftkW+z2SV1xlFsOv/85kyGSMi6TmFWYmbJBah cn1whlC5ljt60dRTnRibuKWH5Wo5/cGmW+WiWQAsXO7sgdD9EPS8+Wt0cFv4KVGnMqQ6 R191EBKU+8xNp+aGZ1wJiTT2naSy1jiTv48wQ= Received: by 10.141.23.19 with SMTP id a19mr3179353rvj.63.1276150377291; Wed, 09 Jun 2010 23:12:57 -0700 (PDT) Received: from [192.168.0.204] (deviant.freebsdgirl.com [173.8.183.73]) by mx.google.com with ESMTPS id o38sm3642620rvp.2.2010.06.09.23.12.55 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 09 Jun 2010 23:12:56 -0700 (PDT) Mime-Version: 1.0 (Apple Message framework v1078) Content-Type: text/plain; charset=us-ascii From: Garrett Cooper In-Reply-To: <82F59C7F-7984-449A-BA04-796F0A17DC3E@gmail.com> Date: Wed, 9 Jun 2010 23:12:52 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: <201006091759.o59HxrJw096803@svn.freebsd.org> <20100609190445.GE56080@hoeg.nl> <82F59C7F-7984-449A-BA04-796F0A17DC3E@gmail.com> To: Pawel Worach X-Mailer: Apple Mail (2.1078) Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org Subject: Re: svn commit: r208954 - in head/contrib/llvm: . docs test tools/clang tools/clang/docs tools/clang/test website X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jun 2010 06:39:55 -0000 On Jun 9, 2010, at 3:20 PM, Pawel Worach wrote: > On Jun 9, 2010, at 23:58, Garrett Cooper wrote: >=20 >> On Wed, Jun 9, 2010 at 12:04 PM, Ed Schouten wrote: >>> Hi Garrett, >>>=20 >>> * Garrett Cooper wrote: >>>> Why strip test? This might actually be helpful for folks trying to >>>> evaluate whether or not they should upgrade to newer versions of >>>> clang. >>>=20 >>> The testsuite can be checked out separately from the vendor space. = It >>> will account for about 50 MB of additional disk space usage. >>=20 >> Yes, but the tests can change at any point in time and might not >> reflect the code stored in the repository. >>=20 >> Could a note at least be provided, or a port maintained to help = others >> (say vendors) who have to qualify FreeBSD that this is the particular >> version that needs to be picked up from in order to test our >> shiny new compiler? That would make things considerably easier to = work >> with, as I've worked as QA in a Linux shop in the past, where a lot = of >> Linux provided packages that we had in the tree did not have the >> associated test code, and as QA that made my job a pain to deal with. >>=20 >> Just a small file that stated where and/or how to obtain things would >> make this a lot easier. >>=20 >> I would think that developers would also like the same to see whether >> or not there are any particular known issues with clang/llvm. >>=20 >=20 > We have a buildbot running here[1] that has the goal of serving as a = qualification system. >=20 > It does a nightly build of llvm/clang trunk in a self-host config, = runs the test suite in both stages and then builds freebsd trunk = world+kernel and boots the result. Yes, but how much is tested, and what code paths are tested in the = compiled kernel? Do you have all of the devices required to test all of = the drivers and extensive tests to exercise everything? Probably not; it = sounds like what you have setup wise is a smoke test box -- which is = good but it's not comprehensive enough to catch all issues... That also doesn't test out i386, or powerpc. Ideally the project should have a cluster of dedicated machines for = regression testing available in a colo somewhere where someone can load = images on the machine via netboot, or install, then execute a set of = functional and/or performance tests to gauge whether or not the code has = functional or performance regressions. Don't get me wrong, what you guys have is good -- but it's just not 100% = convincing for someone making a switchover to a newer toolchain, because = of all of the different variables that might be out there with all of = the code in FreeBSD, as well as interactions with compiled code, and 3rd = party software as well. But this is just the beginning of clang on FreeBSD and the project in = and of itself has a long way to go. Adding the bits so that other groups = could help evaluate, add tests, etc to ensure that clang/llvm works as = expected is what we should be focusing on, not just runtime tests and = `works for me' scenarios. Thanks, -Garrett=