From owner-freebsd-arch@FreeBSD.ORG Wed Dec 22 10:08:51 2010 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F8221065670; Wed, 22 Dec 2010 10:08:51 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id 550C48FC08; Wed, 22 Dec 2010 10:08:51 +0000 (UTC) Received: from [192.168.2.104] (host86-163-246-141.range86-163.btcentralplus.com [86.163.246.141]) by cyrus.watson.org (Postfix) with ESMTPSA id 0911246B32; Wed, 22 Dec 2010 05:08:34 -0500 (EST) Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: "Robert N. M. Watson" In-Reply-To: Date: Wed, 22 Dec 2010 10:08:28 +0000 Content-Transfer-Encoding: quoted-printable Message-Id: <96AE5824-D19C-4CD2-9DC1-98D7D314CE1A@FreeBSD.org> References: <201012211500.16131.jhb@freebsd.org> To: Erik Cederstrand X-Mailer: Apple Mail (2.1082) Cc: mdf@freebsd.org, freebsd-arch@freebsd.org Subject: Re: Schedule for releases X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Dec 2010 10:08:51 -0000 On 22 Dec 2010, at 07:21, Erik Cederstrand wrote: > Den 21/12/2010 kl. 23.28 skrev Robert Watson: >=20 >> Looking at 7.x, I'm struck by how much it has slowed down. There's a = significant user community, but not a significant developer community.=20= >=20 > Which pretty much sums up a dilemma in the development of FreeBSD, I = think. Developers want users to try out their new shiny stuff, but users = don't want to spend time upgrading. >=20 > I think one of many things that would be great to do is to improve the = usability and coverage of the regression tests. This would take at least = some of the burden off developers who want to MFC their work. We already = have the tinderboxes, Coverity and Clang Static Analyzer, but apart from = pho's stress tests we don't have any automated runtime testing (as far = as I know). We have an increasing quantity of moderately decent regression tests in = our tools/regression tree, but they are only run selectively and = moderately rarely. A very nice contribution someone could make would be: (1) Make it so a recursive regression run is easily accessible -- "make = tests ; make runtests" -- in the root or similar. (2) Set up a "test tinderbox", perhaps using a hosted server at Sentex, = and perhaps using virtual machines so that many versions can be run. (3) Arrange a report website and report e-mails identifying the = introduction of regressions in branches. One known problem with our regression suite is that it includes tests = that crash certain versions (and ideally not later versions), so we'd = need something flexible enough to skip tests known to fail on particular = branches in some usefully configurable way. All this would be extremely useful -- we can then start pulling in more = regression tests written by other communities (other *BSD, etc), and = push people to produce more tests when they fix bugs. It would be = especially nice to run our security regression tests regularly (testing = permissions, ACL semantics, inter-process protections, etc). Robert=