From owner-freebsd-current@FreeBSD.ORG Wed Aug 12 12:29:36 2009 Return-Path: Delivered-To: current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1EA82106567B for ; Wed, 12 Aug 2009 12:29:36 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42]) by mx1.freebsd.org (Postfix) with ESMTP id F02638FC57 for ; Wed, 12 Aug 2009 12:29:35 +0000 (UTC) Received: from fledge.watson.org (fledge.watson.org [65.122.17.41]) by cyrus.watson.org (Postfix) with ESMTPS id A642446B35 for ; Wed, 12 Aug 2009 08:29:35 -0400 (EDT) Date: Wed, 12 Aug 2009 13:29:35 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: current@FreeBSD.org Message-ID: User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: Call for regression and performance testing - 8.0 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2009 12:29:36 -0000 Dear all: As we're approaching 8.0 BETA3, now would be a really good time to start identifying functional and performance regressions from the 7.x series. We've done a mixed job at this in the past, but when it comes to "I'll do better some day", there's no time like the present :-). Some notes: Functional testing We actually have a sizable regression test suite in src/tools/regression -- some of these tools will have broken as a result of 8-CURRENT development, so the first task may be to fix them. The next is to run them on 7-STABLE and 8-CURRENT and decide if things have gotten worse -- or maybe we have a bug to fix in both. Pick the tool of your choice, and give it a spin. More than one person per tool is fine, because that way we get more diverse testing. Performance testing On the performance front, life is always a bit more tricky -- performance testing is a subtle art. However, the most clear lessons are that (a) testing with diverse workloads and diverse environments is extremely important, (b) you should do multiple runs and use ministat(1) to analyze results, and (c) that you want to compare apples with apples -- use the same hardware/configuration wherever possible. Watch out for annoying nits such as partition layout affecting I/O throughput for two different installs on the same disk. Pick something you think is important to you: bytes/sec over TCP on loopback, web hits/sec, NFS ops/sec, disk I/O transactions/sec, and do some comparison between 7.2 and 8.0. If you find improvement -- great! If you find a regression, please start a thread on current@ to help get it diagnosed. And if you want help doing performance measurement for a particular workload that isn't well studied, send some e-mail to performance@ to ask for advice on how best to measure it. Thanks, Robert N M Watson Computer Laboratory University of Cambridge