From owner-freebsd-qa Fri Dec 21 19:29:19 2001 Delivered-To: freebsd-qa@freebsd.org Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by hub.freebsd.org (Postfix) with ESMTP id B694037B405; Fri, 21 Dec 2001 19:29:12 -0800 (PST) Received: from fledge.watson.org (robert@fledge.pr.watson.org [192.0.2.3]) by fledge.watson.org (8.11.6/8.11.5) with SMTP id fBM3TAi15086; Fri, 21 Dec 2001 22:29:10 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Fri, 21 Dec 2001 22:29:09 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: stable@FreeBSD.org, qa@FreeBSD.org Cc: re@FreeBSD.org Subject: Testing guide for 4.5-PRERELEASE Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: owner-freebsd-qa@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG Testing Guide for 4.5-RELEASE As part of our on-going effort to improve the release engineering process, Murray has asked me to post my guide to things that Real Need Testing in the release candidate phase. Below, I describe changes in 4.5-PRERELEASE that we feel merit the most attention due to their involving substantial changes to the system, or having arrived late in the development cycle leading up to the release. In general, our goal in the QA process is to attempt to check a number of things: (1) The system has not regressed with respects to stability, correctness, interoperability, or performance of features present in prior releases. (2) New features result in the desired improvement in stability, correctness, interoperability, or performance. To effectively determine this, it's desirable to test the system in a diverse set of environments, applying a wide set of workloads, forcing the system to operate both within and outside its normal specification. Particular focus should often be placed on the continuing (or new) capability of the system to perform correctly when used in concert with systems from other vendors. Features to explore carefully: (1) Recent TCP changes, especially relating to the delayed ACK fix, congestion response, syncache, syncookies, increased socket buffer sizes, et al. We're interested in testing interoperability with as many platforms as possible, demonstrating continued strong (and better) scalability and performance, and watching out for quirks (connection stalls, ...), not to mention crashes. Jonathan Lemon was responding to a panic report on freebsd-current earlier today regarding a PCB call, which is something we should keep an eye on. On the other hand, Y! is now deploying this code, and that should help test it a great deal. (2) VFS/VM/NFS fixes. We need to continue to test performance, correctness, and interoperability. In particular, I'd like to see a lot of inter-platform performance testing (FreeBSD->Solaris, vice versa, etc). We'd also like careful investigation of low-memory situations. (3) FFS fixes. We had some reports of deadlocks in FFS; it sounds like Matt Dillon has caught most of them, but combinations I'd particular like to see tested involve Quotas, Chroot, and NFS, under load, and involving memory mapping and heavy directory operations. (4) NTP 4.1. This is probably reasonable safe, but it doesn't hurt to do interop testing, especially on Alpha. (5) SMBfs. We need stability testing, mostly, I suspect. Performance is probably not a large focus. While SMBfs support has been available on -STABLE through a port previously, determining that the integration with the base system (especially the boot process) was done correctly is important. Attempting to use SMBfs in /etc/fstab in a diskless environment might be one thing to explore, for example. (6) Once the man page change goes in (which I think it should) we'll want some basic testing of the man command. (7) cdboot. Late in the release cycle, a new implementation of the CD-based boot loader was introduced. This should generally improve support for booting or installing from CD, but this change requires testing on a variety of architectures and devices. The release notes will always be a good place to look for things to test. There are a number of new drivers, including if_em, which would probably benefit from more exposure. Please report bugs to the qa@FreeBSD.org list, and/or via send-pr with a heads up to the qa list. Robert N M Watson FreeBSD Core Team, TrustedBSD Project robert@fledge.watson.org NAI Labs, Safeport Network Services To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-qa" in the body of the message