Date: Fri, 21 Dec 2001 22:29:09 -0500 (EST) From: Robert Watson <rwatson@FreeBSD.org> To: stable@FreeBSD.org, qa@FreeBSD.org Cc: re@FreeBSD.org Subject: Testing guide for 4.5-PRERELEASE Message-ID: <Pine.NEB.3.96L.1011221222530.74838A-100000@fledge.watson.org>
next in thread | raw e-mail | index | archive | help
Testing Guide for 4.5-RELEASE
As part of our on-going effort to improve the release engineering process,
Murray has asked me to post my guide to things that Real Need Testing in
the release candidate phase. Below, I describe changes in 4.5-PRERELEASE
that we feel merit the most attention due to their involving substantial
changes to the system, or having arrived late in the development cycle
leading up to the release. In general, our goal in the QA process is to
attempt to check a number of things:
(1) The system has not regressed with respects to stability, correctness,
interoperability, or performance of features present in prior
releases.
(2) New features result in the desired improvement in stability,
correctness, interoperability, or performance.
To effectively determine this, it's desirable to test the system in a
diverse set of environments, applying a wide set of workloads, forcing the
system to operate both within and outside its normal specification.
Particular focus should often be placed on the continuing (or new)
capability of the system to perform correctly when used in concert with
systems from other vendors.
Features to explore carefully:
(1) Recent TCP changes, especially relating to the delayed ACK fix,
congestion response, syncache, syncookies, increased socket buffer
sizes, et al. We're interested in testing interoperability with as
many platforms as possible, demonstrating continued strong (and
better) scalability and performance, and watching out for quirks
(connection stalls, ...), not to mention crashes. Jonathan Lemon was
responding to a panic report on freebsd-current earlier today
regarding a PCB call, which is something we should keep an eye on. On
the other hand, Y! is now deploying this code, and that should help
test it a great deal.
(2) VFS/VM/NFS fixes. We need to continue to test performance,
correctness, and interoperability. In particular, I'd like to see a
lot of inter-platform performance testing (FreeBSD->Solaris, vice
versa, etc). We'd also like careful investigation of low-memory
situations.
(3) FFS fixes. We had some reports of deadlocks in FFS; it sounds like
Matt Dillon has caught most of them, but combinations I'd particular
like to see tested involve Quotas, Chroot, and NFS, under load, and
involving memory mapping and heavy directory operations.
(4) NTP 4.1. This is probably reasonable safe, but it doesn't hurt to do
interop testing, especially on Alpha.
(5) SMBfs. We need stability testing, mostly, I suspect. Performance is
probably not a large focus. While SMBfs support has been available on
-STABLE through a port previously, determining that the integration
with the base system (especially the boot process) was done correctly
is important. Attempting to use SMBfs in /etc/fstab in a diskless
environment might be one thing to explore, for example.
(6) Once the man page change goes in (which I think it should) we'll want
some basic testing of the man command.
(7) cdboot. Late in the release cycle, a new implementation of the
CD-based boot loader was introduced. This should generally improve
support for booting or installing from CD, but this change requires
testing on a variety of architectures and devices.
The release notes will always be a good place to look for things to test.
There are a number of new drivers, including if_em, which would probably
benefit from more exposure. Please report bugs to the qa@FreeBSD.org
list, and/or via send-pr with a heads up to the qa list.
Robert N M Watson FreeBSD Core Team, TrustedBSD Project
robert@fledge.watson.org NAI Labs, Safeport Network Services
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.NEB.3.96L.1011221222530.74838A-100000>