Date: Tue, 23 Nov 1999 23:46:29 -0500 From: "Brian J. McGovern" <mcgovern@spoon.beta.com> To: stable@FreeBSD.ORG Subject: QA effort stalled in june due to lack of interest (was Re: speaking of 3.4) Message-ID: <199911240446.XAA21146@spoon.beta.com> In-Reply-To: Your message of "Tue, 23 Nov 1999 16:33:24 PST." <bulk.42082.19991123163324@hub.freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
After Usenix last year, I popped a few emails out and around asking if anyone was interested in doing OS QA, and what people thought should be checked during these QA runs. The response was terribly underwhelming. No one volunteered to do the work, and only a handful submitted ideas/code for test cases. I dropped the idea of trying to work out a QA plan so that I could spend my time working on applications, and pulling more users in to the FreeBSD fold. Even now, the latter has been far more useful than trying to do QA alone. I agree with David that you're not going to be able to freeze the tree for a month. But, what I consider a "freeze" and what David considers a freeze is probably different. What _might_ work is to either apply only bug fixes during the "release" cycle, or branch the release candidate, and apply bug fixes to try to get the release stabilized, and then merge them (the fixes) with -stable or -current (for 3.x and 4.x, respectively) if the fixes prove worth while. To a QA tester, however, the method for this is not relevant. What is relevant is that they're given a "fixed" version to continue testing. Therefore, I'll leave how to manage the source tree to the committers and the release engineer. The timeframe for the release testing is going to be a hot issue. Right now, it is 0 time. After all, no formal testing is done. Jordan does what he can, twiddles his thumbs getting no response, releases what appears to have "no" obvious bugs, then gets hammered as the user community installs the new release. The big question in my mind is why this happens. Now, even in my limited time, I will still grab one or two release candidates (RCs), and install them, just to see how much is fubar. A lot of times, I'm very surprised how much works, and its rare that I find serious bugs. However, usually my testing is limited to the basics, which is what I'm sure everyone else tests. Unfortunately, most places of business (including my own) claim that OS release testing is "Somebody Else's Problem", and doesn't fit in to the business' objectives and goals, and it isn't [your] responsibility to go test out the "bleeding edge" of OS technology. Unfortunately, if your business is going to make its living off the platform, it needs to be your responsibilty, even if its on one box in the back room running the applications you need it to run. Anyhow, I digress... Getting back on topic, what would people say to a 14 day release window? That would mean that Jordan will freeze code on 12/1, as mentioned in the thread, and a release will be forthcoming on or about 12/15. It could be later if there are serious problems found. But, I think this is a reasonable time to get some mileage on a release. Next question is who is going to do the work. This is the kicker. If we can find 6-12 people (and I'll be the first to volunteer) who are serious about getting some testing time in between 12/1 and 12/14, we might be able to scramble something together. By "serious", I mean that you have a machine or two to play with, and have a few hours daily, no less than every other day, to install a full new version, and bang the hell out of it. You should probably also consider the bandwidth you have to download releases. One or two may also have the luxury of installing a prior release, and then testing the "Upgrade" route. This means that a worst-case scenario, you have 42 machine/releases played with for a couple of hours each. The last hard task will be figuring out what/how to test. My first thought is to plow through the pr's that are either a.) closed since the 3.2-RELEASE, or b.) open. a.) gives you the chance to verify that the "fixes" really fixed things, and b.) lets you verify the bug really exists, and possibly give you a chance to debug it more fully. From this, we can hopefully pull a test set together. So, anyone up for actually doing the work? As mentioned, I want to start with 6-12 people, so the group is managable. I'm willing to coordinate, if Jordan is willing to plug in to the process, as well. After all, we'll need someone who is in the middle of the mess who knows who is responsible for what ;) I think if/when we can get things running 'smoothly' (and I use that word loosely), he can probably fall back a bit, but we'll need a jumpstart of information up front. Anyhow, if you're willing to basically give up the next three weeks of free time for your favorite OS, drop me a line, and we can start coordinating events. If this email flops as badly as the first, I'll summarize by Friday that QA is impossible, due to a lack of interest in a concerted effort. -Brian To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199911240446.XAA21146>