From owner-freebsd-hackers Thu Dec 21 12:15: 0 2000 From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 21 12:14:57 2000 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from chopper.Poohsticks.ORG (chopper.poohsticks.org [63.227.60.73]) by hub.freebsd.org (Postfix) with ESMTP id 8EF3D37B400 for ; Thu, 21 Dec 2000 12:14:56 -0800 (PST) Received: from chopper.Poohsticks.ORG (drew@localhost.poohsticks.org [127.0.0.1]) by chopper.Poohsticks.ORG (8.10.1/8.10.1) with ESMTP id eBLKEfh12072; Thu, 21 Dec 2000 13:14:41 -0700 Message-Id: <200012212014.eBLKEfh12072@chopper.Poohsticks.ORG> To: "SteveB" Cc: freebsd-hackers@FreeBSD.ORG Subject: Re: Sitting on hands (no longer Re: FreeBSD vs Linux, Solaris, and NT) In-reply-to: Your message of "Thu, 21 Dec 2000 09:48:44 PST." MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <12068.977429681.1@chopper.Poohsticks.ORG> Date: Thu, 21 Dec 2000 13:14:41 -0700 From: Drew Eckhardt Sender: drew@chopper.Poohsticks.ORG Sender: owner-freebsd-hackers@FreeBSD.ORG Precedence: bulk X-Loop: FreeBSD.ORG In message , admin@bsdfan .cncdsl.com writes: >Here's the thing about open software that still concerns me. My >background is with the major software development tools companies, so >that is my point of reference. It is great that code is available and >fixes are made and pushed out, but who is doing real testing of these >fixes. Sure the obvious problem is fixed, but what other problems has >it uncovered, what side effect has it created, and how about >compatibility with other software or drivers in this case. > >With commercial software (well at least the places I worked) nothing >could go out the door without a complete QA cycle performed on it. In a past life, I did half the design and implementation of the software tracking calls and letting the billing software know about them on a CDMA cellular base station. For hardware, we used machines from the biggest workstation vendor with a three letter name, running the latest production release of their Unix. Before booting the putz from our team who'd crippled our software with threads and excised the damage he'd done, we regularly dumped the machines out to the ROM monitor. I know people who work in several operating systems groups at that company, know a bit about their quality control process, and know that it was insufficient. I've yet to encounter a bug of that severity in any released version of free software (about the worst which wasn't hardware related is the FreeBSD Tulip driver not working correctly in full-duplex 100baseT mode). >So who is testing these fixes in open source world? Cygnus is/was doing automated regression testing on GCC. >Just seeing if >the problem at hand is gone isn't real testing, even claiming >thousands of people are now using it isn't enough. In theory, a standard suite of white and black box tests should be superior. Given inumerable bad experiences with Adobe, IBM, HP, Microsoft, Sun and other smaller companies, in practice it doesn't seem to work any better than the million-monkeys approach. >QA is unglamorous work, but needs to be done. Does this mean you're volunteering? To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-hackers" in the body of the message