From owner-freebsd-stable Sun Jun 24 23:36:53 2001 Delivered-To: freebsd-stable@freebsd.org Received: from c576194-a.saltlk1.ut.home.com (c576194-a.saltlk1.ut.home.com [65.5.60.246]) by hub.freebsd.org (Postfix) with SMTP id 7BACB37B405 for ; Sun, 24 Jun 2001 23:36:44 -0700 (PDT) (envelope-from mupi@mknet.org) Received: (qmail 85754 invoked from network); 25 Jun 2001 06:36:44 -0000 Received: from localhost (HELO mukappa.home.com) (ovsjiv@127.0.0.1) by localhost with SMTP; 25 Jun 2001 06:36:44 -0000 From: Mike Porter Reply-To: mupi@Mknet.org To: Chris BeHanna , FreeBSD-Stable Subject: Re: Staying *really stable* in FreeBSD Date: Mon, 25 Jun 2001 00:36:44 -0600 X-Mailer: KMail [version 1.1.99] Content-Type: text/plain; charset="US-ASCII" References: In-Reply-To: MIME-Version: 1.0 Message-Id: <01062500364404.80326@mukappa.home.com> Content-Transfer-Encoding: 8bit Sender: owner-freebsd-stable@FreeBSD.ORG Precedence: bulk List-ID: List-Archive: (Web Archive) List-Help: (List Instructions) List-Subscribe: List-Unsubscribe: X-Loop: FreeBSD.ORG On Sunday 24 June 2001 19:20, Chris BeHanna wrote: Having recently stumbled across a library copy of "The Software Conspiracy" (if you are a contributor, I would have to urge you in the strongest possible terms to get your hands on a copy of that book and read it, if you haven't already....it contains a lot of pretty valuable information IMHO for the "rest" of us, too....). I really really like the idea of CMM level 5 stuff, but I am not sure it is entirely practical in a volunteer situation, as there are no "managers" to track the paperwork side, and keeping tabs on stuff as described below would be pretty hideous. That said, I would imagine that there are a number of people like me, who have a vague understanding of the basic ideas, but not much practical experience, and worse, very little time to learn. I can keep my system running smoothly, and pretty much self handle most any issue that arises (with the help of the handbook and/or brute forcing my way logically through the problem), but I have no real desire to penetrate much deeper than what I need to know for troubleshooting and fixing stuff ('man ....' <(}:). However, the idea of some consistent procedures for testing, bug fixing, estimating project completion dates, and a way to document and track such activity is extremely appealing, and the sort of thing that I could do easily. I have a hard time believing on a list this size that I am the only one in that situation. It would be most amusing to see the M$ guys eat thier shorts trying to explain how "open source" (did somebody say "potentially viral software"?? damn, I must be hearing things again....) software produced by volunteers can be CMM 5 certified, but their expensive fancy whatever OS + service pack of the hour can't.... The balance would have to be very carefully drawn however, as the people such as myself who might volunteer in these positions would be doing functions traditionally carried out by "Management" in CMM enterprises, I would certainly not want to be seen that way (how can yuo "manage" a volunteer anyway, really?) I guess if we take the ideas expressed byt he author of "the software consipiracy" to their logical step, what we would wind up with is three groups of volunteers: the "genius hackers" working on new features and stuff in current, the "army of 'average' programmers" working primarily on bug fixes, fleshing out the ideas proposed by the "hacker" core, and the "invisibles", tracking the progress of the two groups, perhaps assigning projects to the "average" group based on past experience with that persons work (ie how many bugs/line of code, how accurate time estimates are, etc). If he is right, such a structured environment would produce better code, faster, and allow more people to participate. I guess in function the current system isn't too different from this, with committers and "core" group assigning projects and doing a lot of the "dream" work in current ( and though I run stable not current, -stable was at some point in the past a current, too). The biggest difference would be the addition of specific documentation procedures to track stuff in a more concrete manner. Or am I missing something here? If this scenario were to be implemented, then the "invisibles" would have a pretty good grasp at any given time of the actual relative stability of stable (and current, too for that matter), although of course, as mentioned previously in this thread complex software and hardware ineteractions can be, well, complex, and the only guarantee of software reliability would be to actually test on the same hardware. (even then, such minor details as a flakey few bits in a SDRAM chip could cause wierd things to happen on one machine that don't happen on another, but those sorts of things, in general, are becoming less frequent than they used to be. ( i guess there may never be a cure for the "it works OK in THIS PCI/DIMM slot, but if I move it to THAT slot, it fails. But this OTHER card works OK in either slot....but hey we can dream can't we). Some purely theoretical side effects of this idea would be that 1) the code freeze at -RELEASE could most likely be shortened (if there are fewer bugs to start with, there is less need for testing, right?) and 2) spinning off an interim release at any given point (or traking stable) becomes much less risky. and 3) the people in the "invisibles" category would know about bugs pretty quickly in most cases and would be good people to contact if you were unsure about the state of the code at any given arbitrary point in time. And cna't CVS track to a specific date, anyway, so you could "roll back" to a more stable period? as a side effect of 2) and 3) it becomes more practical to attempt binary releases (though no guarantee, look at sysinstalls success rate installing binaries compared to the ports trees....face it, FreeBSD as an OS is pretty much designed to be built from scratch to update it; if you want binary patches and the like, go to BSDi or somebody. But it occurs to me that if we decrease BSDI's workload, we just might find additional funding, too, and that wouldn't be a bad thing, would it? Speaking of half-baked ramblings, I better stop now... mike > > I'm just thinking out loud here; none of what follows should be > read as "Somebody *should* do this." By way of background, I worked > on the SVR 4.2 MIPS port back in 1992, and was involved in some detail > with what is done to verify the quality of a commercial UNIX operating > system release. I've worked in the commercial software field ever > since, including for the world's major source of telecomm software, > and I've lived firsthand the CMM Level 5 lifestyle. I like FreeBSD, > and if things continue as they have been, I'll continue using it, > tracking the list and cvsup'ing at what appear to be opportune times, > tar-ing off my last good build beforehand, and saving my last known > good kernels. I am, to my great chagrin, NOT an experienced kernel > hacker, and I haven't spent much time in the depths of libc (apart > from the SYSV rtld code). Almost all of my experience is in > user-level code. > > That said: > > I don't have the time to write an entire "BVS" ("BSD Verification > Suite"), nor to keep it up-to-date; however, I'd be willing to > contribute (and maintain) pieces here and there if a test framework > and/or harness (read: API/coding convention) were put into place. > There's DejaGNU, but it suffers from the GPL. eTET might be a > possibility. I'll be happy to investigate this in my (limited) spare > time if Jordan (or whomever else) is willing to help me find a place > to put it (be kind! :-). > > Ideally, the authors of a given piece of FreeBSD would write and > maintain the code that tested their piece, and would add tests to > reproduce bugs as they attacked those bugs. Inevitably, some parts of > the test suite would lag behind, and would emit spurious failure > reports. The code freeze on FreeBSD proper prior to a release might > be a good time to update those parts of the test suite. > > Once an (up-to-date) test suite is in place, regression will help > to verify that -STABLE really is stable--but again, only on the > platform and for the configuration on which the test suite is run. A > list of volunteers could submit their results leading up to a release, > and that list could be posted ("FreeBSD m.n has been verified to run > on such-and-so hardware with such-and-so configuration; deviate at > your own risk...."). > > A similar service could be provided (again by volunteers) for > given snapshots of -STABLE, and for testing of binary patches. > > Tracking of the rates of bugs of varying severities is as simple > as a cron job firing off a script, but someone has to have the time to > write it. For commercial purposes, aging of bugs could also be > tracked, although, given that this is a volunteer effort, that might > put undue pressure on committers, and might make them quit if people > hound them ceaselessly to do more than they have time to do. (Heck, I > have bugs in my own for-pay job that are older than I want them to > be, yet no one there would remotely consider characterizing me as a > slacker.) > > OK, no more half-baked ramblings (for now). To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-stable" in the body of the message