Date: Mon, 25 Jun 2001 00:36:44 -0600 From: Mike Porter <mupi@mknet.org> To: Chris BeHanna <behanna@zbzoom.net>, FreeBSD-Stable <stable@FreeBSD.ORG> Subject: Re: Staying *really stable* in FreeBSD Message-ID: <01062500364404.80326@mukappa.home.com> In-Reply-To: <Pine.BSF.4.32.0106242057320.18238-100000@topperwein.dyndns.org> References: <Pine.BSF.4.32.0106242057320.18238-100000@topperwein.dyndns.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sunday 24 June 2001 19:20, Chris BeHanna wrote:
Having recently stumbled across a library copy of "The Software Conspiracy"
(if you are a contributor, I would have to urge you in the strongest possible
terms to get your hands on a copy of that book and read it, if you haven't
already....it contains a lot of pretty valuable information IMHO for the
"rest" of us, too....). I really really like the idea of CMM level 5 stuff,
but I am not sure it is entirely practical in a volunteer situation, as there
are no "managers" to track the paperwork side, and keeping tabs on stuff as
described below would be pretty hideous.
That said, I would imagine that there are a number of people like me, who
have a vague understanding of the basic ideas, but not much practical
experience, and worse, very little time to learn. I can keep my system
running smoothly, and pretty much self handle most any issue that arises
(with the help of the handbook and/or brute forcing my way logically through
the problem), but I have no real desire to penetrate much deeper than what I
need to know for troubleshooting and fixing stuff ('man ....' <(}:).
However, the idea of some consistent procedures for testing, bug fixing,
estimating project completion dates, and a way to document and track such
activity is extremely appealing, and the sort of thing that I could do
easily. I have a hard time believing on a list this size that I am the only
one in that situation.
It would be most amusing to see the M$ guys eat thier shorts trying to
explain how "open source" (did somebody say "potentially viral software"??
damn, I must be hearing things again....) software produced by volunteers can
be CMM 5 certified, but their expensive fancy whatever OS + service pack of
the hour can't....
The balance would have to be very carefully drawn however, as the people such
as myself who might volunteer in these positions would be doing functions
traditionally carried out by "Management" in CMM enterprises, I would
certainly not want to be seen that way (how can yuo "manage" a volunteer
anyway, really?)
I guess if we take the ideas expressed byt he author of "the software
consipiracy" to their logical step, what we would wind up with is three
groups of volunteers: the "genius hackers" working on new features and stuff
in current, the "army of 'average' programmers" working primarily on bug
fixes, fleshing out the ideas proposed by the "hacker" core, and the
"invisibles", tracking the progress of the two groups, perhaps assigning
projects to the "average" group based on past experience with that persons
work (ie how many bugs/line of code, how accurate time estimates are, etc).
If he is right, such a structured environment would produce better code,
faster, and allow more people to participate. I guess in function the
current system isn't too different from this, with committers and "core"
group assigning projects and doing a lot of the "dream" work in current ( and
though I run stable not current, -stable was at some point in the past a
current, too). The biggest difference would be the addition of specific
documentation procedures to track stuff in a more concrete manner. Or am I
missing something here?
If this scenario were to be implemented, then the "invisibles" would have a
pretty good grasp at any given time of the actual relative stability of
stable (and current, too for that matter), although of course, as mentioned
previously in this thread complex software and hardware ineteractions can be,
well, complex, and the only guarantee of software reliability would be to
actually test on the same hardware. (even then, such minor details as a
flakey few bits in a SDRAM chip could cause wierd things to happen on one
machine that don't happen on another, but those sorts of things, in general,
are becoming less frequent than they used to be. ( i guess there may never be
a cure for the "it works OK in THIS PCI/DIMM slot, but if I move it to THAT
slot, it fails. But this OTHER card works OK in either slot....but hey we
can dream can't we).
Some purely theoretical side effects of this idea would be that 1) the code
freeze at -RELEASE could most likely be shortened (if there are fewer bugs to
start with, there is less need for testing, right?) and 2) spinning off an
interim release at any given point (or traking stable) becomes much less
risky. and 3) the people in the "invisibles" category would know about bugs
pretty quickly in most cases and would be good people to contact if you were
unsure about the state of the code at any given arbitrary point in time. And
cna't CVS track to a specific date, anyway, so you could "roll back" to a
more stable period? as a side effect of 2) and 3) it becomes more practical
to attempt binary releases (though no guarantee, look at sysinstalls success
rate installing binaries compared to the ports trees....face it, FreeBSD as
an OS is pretty much designed to be built from scratch to update it; if you
want binary patches and the like, go to BSDi or somebody. But it occurs to
me that if we decrease BSDI's workload, we just might find additional
funding, too, and that wouldn't be a bad thing, would it?
Speaking of half-baked ramblings, I better stop now...
mike
>
> I'm just thinking out loud here; none of what follows should be
> read as "Somebody *should* do this." By way of background, I worked
> on the SVR 4.2 MIPS port back in 1992, and was involved in some detail
> with what is done to verify the quality of a commercial UNIX operating
> system release. I've worked in the commercial software field ever
> since, including for the world's major source of telecomm software,
> and I've lived firsthand the CMM Level 5 lifestyle. I like FreeBSD,
> and if things continue as they have been, I'll continue using it,
> tracking the list and cvsup'ing at what appear to be opportune times,
> tar-ing off my last good build beforehand, and saving my last known
> good kernels. I am, to my great chagrin, NOT an experienced kernel
> hacker, and I haven't spent much time in the depths of libc (apart
> from the SYSV rtld code). Almost all of my experience is in
> user-level code.
>
> That said:
>
> I don't have the time to write an entire "BVS" ("BSD Verification
> Suite"), nor to keep it up-to-date; however, I'd be willing to
> contribute (and maintain) pieces here and there if a test framework
> and/or harness (read: API/coding convention) were put into place.
> There's DejaGNU, but it suffers from the GPL. eTET might be a
> possibility. I'll be happy to investigate this in my (limited) spare
> time if Jordan (or whomever else) is willing to help me find a place
> to put it (be kind! :-).
>
> Ideally, the authors of a given piece of FreeBSD would write and
> maintain the code that tested their piece, and would add tests to
> reproduce bugs as they attacked those bugs. Inevitably, some parts of
> the test suite would lag behind, and would emit spurious failure
> reports. The code freeze on FreeBSD proper prior to a release might
> be a good time to update those parts of the test suite.
>
> Once an (up-to-date) test suite is in place, regression will help
> to verify that -STABLE really is stable--but again, only on the
> platform and for the configuration on which the test suite is run. A
> list of volunteers could submit their results leading up to a release,
> and that list could be posted ("FreeBSD m.n has been verified to run
> on such-and-so hardware with such-and-so configuration; deviate at
> your own risk....").
>
> A similar service could be provided (again by volunteers) for
> given snapshots of -STABLE, and for testing of binary patches.
>
> Tracking of the rates of bugs of varying severities is as simple
> as a cron job firing off a script, but someone has to have the time to
> write it. For commercial purposes, aging of bugs could also be
> tracked, although, given that this is a volunteer effort, that might
> put undue pressure on committers, and might make them quit if people
> hound them ceaselessly to do more than they have time to do. (Heck, I
> have bugs in my own for-pay job that are older than I want them to
> be, yet no one there would remotely consider characterizing me as a
> slacker.)
>
> OK, no more half-baked ramblings (for now).
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?01062500364404.80326>