Date: Mon, 20 Dec 1999 20:57:14 -0600 From: Jason Young <doogie@staff.accessus.net> To: "'mistwolf@mushhaven.net'" <mistwolf@mushhaven.net>, freebsd-qa@FreeBSD.ORG Cc: "'jkh@freebsd.org'" <jkh@FreeBSD.ORG> Subject: RE: After 3.4 finally goes out the door Message-ID: <ABD44D466F85D311A69900A0C900DB6BC567@staff.accessus.net>
next in thread | raw e-mail | index | archive | help
This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01BF4B5F.18AB1FAC Content-Type: text/plain; charset="iso-8859-1" Well, I was trying to avoid the 'someone oughta' and 'you really should' type of rambling. :) I simply propose that if we could just gain some more time to QA the "real" release, then we'd get some truly useful work done. As for a standardized testing methodology or procedure? IMHO, that would be counterproductive. Everybody's got a different usage pattern, and different hardware, and trying to force people into the same mold will be hard and ultimately bad and wrong. For instance, if you had some script to follow in sysinstall - "test this, and then this, and then this, and test this" - would the sysinstall bugs have been found? If end-users with different hardware and configurations and usage patterns hadn't tested moused, would we have shipped with roller wheel support? No, it's people doing what they normally do, which is quite likely to be outside of what the developer tested. This is a Good Thing, and we need people doing their own unexpected, possibly weird, sometimes nonobvious stuff. The developers aren't total morons and the obvious stuff obviously works fine. The developers already have and already run the various regression tests that are available. A technically savvy QA volunteer might want to run those, and certainly that's fine, but I don't think we're short of dedicated and technical savvy developers familiar with the OS and its internals. What we need, IMHO, is a bunch of people putting it in real world situations, testing it under their various application and user loads, and seeing how it does. We had this in the form of the 3.4-RELEASE QA volunteer team, and everybody did a great job with the time and resources available. I hope that jkh and friends can see this, and see that just a little bit of foresight we can make sure that the next CD that ships holds a truly solid and kickass release. > -----Original Message----- > From: Jamie Norwood [mailto:mistwolf@mushhaven.net] > Sent: Monday, December 20, 1999 8:00 PM > To: Jason Young > Cc: 'Colin'; freebsd-qa@FreeBSD.ORG; 'jkh@freebsd.org' > Subject: Re: After 3.4 finally goes out the door > > > I agree with this totally. I also feel someone should step forward > and coordinate; I didn't do more than install and see if it worked > as I am not familier with how things should be tested. Also, I feel > that some things should be made available such as CVSUP scripts > and such to make sure the people involved really are testing the > right things. Could we maybe get a 'branch' named 'qa' that would show > rc or qa in the uname output? That would also make things > easier as we > could then test that tree for the cd, and bug fixes that are essential > could then be made to that tree only, and normal developement could > continue on the -STABLE/-RELEASE tree. > > Jamie > > > On Mon, Dec 20, 1999 at 07:50:12PM -0600, Jason Young wrote: > > > > I think it's a good idea to keep the list around, but I > don't think that the > > QA volunteers were really used to their full potential. > > > > I tested when I could (early in the testing cycle), and I > feel that most of > > my efforts were for naught, because as usual, a large > percentage of the > > commits in the testing period were very last minute (after > most or all of > > the QA builds). > > > > Yeah, it's a free software project and nobody's paid, and > what gets out the > > door is still a fine piece of work (and major kudos to jkh > and the rest of > > the guys on that). But, I don't think much meaningful > testing of the real > > release was accomplished due to all the really really late > changes. The last > > minute "oh I really need such and such MFC'd" trips are > killers (sysinstall, > > moused, you won't have to rack your brain too hard for examples). > > > > I assume that the schedule for 3.5-RELEASE is pretty much > set already. Would > > it be possible to back most of that schedule up about a > week, and in the > > week before the CD is cut, put the "pretty-much-final, > we-really-mean-it, > > won't touch it unless the release is definitely fscked" up > for general FTP? > > If the schedules are set well in advance, nobody will be hurting or > > complaining of the loss of one week's time in a three or > four month -STABLE > > release cycle. The QA volunteers need a set release to > assure its quality, > > ya know? This is no extra work or major change for anyone. > > > > I think that if we could get the "frozen" release candidate > out there in > > front of people for at least a week or so, we could muchly > reduce the size > > of our ERRATA files. > > > > > -----Original Message----- > > > From: Colin [mailto:cwass99@home.com] > > > Sent: Monday, December 20, 1999 7:18 PM > > > To: freebsd-qa@FreeBSD.ORG > > > Subject: After 3.4 finally goes out the door > > > > > > > > > I realize we got off to a rather slow start, but all > > > things considered > > > better than many expected. My question is, is there any > > > intent for this list > > > to survive the release of 3.4, as basically a home for > people with the > > > time/energy/hardware to talk about testing? > > > I think it would be a marvelous idea, assuming > > > sufficient interest of > > > course, to continue this with 4.0-RC when that starts the > > > long arduous path > > > from -CURRENT ;) > > > > > > Cheers, > > > Colin > > > > > > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > > > with "unsubscribe freebsd-qa" in the body of the message > > > > > > To Unsubscribe: send mail to majordomo@FreeBSD.org > with "unsubscribe freebsd-qa" in the body of the message > ------_=_NextPart_001_01BF4B5F.18AB1FAC Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN"> <HTML> <HEAD> <META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; = charset=3Diso-8859-1"> <META NAME=3D"Generator" CONTENT=3D"MS Exchange Server version = 5.5.2650.12"> <TITLE>RE: After 3.4 finally goes out the door</TITLE> </HEAD> <BODY> <BR> <P><FONT SIZE=3D2>Well, I was trying to avoid the 'someone oughta' and = 'you really should' type of rambling. :) I simply propose that if we = could just gain some more time to QA the "real" release, then = we'd get some truly useful work done.</FONT></P> <P><FONT SIZE=3D2>As for a standardized testing methodology or = procedure? IMHO, that would be counterproductive. Everybody's got a = different usage pattern, and different hardware, and trying to force = people into the same mold will be hard and ultimately bad and = wrong.</FONT></P> <P><FONT SIZE=3D2>For instance, if you had some script to follow in = sysinstall - "test this, and then this, and then this, and test = this" - would the sysinstall bugs have been found? If end-users = with different hardware and configurations and usage patterns hadn't = tested moused, would we have shipped with roller wheel support? No, = it's people doing what they normally do, which is quite likely to be = outside of what the developer tested. This is a Good Thing, and we need = people doing their own unexpected, possibly weird, sometimes nonobvious = stuff. The developers aren't total morons and the obvious stuff = obviously works fine.</FONT></P> <P><FONT SIZE=3D2>The developers already have and already run the = various regression tests that are available. A technically savvy QA = volunteer might want to run those, and certainly that's fine, but I = don't think we're short of dedicated and technical savvy developers = familiar with the OS and its internals. What we need, IMHO, is a bunch = of people putting it in real world situations, testing it under their = various application and user loads, and seeing how it does.</FONT></P> <P><FONT SIZE=3D2>We had this in the form of the 3.4-RELEASE QA = volunteer team, and everybody did a great job with the time and = resources available. I hope that jkh and friends can see this, and see = that just a little bit of foresight we can make sure that the next CD = that ships holds a truly solid and kickass release.</FONT></P> <P><FONT SIZE=3D2>> -----Original Message-----</FONT> <BR><FONT SIZE=3D2>> From: Jamie Norwood [<A = HREF=3D"mailto:mistwolf@mushhaven.net">mailto:mistwolf@mushhaven.net</A>= ]</FONT> <BR><FONT SIZE=3D2>> Sent: Monday, December 20, 1999 8:00 PM</FONT> <BR><FONT SIZE=3D2>> To: Jason Young</FONT> <BR><FONT SIZE=3D2>> Cc: 'Colin'; freebsd-qa@FreeBSD.ORG; = 'jkh@freebsd.org'</FONT> <BR><FONT SIZE=3D2>> Subject: Re: After 3.4 finally goes out the = door</FONT> <BR><FONT SIZE=3D2>> </FONT> <BR><FONT SIZE=3D2>> </FONT> <BR><FONT SIZE=3D2>> I agree with this totally. I also feel someone = should step forward</FONT> <BR><FONT SIZE=3D2>> and coordinate; I didn't do more than install = and see if it worked</FONT> <BR><FONT SIZE=3D2>> as I am not familier with how things should be = tested. Also, I feel</FONT> <BR><FONT SIZE=3D2>> that some things should be made available such = as CVSUP scripts</FONT> <BR><FONT SIZE=3D2>> and such to make sure the people involved = really are testing the</FONT> <BR><FONT SIZE=3D2>> right things. Could we maybe get a 'branch' = named 'qa' that would show</FONT> <BR><FONT SIZE=3D2>> rc or qa in the uname output? That would also = make things </FONT> <BR><FONT SIZE=3D2>> easier as we </FONT> <BR><FONT SIZE=3D2>> could then test that tree for the cd, and bug = fixes that are essential</FONT> <BR><FONT SIZE=3D2>> could then be made to that tree only, and = normal developement could</FONT> <BR><FONT SIZE=3D2>> continue on the -STABLE/-RELEASE tree.</FONT> <BR><FONT SIZE=3D2>> </FONT> <BR><FONT SIZE=3D2>> Jamie</FONT> <BR><FONT SIZE=3D2>> </FONT> <BR><FONT SIZE=3D2>> </FONT> <BR><FONT SIZE=3D2>> On Mon, Dec 20, 1999 at 07:50:12PM -0600, Jason = Young wrote:</FONT> <BR><FONT SIZE=3D2>> > </FONT> <BR><FONT SIZE=3D2>> > I think it's a good idea to keep the list = around, but I </FONT> <BR><FONT SIZE=3D2>> don't think that the</FONT> <BR><FONT SIZE=3D2>> > QA volunteers were really used to their = full potential.</FONT> <BR><FONT SIZE=3D2>> > </FONT> <BR><FONT SIZE=3D2>> > I tested when I could (early in the = testing cycle), and I </FONT> <BR><FONT SIZE=3D2>> feel that most of</FONT> <BR><FONT SIZE=3D2>> > my efforts were for naught, because as = usual, a large </FONT> <BR><FONT SIZE=3D2>> percentage of the</FONT> <BR><FONT SIZE=3D2>> > commits in the testing period were very = last minute (after </FONT> <BR><FONT SIZE=3D2>> most or all of</FONT> <BR><FONT SIZE=3D2>> > the QA builds).</FONT> <BR><FONT SIZE=3D2>> > </FONT> <BR><FONT SIZE=3D2>> > Yeah, it's a free software project and = nobody's paid, and </FONT> <BR><FONT SIZE=3D2>> what gets out the</FONT> <BR><FONT SIZE=3D2>> > door is still a fine piece of work (and = major kudos to jkh </FONT> <BR><FONT SIZE=3D2>> and the rest of</FONT> <BR><FONT SIZE=3D2>> > the guys on that). But, I don't think much = meaningful </FONT> <BR><FONT SIZE=3D2>> testing of the real</FONT> <BR><FONT SIZE=3D2>> > release was accomplished due to all the = really really late </FONT> <BR><FONT SIZE=3D2>> changes. The last</FONT> <BR><FONT SIZE=3D2>> > minute "oh I really need such and = such MFC'd" trips are </FONT> <BR><FONT SIZE=3D2>> killers (sysinstall,</FONT> <BR><FONT SIZE=3D2>> > moused, you won't have to rack your brain = too hard for examples).</FONT> <BR><FONT SIZE=3D2>> > </FONT> <BR><FONT SIZE=3D2>> > I assume that the schedule for 3.5-RELEASE = is pretty much </FONT> <BR><FONT SIZE=3D2>> set already. Would</FONT> <BR><FONT SIZE=3D2>> > it be possible to back most of that = schedule up about a </FONT> <BR><FONT SIZE=3D2>> week, and in the</FONT> <BR><FONT SIZE=3D2>> > week before the CD is cut, put the = "pretty-much-final, </FONT> <BR><FONT SIZE=3D2>> we-really-mean-it,</FONT> <BR><FONT SIZE=3D2>> > won't touch it unless the release is = definitely fscked" up </FONT> <BR><FONT SIZE=3D2>> for general FTP?</FONT> <BR><FONT SIZE=3D2>> > If the schedules are set well in advance, = nobody will be hurting or</FONT> <BR><FONT SIZE=3D2>> > complaining of the loss of one week's time = in a three or </FONT> <BR><FONT SIZE=3D2>> four month -STABLE</FONT> <BR><FONT SIZE=3D2>> > release cycle. The QA volunteers need a = set release to </FONT> <BR><FONT SIZE=3D2>> assure its quality,</FONT> <BR><FONT SIZE=3D2>> > ya know? This is no extra work or major = change for anyone.</FONT> <BR><FONT SIZE=3D2>> > </FONT> <BR><FONT SIZE=3D2>> > I think that if we could get the = "frozen" release candidate </FONT> <BR><FONT SIZE=3D2>> out there in</FONT> <BR><FONT SIZE=3D2>> > front of people for at least a week or so, = we could muchly </FONT> <BR><FONT SIZE=3D2>> reduce the size</FONT> <BR><FONT SIZE=3D2>> > of our ERRATA files.</FONT> <BR><FONT SIZE=3D2>> > </FONT> <BR><FONT SIZE=3D2>> > > -----Original Message-----</FONT> <BR><FONT SIZE=3D2>> > > From: Colin [<A = HREF=3D"mailto:cwass99@home.com">mailto:cwass99@home.com</A>]</FONT> <BR><FONT SIZE=3D2>> > > Sent: Monday, December 20, 1999 7:18 = PM</FONT> <BR><FONT SIZE=3D2>> > > To: freebsd-qa@FreeBSD.ORG</FONT> <BR><FONT SIZE=3D2>> > > Subject: After 3.4 finally goes out = the door</FONT> <BR><FONT SIZE=3D2>> > > </FONT> <BR><FONT SIZE=3D2>> > > </FONT> <BR><FONT SIZE=3D2>> > > I = realize we got off to a rather slow start, but all </FONT> <BR><FONT SIZE=3D2>> > > things considered</FONT> <BR><FONT SIZE=3D2>> > > better than many expected. My = question is, is there any </FONT> <BR><FONT SIZE=3D2>> > > intent for this list</FONT> <BR><FONT SIZE=3D2>> > > to survive the release of 3.4, as = basically a home for </FONT> <BR><FONT SIZE=3D2>> people with the</FONT> <BR><FONT SIZE=3D2>> > > time/energy/hardware to talk about = testing?</FONT> <BR><FONT SIZE=3D2>> > > I think = it would be a marvelous idea, assuming </FONT> <BR><FONT SIZE=3D2>> > > sufficient interest of</FONT> <BR><FONT SIZE=3D2>> > > course, to continue this with 4.0-RC = when that starts the </FONT> <BR><FONT SIZE=3D2>> > > long arduous path</FONT> <BR><FONT SIZE=3D2>> > > from -CURRENT ;)</FONT> <BR><FONT SIZE=3D2>> > > </FONT> <BR><FONT SIZE=3D2>> > > Cheers,</FONT> <BR><FONT SIZE=3D2>> > > Colin</FONT> <BR><FONT SIZE=3D2>> > > </FONT> <BR><FONT SIZE=3D2>> > > </FONT> <BR><FONT SIZE=3D2>> > > To Unsubscribe: send mail to = majordomo@FreeBSD.org</FONT> <BR><FONT SIZE=3D2>> > > with "unsubscribe = freebsd-qa" in the body of the message</FONT> <BR><FONT SIZE=3D2>> > > </FONT> <BR><FONT SIZE=3D2>> </FONT> <BR><FONT SIZE=3D2>> </FONT> <BR><FONT SIZE=3D2>> To Unsubscribe: send mail to = majordomo@FreeBSD.org</FONT> <BR><FONT SIZE=3D2>> with "unsubscribe freebsd-qa" in the = body of the message</FONT> <BR><FONT SIZE=3D2>> </FONT> </P> </BODY> </HTML> ------_=_NextPart_001_01BF4B5F.18AB1FAC-- To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-qa" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ABD44D466F85D311A69900A0C900DB6BC567>