Date: Tue, 19 Mar 1996 20:27:15 +1100 From: "Mark Cheeseman" <cheese@asstdc.com.au> To: hackers@freebsd.org Cc: michael butler <imb@scgt.oz.au> Subject: Re: Comparing FreeBSD and other OSs (fwd) Message-ID: <199603191027.UAA20223@ford.apnpc.com.au>
next in thread | raw e-mail | index | archive | help
Hi Folks, 'scuze me jumping in here. This message was forwarded to me, and I thought I might add something to the discussion. I don't actually subscribe to the list, but I do run FreeBSD and have some experience with magazine reviews and comparisons > > > We contact a number of manufacturers and computer magazines, > > > world-wide, and set up a benchmark web server machine. Every day, > > > we change the operating system, but we don't tell anybody what the > > > OS de jour is. We invite everybody on the web to try to access it, > > > and to tell us what they think of the perceived response time. At > > > the same time, we monitor the performance (and, of course, down time > > > if the be any). > > > > > > This would have the advantage of being a benchmark in the public eye, > > > and one in which the BSD OSs would have a clear lead. Any thoughts? The first problem I see is that the Net is too unpredictable (wrt transit times and loading) to be usable as a test-bed. Even if you could factor in the variance of packet loss and transit times (which obviously impact perceived performance), there's still the issue of server load. You can't expect everybody to check a URL every day, at the same time every day, and do the same things on the site every day. People are unpredictable, and therefore so would the results be. So you'd need a well traffic'ed site on a fast link in order to really show up the differences. I don't think yahoo or playboy.com would want to risk losing traffic on such an experiment :-) Plus, you're relying on the subjective assessment of unknown people, who will probably themselves face differing levels of congestion on their local links. In order to collate the results you have to quantify these results in some say, and you'll never be able to satisfy your audience that the results are meaningful. That said, I'd love to see a test of this sort done, and If I ever get the time I'll probably do it myself. But it needs to be done in a controlled environment, on an isolated LAN with something like WebStone. That way, you have quantifiable results, independent of any outside influence such as Net traffic and the mood that somebody happens to be in on that particular day. > > I think this would be far more valuable if somebody else "thought of it", > > a magazine for instance, since they are far less likely to be accused > > of rigging the demo. There's a few commercial reailties that make it unlikely that a magazine would conduct such a comparison, but I won't go into them here. -- Mark Cheeseman, Technical Manager, APN Computing Group Pty Ltd PO Box 1287, North Sydney 2059, AU. http://www.apnpc.com.au/ cheese@asstdc.com.au Tel +61 2 9936 8689 Fax +61 2 9955 8871
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199603191027.UAA20223>