From owner-freebsd-current@FreeBSD.ORG Sun Feb 6 14:36:41 2005 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3D79316A4CE; Sun, 6 Feb 2005 14:36:41 +0000 (GMT) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9A14343D45; Sun, 6 Feb 2005 14:36:40 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with SMTP id B794F46B14; Sun, 6 Feb 2005 09:36:39 -0500 (EST) Date: Sun, 6 Feb 2005 14:35:44 +0000 (GMT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Scott Long In-Reply-To: <4205F382.8020404@freebsd.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-current@freebsd.org Subject: Re: The case for FreeBSD X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Feb 2005 14:36:41 -0000 On Sun, 6 Feb 2005, Scott Long wrote: > There has been a lot of recent talk and advocacy for NetBSD 2.0 from the > NetBSD team. Most recently there were a series of articles posted my > Chritos Zoulas describing why NetBSD is relevant and why it's a better > choice than either FreeBSD or OpenBSD. While I strongly applaud the > accomplishments of the NetBSD team and happily agree that NetBSD 2.0 is > a strong step forward for them, I take a bit of exception to many of > their claims and much of their criticisms of FreeBSD. I think the technical arguments for FreeBSD in your e-mail are excellent -- we a number of really incredible features that make FreeBSD both an excellent server platform, and an excellent platform to build into other products. These features set us clearly ahead in some very important areas. However, I'm sure Christos's report was not centered on bashing us. I noticed a lot of "We got X from FreeBSD" and "We should get X from FreeBSD". As you might expect from an annual report, it was very introspective -- and appropriately so -- and it spoke to strategies for improving NetBSD. And the observation that code moves in both directions is important: NetBSD has picked up some neat stuff from us in the past few years, including KQueue, UFS2, OpenPAM, etc. Because we work hard and develop good features, they get to reap the benefit (which is part of the BSD model). We've also picked up some neat stuff, including support in developing new hardwre platforms, some of the micro-benchmark optimizations, etc. Hopefully, we've both been good about crediting where these things come from. And you can see that strategy in the plans for the future in both camps as well -- NetBSD's plans to pick up SACK from us are just one example. If I were them, I'd also start grabbing some of our SMP infrastructure -- if not the overall locking strategy, certainly the abstractions. I agree that the "NetBSD is more scalable than FreeBSD" conclusion stemming from a previously posted set of benchmarks was bogus: scalability should be measured as a property of real-world applications, not the constant in the file descriptor allocation routine. I.e., if you really want to measure scalability, run a web benchmark or a database benchmark. If NetBSD still comes out faster in a properly run macro-benchmark, then that's an important thing to know, and an important target -- microbenchmarks are nice to optimize for, but the big picture is more important, and not mentioned in the benchmark paper. Everything was fine in the paper, with the exception of the abstract/conclusion, which had one of those "then a miracle occurs" moments in reasoning. :-) So I wouldn't dismiss the technical results, but the "headline" was pretty out of line with regard to the technical results -- this follows in an increasingly long history of weird open source "scalability" benchmarks, in which a set of microbenchmarks is used to justify an overall argument that isn't supported by the technical content. By reaching less far, the argument would become more accurate. As it turns out, it's a lot easier to run micro-benchmarks properly than macro-benchmarks properly, which is probably why micro-benchmarks predominate. I think it is the case that lately the NetBSD folk have been doing a more effective PR job than they have been previously, and have managed to grab some headlines as a result. Our response, in as much is one is needed, should be three-fold: where NetBSD can demonstrate a quantitative performance of feature benefit over FreeBSD, we should figure out why, and fix it where it makes sense and is important. We should do some comprehensive benchmarking of our own, and use them to improve our system as well as document its benefits. The NetBSD benchmark was a classic example of micro-benchmarks feeding performance optimization, and vice-versa -- not without benefit, but true none-the-less. However, our primary objections to it haven't been that we may not sometimes not win every micro-benchmark, but that it missed what's really important in scalability -- scalable application performance on the platform. There are some important ways that out SMPng/KSE architecture should (and do) perform really well: for example, our threading library runs threads on more than one CPU at a time by default (which the NetBSD SA implementation currently doesn't), and we can excute parallel system calls in kernel, which should be quite measurable (and I've seen results suggesting that it really is). However, because we took such a cut in performance early in 5.x development, we've not been doing continuous or effective long-term benchmarking. I have a nice history of MySQL results improving over the last year -- basically doubling or more in performance as we improved, especially on SMP -- but that's hardly comprehensive. We've built excellent project infrastructure for build management, packaging, etc, but we need to work more effectively as a project, not just as individuals, to manage performance the same way. We should do a better PR job of identifying areas of particular strength -- your list of features would make the great beginnings of a white paper on FreeBSD network stack technology, for example. Many of the features there have set the standard for implementation in other systems -- kqueue is now widely used across many platforms, and those that don't have kqueue try to immitate with a bit of NIH on the side :-). Finally, we should continue our heavy investment of development (and other) resources into the FreeBSD project -- we're a very large project with a very large number of extremely competent developers. The BSD platforms have defined operating system technology for many years, and in many ways continue to do so. We should make sure we continue to do that. Projects like SMPng, KSE, etc, have come an incredibly long way in the past few years, and reflect both the huge challenges they involve, and the level of investment we've made. Commercial UNIX vendors such as Sun easily spent far more in the way of resources to bring these features to Solaris, and probably took even longer than we did to bring them to fruition. We have the advantages of a very open development process, and we take advantage of them. While there's more work to do, we can actually see the finish line on these projects. With recent work by Jeff on VFS/UFS, we're reaching the point where a significant percentage of our kernel is fully threaded, parallelized, and able to run preemptively. Now that the benefits are starting to pay off is the time we can reap the best benefits from careful work -- i.e., we should now be able to perform optimization and illustrate some of the potential gains from the more mature kernel architecture. So I guess my message would be this: we should focus our energy in demonstrating (and making sure) FreeBSD is the best platform out there. We've done an incredible job, but we need to keep doing it. This includes doing a better job with PR. We need to do exactly what you've laid the groundwork for: making an argument for FreeBSD as a platform. Robert N M Watson