Date: Thu, 22 Dec 2011 10:49:32 -0800 From: Matthew Tippett <matthew@phoronix.com> To: Adrian Chadd <adrian@freebsd.org> Cc: Alexander Leidinger <Alexander@leidinger.net>, freebsd-performance@freebsd.org, Erik Cederstrand <erik@cederstrand.dk> Subject: Re: Benchmark (Phoronix): FreeBSD 9.0-RC2 vs. Oracle Linux 6.1 Server Message-ID: <4EF37BBC.3070404@phoronix.com> In-Reply-To: <CAJ-Vmo=jRwprjW%2BKCQ%2Bs7f2J-tFQvZbJ2CwkA_9rGTO5c6JkMw@mail.gmail.com> References: <bfjsm0n2j5fie38pfmjaqbyf.1324493312789@email.android.com> <001CED31-FDD0-4CB1-B972-2F2344EEB9D3@cederstrand.dk> <CAJ-Vmo=jRwprjW%2BKCQ%2Bs7f2J-tFQvZbJ2CwkA_9rGTO5c6JkMw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
(Resending - hopefully without horrific escaping somewhere upstream). Let me suggest an alternative. Within the Phoronix Test Suite ecosystem, we have a continious integration/validation system called Phoromatic (http://www.phoromatic.com/). We have a brief theory of operation on it captured https://docs.google.com/present/view?id=ds439pg_42hpg57m86 . You can see a Linux oriented tracker available at http://phoromatic.com/kernel-tracker.php (the nice graphs are when you select 180 days and the ION 330 systems). We've re-assigned the systems to other projects so it is no longer being updated, but it served our purposes. This system will allow you to just dedicate a machine to be updateable and pick up the directions for which test to be run. The FreeBSD project (or contributors) would maintain the slave or test machines. The test suite would be selected and managed by the FreeBSD project or contributors. The glue code to emit triggers (possibly SVN, git or other submission, or even just daily) and the scripts to update the systems would also be maintained by the FreeBSD community. Phoronix Media would be happy to host it on Phoromatic.com (we've played with hosted with project branding) and provide the data store and the analytics. We'd also be willing to make enhancements to support the FreeBSD project. This should solve the "I don't have time to maintain a automated test infrastructure". You don't need to, just write the glue scripts, and dedicate a couple of machines. I believe FreeBSD vendors like ixSystems may be able to support this effort with a dedicated machine. You can have as many machines as you like demonstrating AMD/Intel/32/64/large mem/low mem,etc. The comments around interactivity can also be measured to some extent. We have the model of a "monitor". This can be configured to determine jitter around a number of system variables and to possibly inject actions to measure impacts. We're more than happy to work with you guys, and are willing to help do a lot of the infrastructure and automation lifting. Regards, Matthew On 12/22/11 8:56 AM, Adrian Chadd wrote: > Guys, girls, fuzzy creatures, > > This is by far the best example of a constructive email in this entire thread. > > If people would like to help, Erik here is exactly the kind of person > with exactly the kind of software that needs a hand. > > I think enough philosophizing has been done - now we have questions > that need answering; theories that need testing. And that requires, > you know, coding. :) > > The best thing right now would be for *BSD people to pick up the > Phronix test suite, try to compile/run it, and provide feedback. Do > your own benchmarks on your own hardware and report back the results. > That's how we fix the "benchmarking problem." We don't fix it by > armchair philosophy, we fix it by getting our hands dirty. :) > > 2c, > > > Adrian > > > On 22 December 2011 03:21, Erik Cederstrand<erik@cederstrand.dk> wrote: >> Den 21/12/2011 kl. 19.48 skrev Alexander Leidinger: >> >>> And related to the subject: wasn't it you who developed the automatic benchmarking stuff? If yes, why not make it available? If you don't have he resources, I offer my help to make it available somewhere. >> Yes, that's me. I'm mostly out of time right now, but I'd like to offer help if someone wants to pick up the project. >> >> For those who haven't heard about it, it's a system designed specifically to track performance of FreeBSD over time by comparing revisions of FreeBSD, everything else being equal. It consists of a tinderbox-like build script for a build server, a script to install FreeBSD and run benchmarks on at least one slave, and a database-backed website to aggregate and visualize results. >> >> The framework does work as-is, but it really needs to be updated: convert the scripts to use the SVN repo instead of CVS, improve visualization and search on the web fronted, and improve the benchmarking script so it's easier to extend. I don't have hardware available to run the benchmarks, but I think there's hardware available in the FreeBSD cluster. >> >> Here's a link to the source code:http://dev.affect-it.dk/tracker.tgz >> And to my thesis describing how it works:http://dev.affect-it.dk/tracker.pdf >> >> Just send me a mail if you're interested. >> >> Thanks, >> Erik_______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to"freebsd-performance-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to"freebsd-performance-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4EF37BBC.3070404>