From owner-freebsd-doc@FreeBSD.ORG Sun Feb 8 05:18:39 2004 Return-Path: Delivered-To: freebsd-doc@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 16FCA16A4CE for ; Sun, 8 Feb 2004 05:18:39 -0800 (PST) Received: from arthur.nitro.dk (port324.ds1-khk.adsl.cybercity.dk [212.242.113.79]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1496643D1F for ; Sun, 8 Feb 2004 05:18:38 -0800 (PST) (envelope-from simon@arthur.nitro.dk) Received: by arthur.nitro.dk (Postfix, from userid 3000) id DEFC211839; Sun, 8 Feb 2004 14:18:36 +0100 (CET) Date: Sun, 8 Feb 2004 14:18:36 +0100 From: "Simon L. Nielsen" To: freebsd-doc@freebsd.org Message-ID: <20040208131834.GG770@arthur.nitro.dk> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="V32M1hWVjliPHW+c" Content-Disposition: inline User-Agent: Mutt/1.5.5.1i Subject: [simon@FreeBSD.org: Re: Hints for precision benchmarking...] X-BeenThere: freebsd-doc@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Documentation project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Feb 2004 13:18:39 -0000 --V32M1hWVjliPHW+c Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable [ Since mailman seems to have sent the first version of this to the eternal bit fields, I'm resending this. ] =46rom: "Simon L. Nielsen" Date: Mon, 26 Jan 2004 22:56:02 +0100 To: freebsd-doc@freebsd.org Cc: Poul-Henning Kamp , Robert Watson User-Agent: Mutt/1.5.5.1i Subject: Re: Hints for precision benchmarking... On 2004.01.26 12:12:58 -0500, Robert Watson wrote: >=20 > On Mon, 26 Jan 2004, Poul-Henning Kamp wrote: >=20 > > * Run in single user mode. Cron(8) and and other daemons > > only add noise. >=20 > Of particular interest here is sshd -- either disable its SSHv1 key > regeneration, or kill the parent sshd (or don't run sshd) during the > tests.=20 I thought these lists were nice to have in documentation for future reference, so I converted them into DocBook. I couldn't really find a good place in the Developers Handbook to put the list, so Robert Watson suggsted I creating a new Regression and Performance Testing chapter. I had to do some rewriting to make the wording more like documentation so I probably managed to totally mess up the gramma, but I hope I don't mess up the actual content. Compiled version at http://people.freebsd.org/~simon/work/testing.html . Comments, suggestions ? Regression and Performance Testing Regression tests are used to exercise a particular bit of the system to check that it works as expected, and to make sure that old bugs are not reintroduced. The &os; regression testing tools can be found in the &os; source tree in the directory src/tools/regression.
Micro Benchmark Checklist This section contains a suggestions for doing proper micro-benchmarking on &os; or off &os; itself. It is not possible to use all of the suggestions below every single time, but the more used, the better the benchmark's ability to test small differences will be. Disable APM and any other kind of clock fiddling (ACPI ?). Run in single user mode. E.g. &man.cron.8;, and and other daemons only add noise. The &man.sshd.8; daemon can also cause problems. If ssh access is required during test either disable the SSHv1 key regeneration, or kill the parent sshd daemon during the tests. Do not run &man.ntpd.8;. If &man.syslog.3; events are generated, run &man.syslogd.8; with an empty /etc/syslogd.conf, otherwise, do not run it. Minimize disk-I/O, avoid it entirely if possible. Do not mount file systems that are not needed. Mount /, /usr, and any other file system as read-only if possible. This removes atime updates to disk (etc.) from the I/O picture. Reinitialize the read/write test file system with &man.newfs.8; and populate it from a &man.tar.1; or &man.dump.8; file before every run. Unmount and mount it before starting the test. This results in a consistent file system layout. For a worldstone test this would apply to /usr/obj (just reinitialize with newfs and mount). To get 100% reproducibility, populate the file system from a &man.dd.1; file (i.e.: dd if=3Dmyimage of=3D/dev/ad0s1h bs=3D1m) Use malloc backed or preloaded &man.md.4; partitions. Reboot between individual iterations of the test, this gives a more consistent state. Remove all non-essential device drivers from the kernel. For instance if USB is not needed for the test, do not put USB in the kernel. Drivers which attach often have timeouts ticking away. Unconfigure hardware that are not in use. Detach disks with &man.atacontrol.8; and &man.camcontrol.8; if the disk are not used for the test. Do not configure the network unless it is being tested, or wait until after the test has been performed to ship the results off to another computer. If the system must be connected to a public network, watch out for spikes of broadcast traffic. Even though it is hardly noticeable, it will take up CPU cycles. Multicast has similar caveats. Put each file system on its own disk. This minimizes jitter from head-seek optimizations. Minimize output to serial or VGA consoles. Running output into files gives less jitter. (Serial consoles easily become a bottleneck.) Do not touch keyboard while the test is running, even space or back-space shows up in the numbers. Make sure the test is long enough, but not too long. If the test is too short, timestamping is a problem. If it is too long temperature changes and drift will affect the frequency of the quartz crystals in the computer. Rule of thumb: more than a minute, less than an hour. Try to keep the temperature as stable as possible around the machine. This affects both quartz crystals and disk drive algorithms. To get real stable clock, consider stabilized clock injection. E.g. get a OCXO + PLL, inject output into clock circuits instead of motherboard xtal. Contact &a.phk; for more information about this. Run the test at least 3 times but it is better to run more than 20 times both for before and after code. Try to interleave if possible (i.e.: do no run 20 times before then 20 times after), this makes it possible to spot environmental effects. Do not interleave 1:1, but 3:3, this makes it possible to spot interaction effects. A good pattern is: bababa{bbbaaa}*. This gives hint after the first 1+1 runs (so it is possible to stop the test if it goes entirely the wrong way), a standard deviation after the first 3+3 (gives a good indication if it is going to be worth a long run) and trending and interaction numbers later on. Use usr/src/tools/tools/ministat to see if the numbers are significant. Consider buying Cartoon guide to statistics ISBN: 0062731025, highly recommended, if you have forgotten or never learned about standard deviation and Student's T. Do not use background &man.fsck.8; unless the test is a benchmark of background fsck. Also, disable background_fsck in /etc/rc.conf unless the benchmark is not started at least 60+fsck runtime seconds after the boot, as &man.rc.8; wakes up and checks if fsck needs to run on any file systems when background fsck is enabled. Likewise, make sure there are no snapshots lying around unless the benchmark is a test with snapshots. If the benchmark show unexpected bad performance, check for things like high interrupt volume from an unexpected source. Some versions of ACPI have been reported to misbehave and generate excess interrupts. To help diagnose odd test results, take a few snapshots of vmstat -i and look for anything unusual. Make sure to be careful about optimization parameters for kernel and userspace, likewise debugging. It is easy to let something slip through and realize later the test was not comparing the same thing. Do not ever benchmark with the WITNESS and INVARIANTS kernel options enabled unless the test is interested to benchmarking those features. WITNESS can cause 400%+ drops in performance. Likewise, userspace &man.malloc.3; parameters default differently in -CURRENT from the way they ship in production releases.
----- End forwarded message ----- --=20 Simon L. Nielsen FreeBSD Documentation Team --V32M1hWVjliPHW+c Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (FreeBSD) iD8DBQFAJjcqh9pcDSc1mlERAieIAJ9nLaK0YtbhL0DczRt0twexP/+V+ACeM866 1T40XMSR8JFbT27jr5jA18w= =CGlf -----END PGP SIGNATURE----- --V32M1hWVjliPHW+c--