From owner-freebsd-performance@FreeBSD.ORG Sun May 8 13:05:42 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 086C516A4E2; Sun, 8 May 2005 13:05:42 +0000 (GMT) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3905743D5D; Sun, 8 May 2005 13:05:39 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.11] (junior-wifi.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j48DAgkj021051; Sun, 8 May 2005 07:10:43 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <427E0E87.1040701@samsco.org> Date: Sun, 08 May 2005 07:05:11 -0600 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050218 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Steven Hartland References: <19879.1115061648@critter.freebsd.dk><20050502214208.M87351@fledge.watson.org> <42770026.80901@he.iki.fi> <01ad01c54fef$a56e6260$b3db87d4@multiplay.co.uk> In-Reply-To: <01ad01c54fef$a56e6260$b3db87d4@multiplay.co.uk> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.8 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org cc: Eric Anderson cc: Poul-Henning Kamp cc: Robert Watson cc: Petri Helenius cc: freebsd-performance@freebsd.org Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 May 2005 13:05:42 -0000 Steven Hartland wrote: > Summary of results: > RAID0: > Changing vfs.read_max 8 -> 16 and MAXPHYS 128k -> 1M > increased read performance significantly from 129Mb/s to 199MB/s > Max raw device speed here was 234Mb/s > FS -> Raw device: 35Mb/s 14.9% performance loss > > RAID5: Changing vfs.read_max 8 -> 16 produced a small increase > 129Mb/s to 135Mb/s. > > Increasing MAXPHYS 128k -> 1M prevented vfs.read_max from > having any effect > Max raw device speed here was 200Mb/s > FS -> Raw device: 65Mb/s 32.5% performance loss > > Note: This batch of tests where done on uni processor kernel to > keep variation down to a minimum, so are not directly comparable > with my previous tests. All tests where performed with 16k RAID > stripe across all 5 disks and a default newfs. Increasing or decreasing > the block size for the fs was tried but only had negative effects. Changing MAXPHYS is very dangerous, unfortunately. The root of the problem is that kernel virtual memory (KVA) gets assigned to each I/O buffer as it passes through the kernel. If we allow too much I/O through at once then we have the very real possibility of exhausting the kernel address space and causing a deadlock and/or panic. That is why MAXPHYS is set so low. Your DD test is unlikely to trigger a problem, but try doing a bunch of DD's is parallel and you likely will. The solution is to re-engineer the way that I/O buffers pass through the kernel and only assign KVA when needed (for doing software parity calculations, for example). That way we could make MAXPHYS be any arbitrarily large number and not worry about exhausting KVA. I believe that there is some work in progress in this area, but it's a large project since nearly every single storage driver would need to be changed. Another possibility is to recognise that amd64 doesn't have the same KVA restrictions as i386 and thus can be treated differently. However, doing the KVA work is still attractive since it'll yeild some performance benefits too. Scott From owner-freebsd-performance@FreeBSD.ORG Sun May 8 17:15:25 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 95A6016A4E4; Sun, 8 May 2005 17:15:25 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id 807DA43D4C; Sun, 8 May 2005 17:15:24 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j48HFPvQ022846; Sun, 8 May 2005 19:15:28 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Scott Long From: "Poul-Henning Kamp" In-Reply-To: Your message of "Sun, 08 May 2005 07:05:11 MDT." <427E0E87.1040701@samsco.org> Date: Sun, 08 May 2005 19:15:25 +0200 Message-ID: <22845.1115572525@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: Robert Watson cc: freebsd-performance@freebsd.org cc: Steven Hartland cc: Petri Helenius cc: Eric Anderson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 May 2005 17:15:25 -0000 In message <427E0E87.1040701@samsco.org>, Scott Long writes: >The solution is to re-engineer the way that I/O buffers pass through >the kernel and only assign KVA when needed (for doing software parity >calculations, for example). I've hacked up a prototype to do this and there is no doubt that this is the way we are headed, hopefully sometime in the 7-CURRENT period. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Mon May 9 11:11:32 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6AD4116A4E7; Mon, 9 May 2005 11:11:32 +0000 (GMT) Received: from mailout1.pacific.net.au (mailout1.pacific.net.au [61.8.0.84]) by mx1.FreeBSD.org (Postfix) with ESMTP id B3D0E43DBA; Mon, 9 May 2005 11:11:31 +0000 (GMT) (envelope-from bde@zeta.org.au) Received: from mailproxy1.pacific.net.au (mailproxy1.pacific.net.au [61.8.0.86])j49BBMrI008312; Mon, 9 May 2005 21:11:22 +1000 Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) j49BBFx7022833; Mon, 9 May 2005 21:11:16 +1000 Date: Mon, 9 May 2005 21:11:16 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Scott Long In-Reply-To: <427E0E87.1040701@samsco.org> Message-ID: <20050509203849.J20942@delplex.bde.org> References: <19879.1115061648@critter.freebsd.dk><20050502214208.M87351@fledge.watson.org> <01ad01c54fef$a56e6260$b3db87d4@multiplay.co.uk> <427E0E87.1040701@samsco.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed cc: Petri Helenius cc: Poul-Henning Kamp cc: Robert Watson cc: Steven Hartland cc: freebsd-performance@FreeBSD.org cc: Eric Anderson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 May 2005 11:11:32 -0000 On Sun, 8 May 2005, Scott Long wrote: > Changing MAXPHYS is very dangerous, unfortunately. The root of the > problem is that kernel virtual memory (KVA) gets assigned to each I/O > buffer as it passes through the kernel. If we allow too much I/O through > at once then we have the very real possibility of exhausting the kernel > address space and causing a deadlock and/or panic. That is why MAXPHYS > is set so low. Your DD test is unlikely to trigger a problem, but try > doing a bunch of DD's is parallel and you likely will. > > The solution is to re-engineer the way that I/O buffers pass through > the kernel and only assign KVA when needed (for doing software parity > calculations, for example). That way we could make MAXPHYS be any > arbitrarily large number and not worry about exhausting KVA. I believe > that there is some work in progress in this area, but it's a large > project since nearly every single storage driver would need to be > changed. Another possibility is to recognise that amd64 doesn't have > the same KVA restrictions as i386 and thus can be treated differently. > However, doing the KVA work is still attractive since it'll yeild some > performance benefits too. How would it yield anything except complexity and negative performance benefits on machines that don't need it? The most efficient way to map kva for disk buffers is to do it only once at boot time using a simple sparse mapping technique as in 4.4BSD-Lite1. Simple sparse mapping needs lots of kva, and i386's don't have enough for machines with more than about 16MB of RAM, but machines with a 64-bit address space can have 2^32 times as much kva as i386's. Bruce From owner-freebsd-performance@FreeBSD.ORG Mon May 9 12:14:27 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 16C4616A4E7; Mon, 9 May 2005 12:14:27 +0000 (GMT) Received: from critter.freebsd.dk (0x535c0e2a.sgnxx1.adsl-dhcp.tele.dk [83.92.14.42]) by mx1.FreeBSD.org (Postfix) with ESMTP id E3CDF43D6B; Mon, 9 May 2005 12:14:25 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.3/8.13.3) with ESMTP id j49CEEfb000692; Mon, 9 May 2005 14:14:17 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Bruce Evans From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 09 May 2005 21:11:16 +1000." <20050509203849.J20942@delplex.bde.org> Date: Mon, 09 May 2005 14:14:14 +0200 Message-ID: <691.1115640854@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: Scott Long cc: Petri Helenius cc: Robert Watson cc: Steven Hartland cc: freebsd-performance@FreeBSD.org cc: Eric Anderson Subject: Re: Very low disk performance on 5.x X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 May 2005 12:14:27 -0000 In message <20050509203849.J20942@delplex.bde.org>, Bruce Evans writes: >> The solution is to re-engineer the way that I/O buffers pass through >> the kernel and only assign KVA when needed (for doing software parity >> calculations, for example). > >How would it yield anything except complexity and negative performance >benefits on machines that don't need it? The crucial point here is that disk controllers do not need it. Most disk-controller hardware is able to do scatter/gather these days and we cannot exploit that because we insist on disk-I/O happening from sequential KVM addresses. The silly thing about us insisting on KVM mapping in the first place is that there is practically no modern hardware that needs that. ATA in PIO mode. RAID5 parity calculation. GBDE encryption and that is about it. Of course we need to cater for those in the new world order too, but it seems only fair to put the overhead in front of those three since they are already slow operations. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-performance@FreeBSD.ORG Tue May 10 13:52:20 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7C49A16A4CE for ; Tue, 10 May 2005 13:52:20 +0000 (GMT) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 62C7643D5E for ; Tue, 10 May 2005 13:52:17 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.11] (junior.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j4ADv2Ws034608; Tue, 10 May 2005 07:57:02 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <4280BC75.7040408@samsco.org> Date: Tue, 10 May 2005 07:51:49 -0600 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050218 X-Accept-Language: en-us, en MIME-Version: 1.0 To: noackjr@alumni.rice.edu References: <427FAC30.80507@alumni.rice.edu> In-Reply-To: <427FAC30.80507@alumni.rice.edu> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.8 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org cc: ewan@mathcode.net cc: performance@freebsd.org cc: Pete French Subject: Re: Performance issue X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 May 2005 13:52:20 -0000 Jonathan Noack wrote: > On 5/9/2005 12:31 PM, Pete French wrote: > >>> 5.3 ships with SMP turned on, which makes lock operations rather >>> expensive on single-processor machines. 4.x does not have SMP >>> turned on by default. Would you be able to re-run your test with >>> SMP turned off? >> >> >> >> I just ran a test here with SMP turned of on 5.4-RC4 (GENERIC) I got the >> following result: >> >> 67.52 real 41.13 user 26.16 sys >> 7034 involuntary context switches >> >> i.e. it still has system time a a huge proportion of the total compared >> to the 4.11 kernel. Interesingly, after reading Holger Kipp's results >> I tried it on a genuine multi-processor box with SMP enabled running 5.3. >> He got a very small percentage of the time in sys (3.51 out of 81.90) but >> I got: >> 255.30 real 160.20 user 88.50 sys >> >> Once again a far higher proprtion of the time spent in sys than you would >> expect. > > > I ran into a similar issue when attempting to thread a card game solver > program I wrote. Performance in early versions was horrific and I > noticed tons of context switches. I resolved the issue by allocating > pools of memory beforehand. This seems to point the finger to malloc > and context switch overhead. > > In any case, I believe this is related to threading. Check your results > with libthr instead. The following are on my 2.53 GHz P4 which is > running CURRENT from last night (with INVARIANTS on). I have benchmarks from other programs that show that process scope threads in libpthread are extremely slow, while system scope threads a much much faster. The new libthr is even faster, but I'd consider it very experimental at this time. It is possible to build a version of libpthread that uses only system scope threads, look in /usr/src/lib/libpthread/Makefile for a comment block that talks about it. David Xu is actively working on libthr and I'm hoping that it matures over the next few months and becomes a viable alternative. However, I'd also like to see libpthread get fixed. The performance problems point to the UTS being extremely inefficient, and there are reports that it does at least one syscall for every thread switch. Since the whole point of KSE/SA is to avoid syscalls on thread switches, the problem might be an obvious one, thought the solution might not be. The kernel side of KSE also does a malloc on every upcall, which again is highly inefficient for process scope threads. The solution here seems to be fairly simple, it just needs someone to sit down for a few hours and work on it. Scott From owner-freebsd-performance@FreeBSD.ORG Tue May 10 15:18:49 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8B22D16A4CE for ; Tue, 10 May 2005 15:18:49 +0000 (GMT) Received: from gate.bitblocks.com (bitblocks.com [209.204.185.216]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4C3A743D7B for ; Tue, 10 May 2005 15:18:49 +0000 (GMT) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost [127.0.0.1]) by gate.bitblocks.com (8.13.3/8.13.1) with ESMTP id j4AFImSv071163 for ; Tue, 10 May 2005 08:18:48 -0700 (PDT) (envelope-from bakul@bitblocks.com) Message-Id: <200505101518.j4AFImSv071163@gate.bitblocks.com> To: performance@freebsd.org In-reply-to: Your message of "Tue, 10 May 2005 07:51:49 MDT." <4280BC75.7040408@samsco.org> Date: Tue, 10 May 2005 08:18:48 -0700 From: Bakul Shah Subject: Regression testing (was Re: Performance issue) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 May 2005 15:18:49 -0000 This thread makes me wonder if there is value in runing performance tests on a regular basis. This would give an early warning of any peformance loss and can be a useful forensic tool (one can pinpoint when some performance curve changed discontinuously even though at the time of change it may be too small to be noticed). Over a period of time one can gain a view of how the performance evolves. This would not be a single metric but a set of low and high level measures: such as syscall overhead, interrupt overhead, specific h/w devices, disk and fs performance for various filesystems and file sizes, networking data and pkt throughput, routing performance, VM, other subsystems, effect of SMP, various threading libraries, scaling with number of users/programs/cpus/memory, typical applications under normal and stressed loads, compile time for the system and kernel etc. etc. etc. The setup would allow for easy addition of new benchmarks (the only way anything like this can be bootstrapped). Of course, one would need to record disk/processor/memory speed and capacities + kernel config options, system build tools and their options to interpret the results as best as possible. For the results to be useful the setup has to remain as stable as possible for a long time. [While I am dreaming...] A follow on project would be to create visualization tools -- mainly graphing and comparing graphs. It would be neat if one can click on a performance graph to zoom in or see commits made during some selected period. Such a detailed look, combined with profiling can help people focus on specific hotspots & feel good about any improvements they are making. This can be a great way to rope in new people;-) From owner-freebsd-performance@FreeBSD.ORG Tue May 10 19:11:15 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 7140C16A4CE for ; Tue, 10 May 2005 19:11:15 +0000 (GMT) Received: from smtp815.mail.sc5.yahoo.com (smtp815.mail.sc5.yahoo.com [66.163.170.1]) by mx1.FreeBSD.org (Postfix) with SMTP id 337CF43D48 for ; Tue, 10 May 2005 19:11:15 +0000 (GMT) (envelope-from noackjr@alumni.rice.edu) Received: from unknown (HELO optimator.noacks.org) (noacks@swbell.net@70.240.205.64 with login) by smtp815.mail.sc5.yahoo.com with SMTP; 10 May 2005 18:35:50 -0000 Received: from localhost (localhost [127.0.0.1]) by optimator.noacks.org (Postfix) with ESMTP id 128266125; Tue, 10 May 2005 13:35:49 -0500 (CDT) Received: from optimator.noacks.org ([127.0.0.1]) by localhost (optimator.noacks.org [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 41547-04; Tue, 10 May 2005 13:35:47 -0500 (CDT) Received: from [127.0.0.1] (optimator [192.168.1.11]) by optimator.noacks.org (Postfix) with ESMTP id 37CF960F3; Tue, 10 May 2005 13:35:47 -0500 (CDT) Message-ID: <4280FEFC.4080107@alumni.rice.edu> Date: Tue, 10 May 2005 13:35:40 -0500 From: Jonathan Noack User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Bakul Shah References: <200505101518.j4AFImSv071163@gate.bitblocks.com> In-Reply-To: <200505101518.j4AFImSv071163@gate.bitblocks.com> X-Enigmail-Version: 0.91.0.0 OpenPGP: id=991D8195; url=http://www.noacks.org/cert/noackjr.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig27BDEB018E649B3C3993C4E7" X-Virus-Scanned: amavisd-new at noacks.org cc: performance@freebsd.org Subject: Re: Regression testing (was Re: Performance issue) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: noackjr@alumni.rice.edu List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 May 2005 19:11:15 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig27BDEB018E649B3C3993C4E7 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit On 5/10/2005 10:18 AM, Bakul Shah wrote: > This thread makes me wonder if there is value in runing > performance tests on a regular basis. This would give an > early warning of any peformance loss and can be a useful > forensic tool (one can pinpoint when some performance curve > changed discontinuously even though at the time of change it > may be too small to be noticed). Over a period of time > one can gain a view of how the performance evolves. > > This would not be a single metric but a set of low and high > level measures: such as syscall overhead, interrupt overhead, > specific h/w devices, disk and fs performance for various > filesystems and file sizes, networking data and pkt > throughput, routing performance, VM, other subsystems, effect > of SMP, various threading libraries, scaling with number of > users/programs/cpus/memory, typical applications under normal > and stressed loads, compile time for the system and kernel > etc. etc. etc. > > The setup would allow for easy addition of new benchmarks > (the only way anything like this can be bootstrapped). Of > course, one would need to record disk/processor/memory speed > and capacities + kernel config options, system build tools > and their options to interpret the results as best as > possible. For the results to be useful the setup has to > remain as stable as possible for a long time. > > [While I am dreaming...] A follow on project would be to > create visualization tools -- mainly graphing and comparing > graphs. It would be neat if one can click on a performance > graph to zoom in or see commits made during some selected > period. > > Such a detailed look, combined with profiling can help people > focus on specific hotspots & feel good about any improvements > they are making. This can be a great way to rope in new > people;-) Sounds great! When do you begin? ;-) This has been proposed before and has been (to my knowledge) universally accepted as a Good Idea. If you have the interest and time to devote to it, I would urge you to work on it. The benefit to the community would be huge. -- Jonathan Noack | noackjr@alumni.rice.edu | OpenPGP: 0x991D8195 --------------enig27BDEB018E649B3C3993C4E7 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (MingW32) iD8DBQFCgP8BUFz01pkdgZURAjrUAKCgrZwGxaFIRgtN8QH4nJ9FH9w3DgCeMg0K t+VXCdCckzeo4fUshP3FP+U= =i/ts -----END PGP SIGNATURE----- --------------enig27BDEB018E649B3C3993C4E7-- From owner-freebsd-performance@FreeBSD.ORG Tue May 10 19:51:15 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1E11E16A4CE for ; Tue, 10 May 2005 19:51:15 +0000 (GMT) Received: from silver.he.iki.fi (helenius.fi [193.64.42.241]) by mx1.FreeBSD.org (Postfix) with ESMTP id E1D5A43DA4 for ; Tue, 10 May 2005 19:51:11 +0000 (GMT) (envelope-from pete@he.iki.fi) Received: from [193.64.42.172] (hac.vuokselantie10.fi [193.64.42.172]) by silver.he.iki.fi (Postfix) with ESMTP id 99EADBC46; Tue, 10 May 2005 22:51:07 +0300 (EEST) Message-ID: <428110D2.8070004@he.iki.fi> Date: Tue, 10 May 2005 22:51:46 +0300 From: Petri Helenius User-Agent: Mozilla Thunderbird 1.0.2 (Windows/20050317) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Bakul Shah References: <200505101518.j4AFImSv071163@gate.bitblocks.com> In-Reply-To: <200505101518.j4AFImSv071163@gate.bitblocks.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: performance@freebsd.org Subject: Re: Regression testing (was Re: Performance issue) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 May 2005 19:51:15 -0000 This sounds somewhat similar to Solaris dtrace stuff? Pete Bakul Shah wrote: >This thread makes me wonder if there is value in runing >performance tests on a regular basis. This would give an >early warning of any peformance loss and can be a useful >forensic tool (one can pinpoint when some performance curve >changed discontinuously even though at the time of change it >may be too small to be noticed). Over a period of time >one can gain a view of how the performance evolves. > >This would not be a single metric but a set of low and high >level measures: such as syscall overhead, interrupt overhead, >specific h/w devices, disk and fs performance for various >filesystems and file sizes, networking data and pkt >throughput, routing performance, VM, other subsystems, effect >of SMP, various threading libraries, scaling with number of >users/programs/cpus/memory, typical applications under normal >and stressed loads, compile time for the system and kernel >etc. etc. etc. > >The setup would allow for easy addition of new benchmarks >(the only way anything like this can be bootstrapped). Of >course, one would need to record disk/processor/memory speed >and capacities + kernel config options, system build tools >and their options to interpret the results as best as >possible. For the results to be useful the setup has to >remain as stable as possible for a long time. > >[While I am dreaming...] A follow on project would be to >create visualization tools -- mainly graphing and comparing >graphs. It would be neat if one can click on a performance >graph to zoom in or see commits made during some selected >period. > >Such a detailed look, combined with profiling can help people >focus on specific hotspots & feel good about any improvements >they are making. This can be a great way to rope in new >people;-) >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > > From owner-freebsd-performance@FreeBSD.ORG Tue May 10 19:53:51 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9567316A4CE for ; Tue, 10 May 2005 19:53:51 +0000 (GMT) Received: from strawberry.seattlefenix.net (dsl231-061-109.sea1.dsl.speakeasy.net [216.231.61.109]) by mx1.FreeBSD.org (Postfix) with ESMTP id 355EA43DA0 for ; Tue, 10 May 2005 19:53:51 +0000 (GMT) (envelope-from roo@seattlefenix.net) Received: from strawberry.seattlefenix.net (localhost [127.0.0.1]) by strawberry.seattlefenix.net (Postfix) with ESMTP id 019851B5A6; Tue, 10 May 2005 12:55:09 -0700 (PDT) Received: (from roo@localhost)j4AJt9wH039884; Tue, 10 May 2005 12:55:09 -0700 (PDT) (envelope-from roo) Date: Tue, 10 May 2005 12:55:09 -0700 From: Benjamin Krueger To: Petri Helenius Message-ID: <20050510195508.GR86767@strawberry.seattlefenix.net> References: <200505101518.j4AFImSv071163@gate.bitblocks.com> <428110D2.8070004@he.iki.fi> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <428110D2.8070004@he.iki.fi> User-Agent: Mutt/1.4.2.1i cc: Bakul Shah cc: performance@freebsd.org Subject: Re: Regression testing (was Re: Performance issue) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Benjamin Krueger List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 May 2005 19:53:51 -0000 * Petri Helenius (pete@he.iki.fi) [050510 12:52]: > > This sounds somewhat similar to Solaris dtrace stuff? > > Pete This sounds more like a QA/Feedback process. dtrace isn't just a conglomeration of the usual performance metrics. It's closer to a total revamp of how you perform said metrics at the system level. > Bakul Shah wrote: > > >This thread makes me wonder if there is value in runing > >performance tests on a regular basis. This would give an > >early warning of any peformance loss and can be a useful > >forensic tool (one can pinpoint when some performance curve > >changed discontinuously even though at the time of change it > >may be too small to be noticed). Over a period of time > >one can gain a view of how the performance evolves. > > > >This would not be a single metric but a set of low and high > >level measures: such as syscall overhead, interrupt overhead, > >specific h/w devices, disk and fs performance for various > >filesystems and file sizes, networking data and pkt > >throughput, routing performance, VM, other subsystems, effect > >of SMP, various threading libraries, scaling with number of > >users/programs/cpus/memory, typical applications under normal > >and stressed loads, compile time for the system and kernel > >etc. etc. etc. > > > >The setup would allow for easy addition of new benchmarks > >(the only way anything like this can be bootstrapped). Of > >course, one would need to record disk/processor/memory speed > >and capacities + kernel config options, system build tools > >and their options to interpret the results as best as > >possible. For the results to be useful the setup has to > >remain as stable as possible for a long time. > > > >[While I am dreaming...] A follow on project would be to > >create visualization tools -- mainly graphing and comparing > >graphs. It would be neat if one can click on a performance > >graph to zoom in or see commits made during some selected > >period. > > > >Such a detailed look, combined with profiling can help people > >focus on specific hotspots & feel good about any improvements > >they are making. This can be a great way to rope in new > >people;-) > >_______________________________________________ > >freebsd-performance@freebsd.org mailing list > >http://lists.freebsd.org/mailman/listinfo/freebsd-performance > >To unsubscribe, send any mail to > >"freebsd-performance-unsubscribe@freebsd.org" > > > > > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" -- -- Benjamin Krueger SysAdmin, CarDomain Network 92 Toyota Turbo MR2 78 Datsun B-210 91 freakin Geo Prizm From owner-freebsd-performance@FreeBSD.ORG Tue May 10 20:32:34 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1261616A4CE for ; Tue, 10 May 2005 20:32:34 +0000 (GMT) Received: from gate.bitblocks.com (bitblocks.com [209.204.185.216]) by mx1.FreeBSD.org (Postfix) with ESMTP id B482043D62 for ; Tue, 10 May 2005 20:32:33 +0000 (GMT) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost [127.0.0.1]) by gate.bitblocks.com (8.13.3/8.13.1) with ESMTP id j4AKWWcD073387; Tue, 10 May 2005 13:32:33 -0700 (PDT) (envelope-from bakul@bitblocks.com) Message-Id: <200505102032.j4AKWWcD073387@gate.bitblocks.com> To: Petri Helenius In-reply-to: Your message of "Tue, 10 May 2005 22:51:46 +0300." <428110D2.8070004@he.iki.fi> Date: Tue, 10 May 2005 13:32:32 -0700 From: Bakul Shah cc: Bakul Shah cc: performance@freebsd.org Subject: Re: Regression testing (was Re: Performance issue) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 May 2005 20:32:34 -0000 > This sounds somewhat similar to Solaris dtrace stuff? Dtrace can be a (very useful) component for collecting performance metrics. What I am talking about is a framework where you'd apply dtrace or other micro/system level performance tests or benchmarks on a regular basis for a variety of machines, loads etc. and collate results in a usable form. The purpose is to provide an ongoing view of how performance of various subsystems and the system as a whole changes for various loads and configurations as the codebase evolves. This gives an early warning of performance loss (as seen in -5.x versus -4.x releases) as well as early confirmation of improvements (as seen in -6.x versus -5.x). Users can provide early feedback witout having to wait for a release. It is difficult and time consuming for developers to measure the impact of their changes across a variety of systems, configurations and loads. A centralized performance measuring system can be very valuable here. If you see that e.g. a new scheduler has a terrible impact on some systems or loads, you'd either come up with something better or provide a knob. If you see that a nifty new feature has a significant performance cost, you'd be less tempted to make it the default (or at least others get a chance to scream early on). From owner-freebsd-performance@FreeBSD.ORG Tue May 10 22:23:10 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6FDFC16A4CE for ; Tue, 10 May 2005 22:23:10 +0000 (GMT) Received: from flake.decibel.org (flake.decibel.org [67.100.216.10]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0E1B343D39 for ; Tue, 10 May 2005 22:23:10 +0000 (GMT) (envelope-from decibel@decibel.org) Received: by flake.decibel.org (Postfix, from userid 1001) id 57FB71522B; Tue, 10 May 2005 17:23:07 -0500 (CDT) Date: Tue, 10 May 2005 17:23:07 -0500 From: "Jim C. Nasby" To: freebsd-performance@freebsd.org Message-ID: <20050510222307.GS31103@decibel.org> References: <200505101518.j4AFImSv071163@gate.bitblocks.com> <4280FEFC.4080107@alumni.rice.edu> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4280FEFC.4080107@alumni.rice.edu> X-Operating-System: FreeBSD 4.11-RELEASE i386 X-Distributed: Join the Effort! http://www.distributed.net User-Agent: Mutt/1.5.8i Subject: Re: Regression testing (was Re: Performance issue) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 May 2005 22:23:10 -0000 On Tue, May 10, 2005 at 01:35:40PM -0500, Jonathan Noack wrote: > Sounds great! When do you begin? ;-) > > This has been proposed before and has been (to my knowledge) universally > accepted as a Good Idea. If you have the interest and time to devote to > it, I would urge you to work on it. The benefit to the community would > be huge. Does FreeBSD have any facilities setup for peoople wishing to work on 'sub projects' collaboratively? Something akin to http://pgfoundry.org for PostgreSQL? Several of us have been using pgFoundry as a means to work on things that will eventually go into PostgreSQL itself, and it's been a big benefit when it comes to bringing people together to work on something. -- Jim C. Nasby, Database Consultant decibel@decibel.org Give your computer some brain candy! www.distributed.net Team #1828 Windows: "Where do you want to go today?" Linux: "Where do you want to go tomorrow?" FreeBSD: "Are you guys coming, or what?" From owner-freebsd-performance@FreeBSD.ORG Thu May 12 15:48:14 2005 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A6EF816A4CE for ; Thu, 12 May 2005 15:48:14 +0000 (GMT) Received: from mailout08.sul.t-online.com (mailout08.sul.t-online.com [194.25.134.20]) by mx1.FreeBSD.org (Postfix) with ESMTP id E8A0343D7E for ; Thu, 12 May 2005 15:48:13 +0000 (GMT) (envelope-from Alexander@Leidinger.net) Received: from fwd27.aul.t-online.de by mailout08.sul.t-online.com with smtp id 1DWFv1-0000k3-02; Thu, 12 May 2005 17:48:11 +0200 Received: from Andro-Beta.Leidinger.net (JbphGuZO8ezwj33ZuTqYNHDaZVdL9CTZ3b1Or3G3B1Zx+CztE96Fsl@[217.81.91.174]) by fwd27.sul.t-online.de with esmtp id 1DWFuz-28Up3g0; Thu, 12 May 2005 17:48:09 +0200 Received: from localhost (localhost [127.0.0.1])j4CFm68v063115; Thu, 12 May 2005 17:48:06 +0200 (CEST) (envelope-from Alexander@Leidinger.net) Received: from 141.113.101.32 ([141.113.101.32]) by netchild.homeip.net (Horde MIME library) with HTTP for ; Thu, 12 May 2005 17:48:06 +0200 Message-ID: <20050512174806.bh9e3zgyskkk8wco@netchild.homeip.net> X-Priority: 3 (Normal) Date: Thu, 12 May 2005 17:48:06 +0200 From: Alexander Leidinger To: "Jim C. Nasby" References: <200505101518.j4AFImSv071163@gate.bitblocks.com> <20050510222307.GS31103@decibel.org> In-Reply-To: <20050510222307.GS31103@decibel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-15; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.0.3) / FreeBSD-4.11 X-ID: JbphGuZO8ezwj33ZuTqYNHDaZVdL9CTZ3b1Or3G3B1Zx+CztE96Fsl@t-dialin.net X-TOI-MSGID: e876229e-8f54-4f7e-8182-d536695394fa cc: freebsd-performance@freebsd.org Subject: Re: Regression testing (was Re: Performance issue) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 May 2005 15:48:14 -0000 "Jim C. Nasby" wrote: > Does FreeBSD have any facilities setup for peoople wishing to work on > 'sub projects' collaboratively? Something akin to http://pgfoundry.org > for PostgreSQL? No, there's no such thing. But what stops you from using SF (or something similar) as a CVS repository? Freel free to discuss high level issues here (or any other @FreeBSD.org list which fits the topic of the work) or at least announce periodically the status of the work. The tools to do this are out there. We don't want to restrict how you work. Bye, Alexander. -- http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 BOFH excuse #225: It's those computer people in X {city of world}, they keep stuffing things up