From owner-freebsd-performance@FreeBSD.ORG Mon Jan 12 22:41:26 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B83C416A4CE for ; Mon, 12 Jan 2004 22:41:26 -0800 (PST) Received: from farside.isc.org (farside.isc.org [204.152.187.5]) by mx1.FreeBSD.org (Postfix) with ESMTP id 710CD43D55 for ; Mon, 12 Jan 2004 22:41:12 -0800 (PST) (envelope-from Peter_Losher@isc.org) Received: from dhcp-2.sql1.plosh.net (tardis-nat.plosh.net [64.139.14.228]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by farside.isc.org (Postfix) with ESMTP id 10A4BA82B for ; Tue, 13 Jan 2004 06:41:12 +0000 (UTC) (envelope-from Peter_Losher@isc.org) From: Peter Losher Organization: ISC Date: Mon, 12 Jan 2004 22:41:19 -0800 User-Agent: KMail/1.5.94 To: performance@freebsd.org MIME-Version: 1.0 Content-Disposition: inline Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Message-Id: <200401122241.19827.Peter_Losher@isc.org> Subject: FreeBSD 5.x performance tips (ISC) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Jan 2004 06:41:26 -0000 =2D----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 So, as many of you know ISC hosts a quad-Xeon server running FreeBSD 5.1 (-p10 to be precise) which hosts half of ftp.freebsd.org, etc. Many of you helped out with some teething pains w/ virtual memory sizes, and kernel panics. Thanks :) The issue with the system now is that while the kernel is SMP-aware, and as I watch 5.2-REL get downloaded today, this system is like the arm muscle that is developed to lift that barbell, but not enough blood is getting everywhere, so the barbell is slowly moving up while the muscle cramps like hell. In this case the system is ~70% idle, and around 150 processes are locked and the performance starts to seriously decrease at times. (Entropy stops getting collected, etc.) Not a pretty sight.=20 The CPU's are all spinlocking on an I/O channel. so high I/O translates into artificial high cpu and load averages. So where can I look for pointers on how I can squeeze better performance out of this configuration? I already have the usual sysctl entries installed. Any chance moving to 5.2 will help the situation? Best Wishes - Peter =2D --=20 Peter_Losher@isc.org | ISC | OpenPGP Key E8048D08 | "The bits must flow" =2D----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (FreeBSD) iD8DBQFAA5MPPtVx9OgEjQgRAowqAJ9Qd34lOAIfOVvqmZRqa/AoIyO+4wCfYdfB g2zHHnsq6WKa/oy/6I/fA6w=3D =3DbrcS =2D----END PGP SIGNATURE----- From owner-freebsd-performance@FreeBSD.ORG Wed Jan 14 07:41:21 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2265016A4CE for ; Wed, 14 Jan 2004 07:41:21 -0800 (PST) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id CB95043D6B for ; Wed, 14 Jan 2004 07:41:17 -0800 (PST) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.12.10/8.12.10) with ESMTP id i0EFdaUd046962; Wed, 14 Jan 2004 10:39:36 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)i0EFdahs046959; Wed, 14 Jan 2004 10:39:36 -0500 (EST) (envelope-from robert@fledge.watson.org) Date: Wed, 14 Jan 2004 10:39:36 -0500 (EST) From: Robert Watson X-Sender: robert@fledge.watson.org To: Peter Losher In-Reply-To: <200401122241.19827.Peter_Losher@isc.org> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: performance@freebsd.org Subject: Re: FreeBSD 5.x performance tips (ISC) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Jan 2004 15:41:21 -0000 On Mon, 12 Jan 2004, Peter Losher wrote: > So, as many of you know ISC hosts a quad-Xeon server running FreeBSD 5.1 > (-p10 to be precise) which hosts half of ftp.freebsd.org, etc. Many of > you helped out with some teething pains w/ virtual memory sizes, and > kernel panics. Thanks :) > > The issue with the system now is that while the kernel is SMP-aware, and > as I watch 5.2-REL get downloaded today, this system is like the arm > muscle that is developed to lift that barbell, but not enough blood is > getting everywhere, so the barbell is slowly moving up while the muscle > cramps like hell. In this case the system is ~70% idle, and around 150 > processes are locked and the performance starts to seriously decrease at > times. (Entropy stops getting collected, etc.) Not a pretty sight. The > CPU's are all spinlocking on an I/O channel. so high I/O translates into > artificial high cpu and load averages. > > So where can I look for pointers on how I can squeeze better performance > out of this configuration? I already have the usual sysctl entries > installed. Any chance moving to 5.2 will help the situation? Not sure how you feel about running more debugging stuff on this system, but it might actually be quite interesting to see the results of running mutex profiling on it. The documentation on mutex profiling isn't great, there's basically just a note in NOTES: # MUTEX_PROFILING - Profiling mutual exclusion locks (mutexes). This # records four numbers for each acquisition point (identified by # source file name and line number): longest time held, total time held, # number of non-recursive acquisitions, and average time held. Measurements # are made and stored in nanoseconds (using nanotime(9)), but are presented # in microseconds, which should be sufficient for the locks which actually # want this (those that are held long and / or often). The MUTEX_PROFILING # option has the following sysctl namespace for controlling and viewing its # operation: # # debug.mutex.prof.enable - enable / disable profiling # debug.mutex.prof.acquisitions - number of mutex acquisitions held # debug.mutex.prof.records - number of acquisition points recorded # debug.mutex.prof.maxrecords - max number of acquisition points # debug.mutex.prof.rejected - number of rejections (due to full table) # debug.mutex.prof.hashsize - hash size # debug.mutex.prof.collisions - number of hash collisions # debug.mutex.prof.stats - profiling statistics # options MUTEX_PROFILING What you want to do is compile it in, wait for load to reach about "average" -- perhaps a few minutes after booting, then turn profiling on using the sysctl. Wait a couple of minutes to get a sample, turn it off, and dump the records from the stats sysctl, which should generate text. The mutex profiling code isn't highly exercised, but can provide some interesting insight into where your system is bottlenecked with the current locking scheme. It sounds like you're smacked up against the Giant lock, which is not surprising given your workload. The socket locking work Sam has in Perforce may help quite a bit, but they aren't yet ready to merge. Once they stabilize a bit, your environment would probably be an excellent one to measure their impact in. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Senior Research Scientist, McAfee Research