From owner-freebsd-stable@FreeBSD.ORG Sat Jan 29 22:17:11 2005 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5052A16A4CE for ; Sat, 29 Jan 2005 22:17:11 +0000 (GMT) Received: from carver.gumbysoft.com (carver.gumbysoft.com [66.220.23.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 07BB943D46 for ; Sat, 29 Jan 2005 22:17:11 +0000 (GMT) (envelope-from dwhite@gumbysoft.com) Received: by carver.gumbysoft.com (Postfix, from userid 1000) id EC4CA72DD4; Sat, 29 Jan 2005 14:17:10 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by carver.gumbysoft.com (Postfix) with ESMTP id E787B72DCB; Sat, 29 Jan 2005 14:17:10 -0800 (PST) Date: Sat, 29 Jan 2005 14:17:10 -0800 (PST) From: Doug White To: Oren Baum In-Reply-To: <1106945920.41faa7806f30b@secure.creativeimage.ca> Message-ID: <20050129140054.S85926@carver.gumbysoft.com> References: <1106945920.41faa7806f30b@secure.creativeimage.ca> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-stable@freebsd.org Subject: Re: Seemingly odd disc i/o behaviour, need help to diagnose X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 29 Jan 2005 22:17:11 -0000 On Fri, 28 Jan 2005, Oren Baum wrote: > Description of Server configuration and problems. > > Configuration: > > FreeBSD 4.4-RELEASE > apache+mod_ssl-1.3.22+2.8.5_1 > mysql-server-3.23.42 > Hard Drive (dual ATA disks, no RAID) on a Dell P4 PowerEDGE server It bears mentioning again ... these releases are very old and have known security issues. If this server is accessible to the public Internet its probably been hacked several times over. An upgrade to at least 4.8-RELEASE+patches, apache 1.3.33, and the latest mysql-server-3.23 is recommended. > We had many speed and timeout issues so we recompiled the kernel with > maxusers=128 instead of the previous 32 and moved the hard drives into a new > P4 2.8Ghz PowerEdge Case. I'd think this is a PE1750 but those have SCSI disks, not IDE. > atapci0: port 0xffa0-0xffaf,0x374-0x377,0x170- > 0x177,0x3f4-0x3f7,0x1f0-0x1f7 irq 11 at device 31.1 on pci0 > ata0: at 0x1f0 irq 14 on atapci0 > atapci1: port 0xfea0-0xfeaf,0xfe30-0xfe33,0xfe20- > 0xfe27,0xfe10-0xfe13,0xfe00-0xfe07 mem 0xdff3fc00-0xdff3ffff irq 5 at device > 31.2 on pci0 > ata2: at 0xfe00 on atapci1 > ata3: at 0xfe20 on atapci1 Also as noted before this may be causing your disks to run at UDMA33 and not at their full interface speed. The 'atacontrol mode ' command will display the current settings (although you'll have to specify a bus, in this case 0, 2, and 3). > Server is configured to hold 2 medium sized MySQL DB's accessed through > various perl and php scripts via websites on the server. > > Uptime: 78463 Threads: 18 Questions: 351685 Slow queries: 328 Opens: 2793 > Flush tables: 1 Open tables: 64 Queries per second avg: 4.482 > > Each httpd process is of size 15MB to 20MB * about 15 processes > Mysqld process is of size 32MB, resident 16MB How much RAM does the machine have? A full dmesg output would be nice for reference. > If we run pine on a large mailbox or any other disk i/o intensive task, > all other processes in motion seem to stall until the disk i/o is > complete. This manifests itself in timeouts on webpages that require DB > data, IMAP timeouts for other mail accounts, and even odd console/shell > behaviour. I suspect this is simple I/O contention from one of two sources: 1. mysql keeps the disks loaded enough that when pine slurps in the big file its able to out-contend the other processes. 2. pine bloats badly with large mailboxes, enough to cause other processes to swap, which kills performance dead. I'd suggest running 'iostat 1' in one window or VTY, 'top -s1' in another, watch for a bit to get a baseline, then start up pine in another window and watch the numbers change. In the top window, watch for the "Swap" line to change with +In and +Out lines, and for the total use to go up. In the iostat window watch how big MB/s gets, and the 'sy' column on the far right, which is system CPU time as a percent. If you're seeing the iostat MB/s go up to a level and stop, but swap stay still, then its case #1. If you see iostat go up and Swap to start increasing then its case #2. In case #2 the System CPU % should go up radically (+10-20%) compared to baseline. To fix case #1 you'll need to get faster storage, either by upgrading the OS to make full use of the available ATA channels or by getting a more capable storage subsystem (SCSI+disk array, for example). To fix case #2 you need to install more physical RAM. ATA does not handle multiple transaction streams well since each bus can only take one command at a time. The ata driver attempts to optimize the order of these operations but compared to SCSI will not scale well in the face of large quantities of diverse transactions. (Before people get up in arms, I'll say that YMMV depending on workload, available cache, interface speeds, etc.) There is, of course, option #3 which is to move your mail to another machine. :-) -- Doug White | FreeBSD: The Power to Serve dwhite@gumbysoft.com | www.FreeBSD.org