From owner-freebsd-questions@FreeBSD.ORG Tue Oct 27 23:31:50 2009 Return-Path: Delivered-To: questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF3071065698 for ; Tue, 27 Oct 2009 23:31:50 +0000 (UTC) (envelope-from jin@daemonfun.com) Received: from server70.appriver.com (server70a.appriver.com [69.20.116.35]) by mx1.freebsd.org (Postfix) with ESMTP id 8C4678FC27 for ; Tue, 27 Oct 2009 23:31:50 +0000 (UTC) X-Policy: GLOBAL - ubicom.com X-Primary: jin@daemonfun.com X-Note: This Email was scanned by AppRiver SecureTide X-Virus-Scan: V- X-Note: TCH-CT/SI:0-24/SG:2 10/27/2009 6:31:09 PM X-GBUdb-Analysis: 0, 216.112.109.98, Ugly c=0.640417 p=-0.904 Source Normal X-Signature-Violations: 0-0-0-5811-c X-Note: Spam Tests Failed: X-Country-Path: PRIVATE->UNITED STATES->UNITED STATES X-Note-Sending-IP: 216.112.109.98 X-Note-Reverse-DNS: 216.112.109.98.ptr.us.xo.net X-Note-WHTLIST: jin@daemonfun.com X-Note: User Rule Hits: X-Note: Global Rule Hits: 115 116 117 118 122 123 220 X-Note: Mail Class: VALID X-Note: Headers Injected Received: from [216.112.109.98] (HELO stork.scenix.com) by server70.appriver.com (CommuniGate Pro SMTP 5.2.16) with ESMTP id 105913893 for questions@freebsd.org; Tue, 27 Oct 2009 18:31:50 -0400 Received: from [10.10.50.211] ([10.10.50.211]) by stork.scenix.com with Microsoft SMTPSVC(6.0.3790.3959); Tue, 27 Oct 2009 15:28:13 -0700 Message-ID: <4AE774BC.8080006@daemonfun.com> Date: Tue, 27 Oct 2009 15:31:24 -0700 From: Jin Guojun User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.13) Gecko/20080928 X-Accept-Language: en-us, en MIME-Version: 1.0 To: questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-OriginalArrivalTime: 27 Oct 2009 22:28:13.0265 (UTC) FILETIME=[C54F2C10:01CA5754] Cc: Subject: What causes random disk access slow down X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 27 Oct 2009 23:31:51 -0000 A 6-7 years old Xeon dual 2.4MHz CPU machine runs FreeBSD 6.4-Release suddenly becomes slow on some tasks requiring disk access. Typical things like ls, objdump etc. Be more specific, a couple of minutes objdump became a several hours job. A several seconds "ls -RC" became a 15-minute task (see output below). It sounds like a hard drive problem, but run sequential disk test on all drives, their throughput meet the original disk spec and disks run very quite, at random disk access, disks generate some rigid noise, so it looks like a random disk access problem. This machine has two IDE PATA drives (ignore da0 -- a USB stick), but No error message has been recorded in dmesg for any dirve a couple of weeks after the problem happened. Machine has been rebooted a few times after slowness occurred, but it won't help. Is there anyway/any tool to find out what is going wrong in the system? -Jin [165] bsd-ms: ls -RC > Dir 3.756u 19.402s 15:29.37 2.4% 30+2938k 49120+76io 0pf+0w monitored from the other terms -- [138] bsd-ms: ll ~/Dir -rw-r--r-- 1 src wheel 6152192 Oct 27 14:53 /home/users/src/Dir [139] bsd-ms: ll ~/Dir -rw-r--r-- 1 src wheel 8019968 Oct 27 14:56 /home/users/src/Dir [140] bsd-ms: ll ~/Dir -rw-r--r-- 1 src wheel 9915957 Oct 27 14:58 /home/users/src/Dir tty ad0 ad1 da0 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 9 365 9.14 6 0.05 12.08 6 0.07 121.91 0 0.00 2 0 1 0 97 0 1020 11.75 87 0.99 17.82 7 0.13 0.00 0 0.00 76 0 14 0 10 0 1005 8.54 262 2.19 52.94 23 1.21 0.00 0 0.00 61 0 29 1 9 0 893 7.54 184 1.36 85.76 34 2.82 0.00 0 0.00 53 0 32 1 14 0 551 3.35 265 0.87 9.38 4 0.04 0.00 0 0.00 47 0 33 1 19 0 594 6.81 201 1.33 37.82 4 0.14 0.00 0 0.00 54 0 16 0 30 0 1106 3.54 252 0.87 55.19 17 0.93 0.00 0 0.00 39 0 33 1 27 0 393 2.88 223 0.63 11.43 2 0.03 0.00 0 0.00 67 0 31 1 1 0 644 4.81 165 0.77 16.00 0 0.01 0.00 0 0.00 87 0 12 1 1 27 339 10.39 180 1.82 15.18 11 0.17 0.00 0 0.00 86 0 13 0 0 32 130 5.06 146 0.72 23.40 46 1.04 0.00 0 0.00 86 0 8 1 5 32 267 8.39 138 1.13 61.09 4 0.22 0.00 0 0.00 73 0 26 1 0 33 340 8.75 222 1.90 61.54 4 0.26 0.00 0 0.00 78 0 21 1 0 32 595 5.85 154 0.88 12.20 3 0.04 0.00 0 0.00 87 0 12 1 0 32 288 5.28 147 0.76 6.00 1 0.01 0.00 0 0.00 86 0 13 1 0