From owner-freebsd-stable@FreeBSD.ORG Fri Nov 26 14:33:30 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 08A231065696; Fri, 26 Nov 2010 14:33:30 +0000 (UTC) (envelope-from jhellenthal@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 9A2B98FC1B; Fri, 26 Nov 2010 14:33:29 +0000 (UTC) Received: by gxk8 with SMTP id 8so1004242gxk.13 for ; Fri, 26 Nov 2010 06:33:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :organization:user-agent:mime-version:to:cc:subject:references :in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; bh=oess6BJtECSOdYrn2ulwumN9NwDvsGhrdvfeE/o9o1E=; b=bJqq/8ixDv/vtYoabY+F0SzYQLq6ij+2pon/m2zrIo1Z/oVQl1i9Ma7xnZPx/bjHdl zgBqGMSfWcV6VXOmjBanB9bQ+v2wPz727hse6yerH/bfDbiNXhJ0VFs/RqgILAp8qJai Zexd0qubHGMFNeLLpwv81+iJAljTk3andrQFI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:organization:user-agent:mime-version:to :cc:subject:references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=p8Z5eBshLyho7F16Y16HsVPk6mYqhnuv5Yqdx8rs0CfkgqzJ7JFg4KewS6ED/Ni6it DpPxpNUawSbdub3Ae+Vdlf5+95CKPE0kbZHm6AnpqAfvvFxiejgkz/Se9jlzXzOT7A6P PnTzeLBCcIaBYfZiaIsRxwMKj1/CsPMBpwolo= Received: by 10.90.4.22 with SMTP id 22mr4621568agd.26.1290782008515; Fri, 26 Nov 2010 06:33:28 -0800 (PST) Received: from centel.dataix.local (adsl-99-181-129-55.dsl.klmzmi.sbcglobal.net [99.181.129.55]) by mx.google.com with ESMTPS id r31sm539670yhc.24.2010.11.26.06.33.25 (version=SSLv3 cipher=RC4-MD5); Fri, 26 Nov 2010 06:33:27 -0800 (PST) Sender: "J. Hellenthal" Message-ID: <4CEFC534.2070904@DataIX.net> Date: Fri, 26 Nov 2010 09:33:24 -0500 From: jhell Organization: http://www.DataIX.net User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.2.12) Gecko/20101028 Lightning/1.0b1 Thunderbird MIME-Version: 1.0 To: Andriy Gapon References: <4CEFB8F7.4080002@freebsd.org> In-Reply-To: <4CEFB8F7.4080002@freebsd.org> X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: Adam Vande More , freebsd-stable@freebsd.org Subject: Re: More IO identification problems X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Nov 2010 14:33:30 -0000 On 11/26/2010 08:41, Andriy Gapon wrote: > on 26/11/2010 08:05 Adam Vande More said the following: >> As a follow-up, after applying both patches presented in this thread(FreeBSD >> 8.2-PRERELEASE #5: Thu Nov 25 19:14:00 CST 2010), >> >> http://lists.freebsd.org/pipermail/freebsd-stable/2010-November/060298.html >> >> "top -m io" now displays much more info and is generally consistent with >> gstat in that IO spikes occer around the same time. However, gstat and "top >> -m io" are still not displaying any stats for 90%+ of my hard drive >> indicator light blinks. Since the issue effects gstat as well, it doesn't >> seem like it could be related to ZFS. When the system is basically idling, >> the only consistent IO related entries are these; >> >> 6 root 15 0 1 12 0 13 20.00% >> {l2arc_feed_threa} >> 6 root 15 0 1 12 0 13 20.00% >> {arc_reclaim_thre} >> 6 root 15 0 1 12 0 13 20.00% {zvol >> zoot/usr/ho} >> 6 root 15 0 1 12 0 13 20.00% >> {txg_thread_enter} >> 6 root 15 0 1 12 0 13 20.00% >> {txg_thread_enter} >> >> Entries like these occur like clockwork on 30 sec intervals. My theory here >> is that since the ZVOL has UFS + SU, this is causing a sync? >> >> What I'm trying to diagnose is a much more frequent hard drive access which >> occurs on approximately 2 sec intervals. I timed this by pinging localhost >> and comparing the response to the blinks. It's very consistent although not >> completely so as once in awhile the blink occurs every second. If I listen >> carefully I can hear increased drive activity during many of these intervals >> so the indicator light seems to be working correctly. >> >> If anyone has ideas or tests they'd like me to run, it would be appreciated. >> > > Perhaps it's some external component? > E.g. hald is known to perform some disk/media checks every two seconds. > As well syslod will also cause sync to happen prematurely when something goes to log. syslog.conf(5): *** To ensure that kernel messages are written to disk promptly, syslog.conf calls fsync(2) after writing messages from the kernel. Other messages are not synced explicitly. You may prefix a pathname with the minus sign, ``-'', to forego syncing the specified file after every kernel message. Note that you might lose information if the system crashes immediately following a write attempt. Neverthe- less, using the ``-'' option may improve performance, especially if the kernel is logging many messages. *** This is obviously the old way of ensuring logs are written to disk before crash but now ZFS is handled differently. Check to see if adding '-' to your syslog entries relieves that problem as ZFS will do the right thing to ensure your data is written to disk. -- jhell,v