From owner-freebsd-stable@FreeBSD.ORG Fri Nov 26 06:05:42 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4919D106566B for ; Fri, 26 Nov 2010 06:05:42 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id D0D248FC17 for ; Fri, 26 Nov 2010 06:05:41 +0000 (UTC) Received: by fxm16 with SMTP id 16so454271fxm.13 for ; Thu, 25 Nov 2010 22:05:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=jOUehNN7WA0AYUDHlGCAJ36oExsx5Z37rIOrz7Sux/Q=; b=H9iFHdTYoUuOgh5gM4ryzztTqld3ThSvisWrTI2YvL+xSYg1ix4rb7nCK1edETAz0N 67vRqowL+HCM4EezP4Uz/oDuPhieQzYWnXnD+q+sd/xUILqfn6pa//O9rbrOKBxZA3Go SMkuG05Zalpukwsi6zx77LH6+/T7UsPCHF0t0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=rAQWq4FFWyyacSiNKLSwuzNPJchWs7K163WmBPm7MCNbROW0CMwGdCYUEeIrF6yIpv a4gUL6ofkXJUsobQJBIh3SmBvXPAIo/RoGK/GUdmRaVdSdiKzcMzow2SsJPx06E2Evur HIPL5I1tvr4cxWIU9g21kPQ7wEQgxq/n8yIe4= MIME-Version: 1.0 Received: by 10.223.117.2 with SMTP id o2mr1665254faq.6.1290751540622; Thu, 25 Nov 2010 22:05:40 -0800 (PST) Received: by 10.223.121.138 with HTTP; Thu, 25 Nov 2010 22:05:40 -0800 (PST) Date: Fri, 26 Nov 2010 00:05:40 -0600 Message-ID: From: Adam Vande More To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: More IO identification problems X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Nov 2010 06:05:42 -0000 As a follow-up, after applying both patches presented in this thread(FreeBSD 8.2-PRERELEASE #5: Thu Nov 25 19:14:00 CST 2010), http://lists.freebsd.org/pipermail/freebsd-stable/2010-November/060298.html "top -m io" now displays much more info and is generally consistent with gstat in that IO spikes occer around the same time. However, gstat and "top -m io" are still not displaying any stats for 90%+ of my hard drive indicator light blinks. Since the issue effects gstat as well, it doesn't seem like it could be related to ZFS. When the system is basically idling, the only consistent IO related entries are these; 6 root 15 0 1 12 0 13 20.00% {l2arc_feed_threa} 6 root 15 0 1 12 0 13 20.00% {arc_reclaim_thre} 6 root 15 0 1 12 0 13 20.00% {zvol zoot/usr/ho} 6 root 15 0 1 12 0 13 20.00% {txg_thread_enter} 6 root 15 0 1 12 0 13 20.00% {txg_thread_enter} Entries like these occur like clockwork on 30 sec intervals. My theory here is that since the ZVOL has UFS + SU, this is causing a sync? What I'm trying to diagnose is a much more frequent hard drive access which occurs on approximately 2 sec intervals. I timed this by pinging localhost and comparing the response to the blinks. It's very consistent although not completely so as once in awhile the blink occurs every second. If I listen carefully I can hear increased drive activity during many of these intervals so the indicator light seems to be working correctly. If anyone has ideas or tests they'd like me to run, it would be appreciated. -- Adam Vande More