From owner-freebsd-stable@FreeBSD.ORG Thu Nov 25 22:35:55 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AAAEE106564A; Thu, 25 Nov 2010 22:35:55 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-fx0-f54.google.com (mail-fx0-f54.google.com [209.85.161.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0349E8FC08; Thu, 25 Nov 2010 22:35:54 +0000 (UTC) Received: by fxm16 with SMTP id 16so250938fxm.13 for ; Thu, 25 Nov 2010 14:35:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:cc:content-type; bh=himBd286ladfs0ix10Mzc2NBqBAGVnTaXgdwTybkoQo=; b=dO19kocfRomG1eXNn2mCbUniudzuQ1/UlXtLnaDDBWB8/HWU6pdBsPNc8u53Q33sAQ Cwm8B9RnzADpC3yUfu8CeqqnP2iymYCI64UDZ/n+3aSnBjkUpfHphCUF2lhFolscpwqc fLKsn6jvETbbVeXBx1HDV8OaWhuX9rZjtJ+gE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=Kc45dC3Mps5mLch0rwbTf4ozaFyt/7szOL2Herimb6PRLEgsW8yeY87N0jLZwHgWQF UkuhdtvW1Lz490QXqhC7sT10rbHEvRLj2EHwNwPE/T0WhukSnoekYIcqliU6RIOMfh5L ch4zgBlfPlz5IAOtPRq66MxC7gpVyf6ywYKuY= MIME-Version: 1.0 Received: by 10.223.79.13 with SMTP id n13mr1339001fak.139.1290724553581; Thu, 25 Nov 2010 14:35:53 -0800 (PST) Received: by 10.223.121.138 with HTTP; Thu, 25 Nov 2010 14:35:53 -0800 (PST) In-Reply-To: <20101125210449.GA57788@icarus.home.lan> References: <4CEE88D9.4070200@DataIX.net> <20101125200813.GA56066@icarus.home.lan> <20101125205733.GA56845@icarus.home.lan> <20101125210449.GA57788@icarus.home.lan> Date: Thu, 25 Nov 2010 16:35:53 -0600 Message-ID: From: Adam Vande More To: Jeremy Chadwick Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Kostik Belousov , freebsd-stable@freebsd.org, Andriy Gapon Subject: Re: top io mode X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Nov 2010 22:35:55 -0000 On Thu, Nov 25, 2010 at 3:04 PM, Jeremy Chadwick wrote: > Bad form to follow up to my own Email of course, but some discussion > material: > I'm a frequent offender myself so I won't be pointing any fingers. "top -m io" doesn't show any I/O writes, while gstat(8) does, and to > numerous devices all which make up some form of ZFS pool. > Yes, it's a generic ZFS mirror. If you do something like "dd if=/dev/urandom of=/pool/file bs=64k", does > "top -m io" show write I/O for the "dd" process? > It does not on my ZFS STABLE system with Kostick's path. So the patch fixes reads, but not writes. cc'ing to notify in case he has more ideas. It does on my UFS CURRENT VM w/ out the patch. > If so, then possibly ZFS "internally" is doing something quietly/behind > the scenes (such as a disk flush or some sort of internal transaction > "thing"), thus the kernel would be responsible and not a process. > That was kind of my thought, and if that's the case there are two problems: 1. Can't identify what's responsible for IO. 2. Any money your company saves due to ZFS features is lost when you get the electric bill. /sarcasm So that said, how about "top -S -m io -U root" instead, which includes > system processes, since we can (probably) assume a non-root process > isn't responsible? > I should have mentioned I had already tried that (-HS) along with -o read/write to see what was happening, but no luck. Hopefully I don't get too confusing here because I'm going to contradict something I said earlier. The stats I posted from gstat are perhaps not directly related to the root issue. Similar stats to what I linked to from gstats can be seen in approx 10 - 15 intervals and is 0.0 %busy in between those updates. However the HD indicator light blinks in 1 second intervals constantly and this is what I want to track down. One more datapoint is that in single user mode, the blinking stops. sa(8) may suffer from the same affliction as a "dd if=/dev/zero of=/tmp/ delete.me bs=64k" had been running along with my constant blinking: galacticdominator# sa -m root 30 0.00cpu 0tio 1972k*sec adam 6 0.27cpu 0tio 2254k*sec Maybe it's some weird hardware issue, but there's at least one other FreeBSD user who is seeing similar behavior. -- Adam Vande More