From owner-freebsd-fs@FreeBSD.ORG Sun Nov 24 03:15:59 2013 Return-Path: Delivered-To: fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 64F0F5C9 for ; Sun, 24 Nov 2013 03:15:59 +0000 (UTC) Received: from mail-qa0-x236.google.com (mail-qa0-x236.google.com [IPv6:2607:f8b0:400d:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 259CD2A24 for ; Sun, 24 Nov 2013 03:15:58 +0000 (UTC) Received: by mail-qa0-f54.google.com with SMTP id f11so6366938qae.20 for ; Sat, 23 Nov 2013 19:15:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=eitanadler.com; s=0xdeadbeef; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=BuszNd3QYXv5O0HnvG+uw3+9JWfYKneA2TZthAx1OjA=; b=Hv45ANryhs/klWr9+7OkZEGteKNCLNeMm7D1eAyCSdzO/ZB7LDIhMwLjZMdOCXaBv0 LgMSDLCBqG0mZDSgajvUMaak0tJyOpHIL0G/1iInVHN+lK5Xg6owIr1CLJKQToAqoHo/ 09SfdxZRkBQ357ue6RyydaSsVI9h4nPVylCBI= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=BuszNd3QYXv5O0HnvG+uw3+9JWfYKneA2TZthAx1OjA=; b=MVnZevHvC/XBMWJrywkOTFlya7Wl64bES4oe+keicyUT5igLfCMTMoe1gAjlmlnsrF 3TfLz+nesFE0DKaEWcF1ZKqzdCLPT7vCXTu7mn3/ZaNSEuw6oObC5GYoVgm4BpRtjs+m NeHq1Opm9eqEnBtfbeQP7gapmP0MgIFOqnm65DoVGyndtohEW/fZMr5ZF8u57JvOUWx3 XDoPf1Ps8u+QuEGo8BjZxZQcv60BM5a+ljklk0BbXulPhSFirmWNE4tQvkUH+CrVhsXe UDZQmHLo+0LT4ZAjy4HK0KbWF52cXLSYp+JroY3HnU3BZRoYPeQh7g6lDU2ZwisuTgbB W3aw== X-Gm-Message-State: ALoCoQljjlBfEA9+xyARkchzVRh0OfkbuY38mydYzJZg1e4GwIMXWJ1/Cqo8Yaau6Y+YxRZFYn0d X-Received: by 10.49.35.112 with SMTP id g16mr34558245qej.13.1385262958118; Sat, 23 Nov 2013 19:15:58 -0800 (PST) MIME-Version: 1.0 Received: by 10.96.63.101 with HTTP; Sat, 23 Nov 2013 19:15:28 -0800 (PST) In-Reply-To: <5290E0CF.20704@gibfest.dk> References: <5290E0CF.20704@gibfest.dk> From: Eitan Adler Date: Sat, 23 Nov 2013 22:15:28 -0500 Message-ID: Subject: Re: ZFS (or something) is absurdly slow To: Thomas Steen Rasmussen Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.16 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Nov 2013 03:15:59 -0000 On Sat, Nov 23, 2013 at 12:07 PM, Thomas Steen Rasmussen wrote: > On 23-11-2013 09:26, Eitan Adler wrote: >> >> Every so often I see absurdly slow tasks stuck on "zio->io_cv". >> >> For example a recent "git checkout file.c" did not complete for many >> minutes >> load: 0.65 cmd: git 74577 [zio->io_cv] 435.58r 0.20u 2.54s 0% 71488k >> >> I have seen "ls ~" take tends of minutes to complete. Even "ls >> /var/empty" can take just as long. This length of time is variable >> but is usually much longer than expected. >> >> Does anyone have any suggestions for helping to figure out what is >> taking a long time? > > Hello, > > If "top -m io -o total" doesn't reveal what is using the disks, Nothing unexpected here though I will pay attention during the slow times. > I've had good experiences with the following dtrace script, > you'd need to build dtrace support in your kernel though: > vfsstat.d https://forums.freebsd.org/showpost.php?p=182070&postcount=6 I can run this script, what output should I be looking for? > You can also check systat -iostast 1 and check the TPS count for > the disks. The regular (spinning) harddisk can manage 200-300 iops > if it is a regular consumer class disk. Are you "running out" of > iops for some reason ? tps stays between 0 and 30 or so. I rarely if ever seen numbers above 30 (This is a consumer grade laptop HDD). tty ada0 ada1 cd0 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 25 1144 6.88 5 0.03 13.76 35 0.46 0.00 0 0.00 1 4 2 0 93 -- Eitan Adler