From owner-freebsd-fs@FreeBSD.ORG  Sat Nov 23 17:15:02 2013
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: fs@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits))
 (No client certificate requested)
 by hub.freebsd.org (Postfix) with ESMTPS id 4C99BD7D
 for <fs@freebsd.org>; Sat, 23 Nov 2013 17:15:02 +0000 (UTC)
Received: from mail.tyknet.dk (mail.tyknet.dk [176.9.9.186])
 (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits))
 (No client certificate requested)
 by mx1.freebsd.org (Postfix) with ESMTPS id 0ACAB21AB
 for <fs@freebsd.org>; Sat, 23 Nov 2013 17:15:01 +0000 (UTC)
Received: from [10.255.193.199] (d153234.upc-d.chello.nl [213.46.153.234])
 (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits))
 (No client certificate requested)
 by mail.tyknet.dk (Postfix) with ESMTPSA id 15C5D1CA2B5;
 Sat, 23 Nov 2013 18:07:31 +0100 (CET)
DKIM-Filter: OpenDKIM Filter v2.8.3 mail.tyknet.dk 15C5D1CA2B5
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=gibfest.dk; s=default;
 t=1385226452; bh=XlJVlbq23oZW1PCnJc0FTfuI9lj2iQtuNhngmM2IDRI=;
 h=Date:From:To:CC:Subject:References:In-Reply-To;
 b=CvQpdafapQrmLIBtmXn+l/tY33FGr/yUGHCQfWuky6RmJXLyJMDi5YkDYY2GIdQNg
 YbGczbMBgOrQ7iJqds0unHHwtHR9zPtAh1GaT2T+OA046xyHnnIyQctaw5284VsH2P
 litSXgbTbe9WzML6T8Yi8F5EtePq0ASMl1y1Bolo=
Message-ID: <5290E0CF.20704@gibfest.dk>
Date: Sat, 23 Nov 2013 18:07:27 +0100
From: Thomas Steen Rasmussen <thomas@gibfest.dk>
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64;
 rv:24.0) Gecko/20100101 Thunderbird/24.1.1
MIME-Version: 1.0
To: Eitan Adler <lists@eitanadler.com>
Subject: Re: ZFS (or something) is absurdly slow
References: <CAF6rxgmepSN9pFPH+QiLaNqhzXxkXwu=59zvfD-6gGEMg9zh1g@mail.gmail.com>
In-Reply-To: <CAF6rxgmepSN9pFPH+QiLaNqhzXxkXwu=59zvfD-6gGEMg9zh1g@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Cc: "freebsd-fs@freebsd.org" <fs@freebsd.org>
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.16
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs/>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Sat, 23 Nov 2013 17:15:02 -0000

On 23-11-2013 09:26, Eitan Adler wrote:
> Every so often I see absurdly slow tasks stuck on "zio->io_cv".
>
> For example a recent "git checkout file.c" did not complete for many minutes
> load: 0.65  cmd: git 74577 [zio->io_cv] 435.58r 0.20u 2.54s 0% 71488k
>
> I have seen "ls ~" take tends of minutes to complete.  Even "ls
> /var/empty" can take just as long.  This length of time is variable
> but is usually much longer than expected.
>
> Does anyone have any suggestions for helping to figure out what is
> taking a long time?
Hello,

If "top -m io -o total" doesn't reveal what is using the disks,
I've had good experiences with the following dtrace script,
you'd need to build dtrace support in your kernel though:

vfsstat.d https://forums.freebsd.org/showpost.php?p=182070&postcount=6

You can also check systat -iostast 1 and check the TPS count for
the disks. The regular (spinning) harddisk can manage 200-300 iops
if it is a regular consumer class disk. Are you "running out" of
iops for some reason ?

Good luck with it,


Best regards
Thomas Steen Rasmussen