From owner-freebsd-fs@FreeBSD.ORG  Wed Jan 16 12:05:35 2013
Return-Path: <owner-freebsd-fs@FreeBSD.ORG>
Delivered-To: freebsd-fs@FreeBSD.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by hub.freebsd.org (Postfix) with ESMTP id 1BAD7574
 for <freebsd-fs@FreeBSD.org>; Wed, 16 Jan 2013 12:05:35 +0000 (UTC)
 (envelope-from avg@FreeBSD.org)
Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140])
 by mx1.freebsd.org (Postfix) with ESMTP id 61E14140
 for <freebsd-fs@FreeBSD.org>; Wed, 16 Jan 2013 12:05:34 +0000 (UTC)
Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua
 [212.40.38.100])
 by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA00475;
 Wed, 16 Jan 2013 14:05:30 +0200 (EET) (envelope-from avg@FreeBSD.org)
Received: from localhost ([127.0.0.1])
 by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD))
 id 1TvRkI-000O2E-51; Wed, 16 Jan 2013 14:05:30 +0200
Message-ID: <50F69788.2040506@FreeBSD.org>
Date: Wed, 16 Jan 2013 14:05:28 +0200
From: Andriy Gapon <avg@FreeBSD.org>
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64;
 rv:17.0) Gecko/17.0 Thunderbird/17.0
MIME-Version: 1.0
To: Nicolas Rachinsky <fbsd-mas-0@ml.turing-complete.org>
Subject: Re: slowdown of zfs (tx->tx)
References: <CAFqOu6gWpMsWN0pTBiv10WfwyGWMfO9GzMLWTtcVxHixr-_i3Q@mail.gmail.com>
 <20130114094010.GA75529@mid.pc5.i.0x5.de>
 <CAFqOu6hxfGt_M6Jo9qWeifDz9YnNc_Bd9H-GEe4RYtutaPvH5w@mail.gmail.com>
 <20130114195148.GA20540@mid.pc5.i.0x5.de>
 <CAFqOu6jwJ4qhbOovN_NhzusdQJvrbvUC3g93sziR=Uw99SGenw@mail.gmail.com>
 <20130114214652.GA76779@mid.pc5.i.0x5.de>
 <CAFqOu6jKX-Ks6C1RK5GwZ51ZVUSnGSe7S99_EfK+fwLPjAFFYw@mail.gmail.com>
 <20130115224556.GA41774@mid.pc5.i.0x5.de>
 <CAFqOu6jJnWdbikPmE1-UML5i_x7meF+iyY=9WBRyv2j7AeOaSg@mail.gmail.com>
 <50F67551.5020704@FreeBSD.org> <20130116095009.GA36867@mid.pc5.i.0x5.de>
 <FD780217EA4548F187715AF1AAF2B91A@multiplay.co.uk>
In-Reply-To: <FD780217EA4548F187715AF1AAF2B91A@multiplay.co.uk>
X-Enigmail-Version: 1.4.6
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Cc: freebsd-fs <freebsd-fs@FreeBSD.org>
X-BeenThere: freebsd-fs@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Filesystems <freebsd-fs.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-fs>
List-Post: <mailto:freebsd-fs@freebsd.org>
List-Help: <mailto:freebsd-fs-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-fs>,
 <mailto:freebsd-fs-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 16 Jan 2013 12:05:35 -0000

on 16/01/2013 12:14 Steven Hartland said the following:
> ----- Original Message ----- From: "Nicolas Rachinsky"
> 
>> * Andriy Gapon <avg@FreeBSD.org> [2013-01-16 11:39 +0200]:
>>> on 16/01/2013 02:16 Artem Belevich said the following:
>>> > It appears that lots of threads are stuck in
>>> > metaslab_activate->space_map_load_wait path.
>>>
>>> another thing to check - is your pool nearly full.
>>
>> Don't think so:
>> NAME                                 USED  AVAIL  REFER  MOUNTPOINT
>> pool1                               5.52T   697G  11.9M  /pool1
> 
> You only have ~11% free so yer it is pretty full ;-)

Nicolas,

just in case, Steve is not kidding.

Those free hundreds of gigabytes could be spread over the terabytes and could be
quite fragmented if the pool has a history of adding and removing lots of files.
 ZFS could be spending quite a lot of time in that case when it looks for some
free space and tries to minimize further fragmentation.

Empirical/anecdotal safe limit on pool utilization is said to be about 70-80%.

You can test if this guess is true by doing the following:
kgdb -w
(kgdb) set metaslab_min_alloc_size=4096

If performance noticeably improves after that, then this is your problem indeed.

-- 
Andriy Gapon