From owner-freebsd-fs@FreeBSD.ORG Tue Jul 16 14:23:41 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4DF269FC for ; Tue, 16 Jul 2013 14:23:41 +0000 (UTC) (envelope-from daniel@digsys.bg) Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.21.123]) by mx1.freebsd.org (Postfix) with ESMTP id CEE059DC for ; Tue, 16 Jul 2013 14:23:39 +0000 (UTC) Received: from dcave.digsys.bg (dcave.digsys.bg [193.68.6.1]) (authenticated bits=0) by smtp-sofia.digsys.bg (8.14.6/8.14.6) with ESMTP id r6GENbX7075932 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 16 Jul 2013 17:23:38 +0300 (EEST) (envelope-from daniel@digsys.bg) Message-ID: <51E55769.4030207@digsys.bg> Date: Tue, 16 Jul 2013 17:23:37 +0300 From: Daniel Kalchev User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130627 Thunderbird/17.0.7 MIME-Version: 1.0 To: Ivailo Tanusheff Subject: Re: ZFS vdev I/O questions References: <51E5316B.9070201@digsys.bg> <20130716115305.GA40918@mwi1.coffeenet.org> <51E54799.8070700@digsys.bg> <9d3cf0be165d4351acc5e757de3868ec@DB3PR07MB059.eurprd07.prod.outlook.com> In-Reply-To: <9d3cf0be165d4351acc5e757de3868ec@DB3PR07MB059.eurprd07.prod.outlook.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: "freebsd-fs@freebsd.org" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jul 2013 14:23:41 -0000 On 16.07.13 17:09, Ivailo Tanusheff wrote: > Isn't this some kind of pool fragmentation? Because this is usually the case in such slow parts of the disk systems. I think your pool is getting full and it is heavily fragmented, that's why you have more data for each request on a different vdev. The pool may be fragmented. But not because it is full. It is fragmented because I forgot to add an ZIL when creating the pool, then proceeded to heavily use dedup and even some compression. Now, I am rewriting the pool's data and hopefully metadata, in userland, for the lack of better technology, primarily by doing zfs send/receive of various datasets then removing the originals. That helps me both balance the data across all vdevs as well as get rid of dedup and compression (that go to other pools with less deletes). My guess is this is more specifically metadata fragmentation. But fragmentation does not fully explain why the writes are so irregular -- writes should be grouped easily, especially metadata rewrites... and what is ZFS doing while not reading or writing (many seconds)? Morale: always add an ZIL to an ZFS pool, as this will save you to deal with fragmentation later. Depending on the pool usage, even an normal drive could do. Writes to the ZIL are sequential. > But this has nothing to do with the single, slow device :( > That drive is slow only when doing lots of small I/O. For bulk writes (which ZFS should be doing anyway with the kind of data this pool holds), it is actually faster than the Hitachi's. It will eventually get replaced soon. Daniel