From owner-freebsd-fs@FreeBSD.ORG Tue Jul 16 14:09:42 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A217462D for ; Tue, 16 Jul 2013 14:09:42 +0000 (UTC) (envelope-from Ivailo.Tanusheff@skrill.com) Received: from db9outboundpool.messaging.microsoft.com (mail-db9lp0249.outbound.messaging.microsoft.com [213.199.154.249]) by mx1.freebsd.org (Postfix) with ESMTP id 0743A93E for ; Tue, 16 Jul 2013 14:09:41 +0000 (UTC) Received: from mail214-db9-R.bigfish.com (10.174.16.236) by DB9EHSOBE012.bigfish.com (10.174.14.75) with Microsoft SMTP Server id 14.1.225.22; Tue, 16 Jul 2013 14:09:33 +0000 Received: from mail214-db9 (localhost [127.0.0.1]) by mail214-db9-R.bigfish.com (Postfix) with ESMTP id A6E9F1E01D2; Tue, 16 Jul 2013 14:09:33 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.249.213; KIP:(null); UIP:(null); IPV:NLI; H:AM2PRD0710HT004.eurprd07.prod.outlook.com; RD:none; EFVD:NLI X-SpamScore: -2 X-BigFish: PS-2(zz98dI9371I542I1432Izz1f42h1ee6h1de0h1fdah2073h1202h1e76h1d1ah1d2ah1fc6hzz17326ah8275dhz2fh2a8h668h839h944hd24hf0ah1220h1288h12a5h12a9h12bdh137ah13b6h1441h1504h1537h153bh162dh1631h1758h18e1h1946h19b5h19ceh1ad9h1b0ah1d07h1d0ch1d2eh1d3fh1de9h1dfeh1dffh1e1dh9a9j1155h) Received-SPF: pass (mail214-db9: domain of skrill.com designates 157.56.249.213 as permitted sender) client-ip=157.56.249.213; envelope-from=Ivailo.Tanusheff@skrill.com; helo=AM2PRD0710HT004.eurprd07.prod.outlook.com ; .outlook.com ; X-Forefront-Antispam-Report-Untrusted: SFV:NSPM; SFS:(189002)(199002)(51704005)(24454002)(13464003)(377454003)(49866001)(74706001)(80022001)(54316002)(47736001)(74662001)(81342001)(74366001)(74502001)(76576001)(50986001)(76786001)(56816003)(16406001)(47976001)(74316001)(15202345003)(81542001)(53806001)(77096001)(76796001)(56776001)(51856001)(66066001)(33646001)(63696002)(74876001)(79102001)(54356001)(4396001)(69226001)(83072001)(46102001)(77982001)(31966008)(65816001)(47446002)(76482001)(59766001)(24736002); DIR:OUT; SFP:; SCL:1; SRVR:DB3PR07MB060; H:DB3PR07MB059.eurprd07.prod.outlook.com; CLIP:217.18.249.148; RD:InfoNoRecords; MX:1; A:1; LANG:en; Received: from mail214-db9 (localhost.localdomain [127.0.0.1]) by mail214-db9 (MessageSwitch) id 1373983771484888_2080; Tue, 16 Jul 2013 14:09:31 +0000 (UTC) Received: from DB9EHSMHS025.bigfish.com (unknown [10.174.16.254]) by mail214-db9.bigfish.com (Postfix) with ESMTP id 7190220006E; Tue, 16 Jul 2013 14:09:31 +0000 (UTC) Received: from AM2PRD0710HT004.eurprd07.prod.outlook.com (157.56.249.213) by DB9EHSMHS025.bigfish.com (10.174.14.35) with Microsoft SMTP Server (TLS) id 14.16.227.3; Tue, 16 Jul 2013 14:09:30 +0000 Received: from DB3PR07MB060.eurprd07.prod.outlook.com (10.242.137.151) by AM2PRD0710HT004.eurprd07.prod.outlook.com (10.255.165.39) with Microsoft SMTP Server (TLS) id 14.16.329.3; Tue, 16 Jul 2013 14:09:24 +0000 Received: from DB3PR07MB059.eurprd07.prod.outlook.com (10.242.137.149) by DB3PR07MB060.eurprd07.prod.outlook.com (10.242.137.151) with Microsoft SMTP Server (TLS) id 15.0.731.16; Tue, 16 Jul 2013 14:09:23 +0000 Received: from DB3PR07MB059.eurprd07.prod.outlook.com ([169.254.2.117]) by DB3PR07MB059.eurprd07.prod.outlook.com ([169.254.2.117]) with mapi id 15.00.0731.000; Tue, 16 Jul 2013 14:09:23 +0000 From: Ivailo Tanusheff To: Daniel Kalchev , "freebsd-fs@freebsd.org" Subject: RE: ZFS vdev I/O questions Thread-Topic: ZFS vdev I/O questions Thread-Index: AQHOghmmLmdfZD/4b0i3hLA3+rJ4o5lnMdyAgAAXNoCAAA4g4A== Date: Tue, 16 Jul 2013 14:09:23 +0000 Message-ID: <9d3cf0be165d4351acc5e757de3868ec@DB3PR07MB059.eurprd07.prod.outlook.com> References: <51E5316B.9070201@digsys.bg> <20130716115305.GA40918@mwi1.coffeenet.org> <51E54799.8070700@digsys.bg> In-Reply-To: <51E54799.8070700@digsys.bg> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [217.18.249.148] x-forefront-prvs: 09090B6B69 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: skrill.com X-FOPE-CONNECTOR: Id%0$Dn%*$RO%0$TLS%0$FQDN%$TlsDn% X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 16 Jul 2013 14:09:42 -0000 Hi danbo :) Isn't this some kind of pool fragmentation? Because this is usually the cas= e in such slow parts of the disk systems. I think your pool is getting full= and it is heavily fragmented, that's why you have more data for each reque= st on a different vdev. But this has nothing to do with the single, slow device :( Best regards, Ivailo Tanusheff -----Original Message----- From: owner-freebsd-fs@freebsd.org [mailto:owner-freebsd-fs@freebsd.org] On= Behalf Of Daniel Kalchev Sent: Tuesday, July 16, 2013 4:16 PM To: freebsd-fs@freebsd.org Subject: Re: ZFS vdev I/O questions On 16.07.13 14:53, Mark Felder wrote: > On Tue, Jul 16, 2013 at 02:41:31PM +0300, Daniel Kalchev wrote: >> I am observing some "strange" behaviour with I/O spread on ZFS vdevs=20 >> and thought I might ask if someone has observed it too. >> > --SNIP-- > >> Drives da0-da5 were Hitachi Deskstar 7K3000 (Hitachi HDS723030ALA640,=20 >> firmware MKAOA3B0) -- these are 512 byte sector drives, but da0 has=20 >> been replaced by Seagate Barracuda 7200.14 (AF) (ST3000DM001-1CH166,=20 >> firmware >> CC24) -- this is an 4k sector drive of a new generation (notice the=20 >> relatively 'old' firmware, that can't be upgraded). > --SNIP-- > As you can see, the initial burst is to all vdevs, saturating drives at 100= %. Then vdev 3 completes, then the Hitachi drives of vdev 1 complete with t= he Seagate drive writing some more and then for few more seconds, only vdev= 2 drives are writing. It seems the amount of data is the same, just vdev 2= writes the data slower. However, drives in vdev 2 and vdev 3 are the same.= They should have the same performance characteristics (and as long as the = drives are not 100% saturated, all vdevs complete more or less at the same = time). At other times, some other vdev would complete last -- it is never t= he same vdev that is 'slow'. Could this be DDT/metadata specific issue? Is the DDT/metadata vdev-specifi= c? The pool initially had only two vdevs and after vdev 3 was added, most o= f the written data had no dedup enabled. Also, the ZIL was added later and = initial metadata could be fragmented. But.. why should this affect writing?= The zpool is indeed pretty full, but performance should degrade for all vd= evs (which are more or less equally full). Daniel _______________________________________________ freebsd-fs@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-fs To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"