From owner-freebsd-hackers@FreeBSD.ORG Mon Jun 8 19:11:18 2009 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CCD05106566C for ; Mon, 8 Jun 2009 19:11:18 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.30]) by mx1.freebsd.org (Postfix) with ESMTP id 656858FC17 for ; Mon, 8 Jun 2009 19:11:18 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by yw-out-2324.google.com with SMTP id 9so1917692ywe.13 for ; Mon, 08 Jun 2009 12:11:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=iE5JIF3920kUkm3R5+RJaSm+23sJrO9GvPNUvz0yCnQ=; b=ENGkS0Y9zItnTXiuK6qtNKf98OhWQJfq9cwnh5hO+AFjIMtX5aBqGWCVKduktdM4e+ Bb3aVZC8UPIL2xFkqZyTh/so8RkrjXJEQbuMnWVjVVC5DQQsD8CprFNp2fpBipyIkmfm QWHKz61RSwfUdrcPa7It+CwK7O2ccF5sBNqy0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=r7kZ0x0TMukJqCeHmvR8jc+S9KlqdGFp/CkWK03bzyxkCiX1uWtLUW1Mn9NDPxsqGq +l5VQNy1NbuV9kg3uK0Xj+RBCybOnBE56LVGSFkviPDmeP/i/q4kfJdronA0y0Rv0IUf akXl+0FrwdDY+XMpVg9NNjLJey6nzhEI40yhk= MIME-Version: 1.0 Received: by 10.151.38.19 with SMTP id q19mr13238060ybj.76.1244488277555; Mon, 08 Jun 2009 12:11:17 -0700 (PDT) In-Reply-To: References: <20090530175239.GA25604@logik.internal.network> <20090530144354.2255f722@bhuda.mired.org> <20090530191840.GA68514@logik.internal.network> <20090530162744.5d77e9d1@bhuda.mired.org> Date: Mon, 8 Jun 2009 12:11:17 -0700 Message-ID: From: Freddie Cash To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Request for opinions - gvinum or ccd? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jun 2009 19:11:19 -0000 On Sun, Jun 7, 2009 at 12:01 AM, Wojciech Puchar< wojtek@wojtek.tensor.gdynia.pl> wrote: >>> (very roughly, in the non-sequential access case) expected to deliver >>> performance of four drives in a RAID0 array? >> >> According to all the Sun documentation, the I/O throughput of a raidz >> configuration is equal to that of a single drive. > > exactly what i say. it's like RAID3. Not RAID5 which have close to n times > single drive throughput on read and rougly n/4 on writes. > >> We remade the pool using 3x 8-drive raidz2 vdevs, and performance has >> been great (400 MBytes/s write, almost 3 GBytes/s sequential read, 800 > > why write performance is so slow? in Sun theory it should have the same > speed as reads. I would say that it should be even better a bit - filesystem > get data first in cache and can plan ahead. > >> MBytes/s random read). > > random read on how big chunks? > > Are you sure you get 3GB/s on read? it would mean each drive must be able to > do 140MB/s > > What disks do you use? 12x 500 GB Seagate EL2 SATA drives, part of their enterprise near-line storage line. 12x 500 GB WD SATA drives, generic off-the-shelf drives I re-ran the iozone tests, letting them run to completion, and here are the results: The iozone command: iozone -M -e -+u -T -t -r 128k -s 40960 -i 0 -i 1 -i 2 -i 8 -+p 70 -C I ran the command using 32, 64, 128, and 256 for Write speeds range from 236 MBytes/sec to 582 MBytes/sec for sequential; and from 242 MBytes/sec to 550 MBytes/sec for random. Read speeds range from 3.3 GBytes/sec to 5.5 GBytes/sec for sequential; and from 1.8 GBytes/sec to 5.5 GBytes/sec for random. All the gory details are below. 32-threads: Children see ... 32 initial writers = 582468.13 KB/sec 32-threads: Parent sees ... 32 initial writers = 108808.46 KB/sec 64-threads: Children see ... 64 initial writers = 236144.47 KB/sec 64-threads: Parent sees ... 64 initial writers = 86942.94 KB/sec 128-threads: Children see ... 128 initial writers = 284706.68 KB/sec 128-threads: Parent sees ... 128 initial writers = 10850.40 KB/sec 256-threads: Children see ... 256 initial writers = 258260.59 KB/sec 256-threads: Parent sees ... 256 initial writers = 9882.16 KB/sec 32-threads: Children see ... 32 rewriters = 545347.52 KB/sec 32-threads: Parent sees ... 32 rewriters = 339308.08 KB/sec 64-threads: Children see ... 64 rewriters = 419838.51 KB/sec 64-threads: Parent sees ... 64 rewriters = 335620.45 KB/sec 128-threads: Children see ... 128 rewriters = 350668.51 KB/sec 128-threads: Parent sees ... 128 rewriters = 319452.97 KB/sec 256-threads: Children see ... 256 rewriters = 317751.52 KB/sec 256-threads: Parent sees ... 256 rewriters = 295579.66 KB/sec 32-threads: Children see ... 32 random writers = 379256.37 KB/sec 32-threads: Parent sees ... 32 random writers = 95298.44 KB/sec 64-threads: Children see ... 64 random writers = 551767.68 KB/sec 64-threads: Parent sees ... 64 random writers = 113397.95 KB/sec 128-threads: Children see ... 128 random writers = 241980.60 KB/sec 128-threads: Parent sees ... 128 random writers = 74584.01 KB/sec 256-threads: Children see ... 256 random writers = 398427.84 KB/sec 256-threads: Parent sees ... 256 random writers = 20219.56 KB/sec 32-threads: Children see ... 32 readers = 5023742.86 KB/sec 32-threads: Parent sees ... 32 readers = 4661309.72 KB/sec 64-threads: Children see ... 64 readers = 5516460.71 KB/sec 64-threads: Parent sees ... 64 readers = 3949337.61 KB/sec 128-threads: Children see ... 128 readers = 4748635.74 KB/sec 128-threads: Parent sees ... 128 readers = 3208982.03 KB/sec 256-threads: Children see ... 256 readers = 4358453.38 KB/sec 256-threads: Parent sees ... 256 readers = 2741593.08 KB/sec 32-threads: Children see ... 32 re-readers = 5502926.62 KB/sec 32-threads: Parent sees ... 32 re-readers = 4650327.75 KB/sec 64-threads: Children see ... 64 re-readers = 5509400.02 KB/sec 64-threads: Parent sees ... 64 re-readers = 4526444.40 KB/sec 128-threads: Children see ... 128 re-readers = 4072363.55 KB/sec 128-threads: Parent sees ... 128 re-readers = 2840317.47 KB/sec 256-threads: Children see ... 256 re-readers = 3329375.95 KB/sec 256-threads: Parent sees ... 256 re-readers = 2183894.33 KB/sec 32-threads: Children see ... 32 random readers = 5555090.45 KB/sec 32-threads: Parent sees ... 32 random readers = 4602383.62 KB/sec 64-threads: Children see ... 64 random readers = 4402270.77 KB/sec 64-threads: Parent sees ... 64 random readers = 2059081.52 KB/sec 128-threads: Children see ... 128 random readers = 3070466.93 KB/sec 128-threads: Parent sees ... 128 random readers = 525076.11 KB/sec 256-threads: Children see ... 256 random readers = 1888676.12 KB/sec 256-threads: Parent sees ... 256 random readers = 293304.53 KB/sec 32-threads: Children see ... 32 mixed workload = 3130000.18 KB/sec 32-threads: Parent sees ... 32 mixed workload = 123281.78 KB/sec 64-threads: Children see ... 64 mixed workload = 1587053.33 KB/sec 64-threads: Parent sees ... 64 mixed workload = 294586.82 KB/sec 128-threads: Children see ... 128 mixed workload = 807349.95 KB/sec 128-threads: Parent sees ... 128 mixed workload = 98998.77 KB/sec 256-threads: Children see ... 256 mixed workload = 393469.55 KB/sec 256-threads: Parent sees ... 256 mixed workload = 112394.90 KB/sec -- Freddie Cash fjwcash@gmail.com