From owner-freebsd-questions@FreeBSD.ORG Fri Sep 16 14:18:41 2011 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 970BA106566B for ; Fri, 16 Sep 2011 14:18:41 +0000 (UTC) (envelope-from freebsd@top-consulting.net) Received: from smtp.emailarray.com (mail.emailarray.com [69.28.212.222]) by mx1.freebsd.org (Postfix) with ESMTP id 39E698FC1D for ; Fri, 16 Sep 2011 14:18:40 +0000 (UTC) Received: (qmail 68480 invoked by uid 89); 16 Sep 2011 14:18:39 -0000 Received: from unknown (HELO localhost) (freebsd@top-consulting.net@192.168.1.6) (POLARISLOCAL) by smtp.emailarray.com with (DHE-RSA-AES256-SHA encrypted) SMTP; 16 Sep 2011 14:18:38 -0000 Received: from modemcable238.94-200-24.mc.videotron.ca (modemcable238.94-200-24.mc.videotron.ca [24.200.94.238]) by mail.top-consulting.net (Horde Framework) with HTTP; Fri, 16 Sep 2011 10:18:33 -0400 Message-ID: <20110916101833.17485ybnq5srjbc4@mail.top-consulting.net> Date: Fri, 16 Sep 2011 10:18:33 -0400 From: freebsd@top-consulting.net Cc: freebsd-questions@freebsd.org References: <20110916063153.200375qdq59crf8c@mail.top-consulting.net> <32990703-D068-4B0D-AF3A-C1E6EA0A4100@elde.net> In-Reply-To: <32990703-D068-4B0D-AF3A-C1E6EA0A4100@elde.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) X-Originating-IP: 24.200.94.238 X-DSPAM-Result: Innocent X-DSPAM-Processed: Fri Sep 16 10:18:39 2011 X-DSPAM-Confidence: 0.9419 X-DSPAM-Improbability: 1 in 1623 chance of being spam X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 1,4e735abf11847029613501 X-DSPAM-Factors: 27, net>+>, 0.01000, the+setup, 0.01000, net+wrote, 0.01000, is+configured, 0.01000, >+two, 0.01000, Note+you, 0.01000, p9, 0.01000, trouble+if, 0.01000, >+On, 0.02929, commands+that, 0.06134, (because, 0.06134, surprised+if, 0.06134, wrote, 0.06330, p12, 0.06384, to+it's, 0.06950, two+small, 0.08225, in+trouble, 0.08523, up+I, 0.09383, FreeBSD, 0.09467, a+fairly, 0.09656, Subject*Re, 0.09826, setup, 0.09893, at+12, 0.10225, synchronous, 0.10409, Subject*(, 0.10954, in+14, 0.11560, >+Note, 0.11560 X-PolarisMail-Flags: x Subject: Re: FS of choice for max random iops ( Maildir ) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Sep 2011 14:18:41 -0000 Quoting Terje Elde : > On 16. sep. 2011, at 12:31, freebsd@top-consulting.net wrote: >> Right now I defined an entire array of 8TB ( all 16 disks ) >> separated in two pieces. 50 GB for FreeBSD to boot and the rest >> available to configure as storage. > > ZFS will want to write to it's ZIL (zfs intent log) before writing > to the final location of the data. Even if you're not waiting for > the ZIL-write to disk (because of the controller ram), those writes > will probably make it through to disk. That gives you twice as many > writes to disk, and a lot more seek. > > If you want to take zfs for a proper spin, I'd like to sugget adding > two small SSDs to the setup, mirrored by zfs. You can use those both > for the ZIL, and also as cache, for the array. That's a fairly small > investment these days, and I would be surprised if it didn't > significantly improve performance, both for your benchmark, and real > load. > > Note: you might be in trouble if you loose your ZIL, thus the > doubling up. I *think* you can SSD a cache without risking dataloss, > but don't take my word for it. > > Terje Well, I tried disabling the ZIL on a new dataset. These are the commands that I ran: zpool create data da1 zfs create data/maildomains zfs set sync=disabled data/maildomains dd bs=1024 if=/dev/zero of=/data/maildomains/t1 count=1M 1048576+0 records in 1048576+0 records out 1073741824 bytes transferred in 14.537711 secs (73859071 bytes/sec) Got a measly 74MB/sec. On the UFS partition however... dd bs=1024 if=/dev/zero of=/usr/t1 count=1M 1048576+0 records in 1048576+0 records out 1073741824 bytes transferred in 5.828395 secs (184225983 bytes/sec) 184MB/sec! And this is synchronous writing, not random! So what is ZFS good for finally ? Synchronous writing or small random iops ? By the way, this is how the array is configured with 3ware: Unit UnitType Status %RCmpl %V/I/M Stripe Size(GB) Cache AVrfy ------------------------------------------------------------------------------ u0 RAID-10 OK - - 64K 7450.5 ON ON VPort Status Unit Size Type Phy Encl-Slot Model ------------------------------------------------------------------------------ p0 OK u0 931.51 GB SATA 0 - WDC WD1002FBYS-01A6 p1 OK u0 931.51 GB SATA 1 - WDC WD1002FBYS-01A6 p2 OK u0 931.51 GB SATA 2 - WDC WD1002FBYS-01A6 p3 OK u0 931.51 GB SATA 3 - WDC WD1002FBYS-01A6 p4 OK u0 931.51 GB SATA 4 - WDC WD1002FBYS-01A6 p5 OK u0 931.51 GB SATA 5 - WDC WD1002FBYS-01A6 p6 OK u0 931.51 GB SATA 6 - WDC WD1002FBYS-01A6 p7 OK u0 931.51 GB SATA 7 - WDC WD1002FBYS-01A6 p8 OK u0 931.51 GB SATA 8 - WDC WD1002FBYS-01A6 p9 OK u0 931.51 GB SATA 9 - WDC WD1002FBYS-01A6 p10 OK u0 931.51 GB SATA 10 - WDC WD1002FBYS-01A6 p11 OK u0 931.51 GB SATA 11 - WDC WD1002FBYS-01A6 p12 OK u0 931.51 GB SATA 12 - WDC WD1002FBYS-01A6 p13 OK u0 931.51 GB SATA 13 - WDC WD1002FBYS-01A6 p14 OK u0 931.51 GB SATA 14 - WDC WD1002FBYS-01A6 p15 OK u0 931.51 GB SATA 15 - WDC WD1002FBYS-01A6