From owner-freebsd-fs@FreeBSD.ORG Tue Jun 8 00:32:20 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1366A106567B for ; Tue, 8 Jun 2010 00:32:20 +0000 (UTC) (envelope-from brad@duttonbros.com) Received: from uno.mnl.com (uno.mnl.com [64.221.209.136]) by mx1.freebsd.org (Postfix) with ESMTP id C80F28FC0C for ; Tue, 8 Jun 2010 00:32:19 +0000 (UTC) Received: from uno.mnl.com (localhost [127.0.0.1]) by uno.mnl.com (Postfix) with ESMTP id 027DE1A2B; Mon, 7 Jun 2010 17:32:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=duttonbros.com; h= message-id:date:from:to:cc:subject:references:in-reply-to :mime-version:content-type:content-transfer-encoding; s=mail; bh=l7lEN0CgegfYT9yurIpXOq9Yu+4=; b=tZyAKeXj1DuHQJCEdl6Rx3+l6ACd KbtuAw2KN9SXGd1ZXhy2uFndD2MLrDKO5CqNZDGJwETDRgiPSIz0jfKPf+uRVj4x aOsrSXWUjorHSWIAa79pTTLTDHGBatHhf0l+BMb/uw/6aYyexh8C43PVt0f+KOTo ScfxdmzTbZC+o2Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=duttonbros.com; h=message-id :date:from:to:cc:subject:references:in-reply-to:mime-version :content-type:content-transfer-encoding; q=dns; s=mail; b=LONbBj Ys6vyu0eQ7KcvkUUylJnXVXhMBtWqlzAz9aBRZT/aMso/3IaijZwZgXHwBh4bm6T GCRc8qi+2wcQLsLGYH4/2/yDoLBiO2VwZwEvzGUAVC408u1j3iH+Iaip4ODSWZfr cuNmZ6JNRZO+ZEvUZRP7By2z5b2VRXrP+wFkE= Received: from localhost (localhost [127.0.0.1]) by uno.mnl.com (Postfix) with ESMTP id E1E821A29; Mon, 7 Jun 2010 17:32:18 -0700 (PDT) Received: from noah.mnl.com (noah.mnl.com [192.168.0.31]) by duttonbros.com (Horde Framework) with HTTP; Mon, 07 Jun 2010 17:32:18 -0700 Message-ID: <20100607173218.11716iopp083dbpu@duttonbros.com> Date: Mon, 07 Jun 2010 17:32:18 -0700 From: "Bradley W. Dutton" To: Bob Friesenhahn References: <20100607154256.941428ovaq2hha0g@duttonbros.com> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Internet Messaging Program (IMP) H3 (4.3.7) / FreeBSD-8.1 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS performance of various vdevs (long post) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Jun 2010 00:32:20 -0000 Quoting Bob Friesenhahn : > On Mon, 7 Jun 2010, Bradley W. Dutton wrote: >> So the normal vdev performs closest to raw drive speeds. Raidz1 is >> slower and raidz2 even more so. This is observable in the dd tests >> and viewing in gstat. Any ideas why the raid numbers are slower? >> I've tried to account for the fact that the raid vdevs have fewer >> data disks. Would a faster CPU help here? > > The sequential throughput on your new drives is faster than the old > drives, but it is likely that the seek and rotational latencies are > longer. ZFS is transaction-oriented and must tell all the drives to > sync their write cache before proceeding to the next transaction > group. Drives with more latency will slow down this step. > Likewise, ZFS always reads and writes full filesystem blocks > (default 128K) and this may cause more overhead when using raidz. The details are little lacking on the Hitachi site but the HDS722020ALA330 says 8.2 seek time. http://www.hitachigst.com/tech/techlib.nsf/techdocs/5F2DC3B35EA0311386257634000284AD/$file/USA7K2000_DS7K2000_OEMSpec_r1.2.pdf The WDC drives say 8.9 so we should be in the same ballpark on seek times. http://www.wdc.com/en/products/products.asp?driveid=399 I thought the NCQ vs no NCQ might tip the scales in favor of the Hitachi array as well. Are there any tools to check the latencies of the disks? > Using 'dd' from /dev/zero is not a very good benchmark test since > zfs could potentially compress zero-filled blocks down to just a few > bytes (I think recent versions of zfs do this) and of course Unix > supports files with holes. I know it's pretty simple but for checking throughput I thought it would be ok. I don't have compression on and based on the drive lights and gstat, the drives definitely aren't idle. > The higher CPU usage might be due to the device driver or the > interface card being used. Definitely a plausible explanation. If this was the case would the 8 parallel dd processes exhibit the same behavior? or is the type of IO affecting how much CPU the driver is using? > If you could afford to do so, you will likely see considerably > better performance by using mirrors instead of raidz since then 128K > blocks will be sent to each disk and with fewer seeks. I agree with you but at this poing I value the extra space more as I don't have a lot of random IO. I read the following and decided to stick with raidz2 when ditching my old raidz1 setup: http://blogs.sun.com/roch/entry/when_to_and_not_to Thanks for the feedback, Brad