From owner-freebsd-fs@FreeBSD.ORG Sun Jan 24 18:12:36 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 84F93106568B for ; Sun, 24 Jan 2010 18:12:36 +0000 (UTC) (envelope-from bfriesen@simple.dallas.tx.us) Received: from blade.simplesystems.org (blade.simplesystems.org [65.66.246.74]) by mx1.freebsd.org (Postfix) with ESMTP id 14B7D8FC21 for ; Sun, 24 Jan 2010 18:12:35 +0000 (UTC) Received: from freddy.simplesystems.org (freddy.simplesystems.org [65.66.246.65]) by blade.simplesystems.org (8.13.8+Sun/8.13.8) with ESMTP id o0OICYOP020782; Sun, 24 Jan 2010 12:12:35 -0600 (CST) Date: Sun, 24 Jan 2010 12:12:34 -0600 (CST) From: Bob Friesenhahn X-X-Sender: bfriesen@freddy.simplesystems.org To: Dan Naumov In-Reply-To: Message-ID: References: <883b2dc51001240905r4cfbf830i3b9b400969ac261b@mail.gmail.com> User-Agent: Alpine 2.01 (GSO 1266 2009-07-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (blade.simplesystems.org [65.66.246.90]); Sun, 24 Jan 2010 12:12:35 -0600 (CST) Cc: freebsd-fs@freebsd.org Subject: Re: 8.0-RELEASE/amd64 - full ZFS install - low read and write disk performance X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jan 2010 18:12:36 -0000 On Sun, 24 Jan 2010, Dan Naumov wrote: > > This works out to 1GB in 36,2 seconds / 28,2mb/s in the first test and > 4GB in 143.8 seconds / 28,4mb/s and somewhat consistent with the > bonnie results. It also sadly seems to confirm the very slow speed :( > The disks are attached to a 4-port Sil3124 controller and again, my > Windows benchmarks showing 65mb/s+ were done on exact same machine, > with same disks attached to the same controller. Only difference was > that in Windows the disks weren't in a mirror configuration but were > tested individually. I do understand that a mirror setup offers > roughly the same write speed as individual disk, while the read speed > usually varies from "equal to individual disk speed" to "nearly the > throughput of both disks combined" depending on the implementation, > but there is no obvious reason I am seeing why my setup offers both > read and write speeds roughly 1/3 to 1/2 of what the individual disks > are capable of. Dmesg shows: There is a mistatement in the above in that a "mirror setup offers roughly the same write speed as individual disk". It is possible for a mirror setup to offer a similar write speed to an individual disk, but it is also quite possible to get 1/2 (or even 1/3) the speed. ZFS writes to a mirror pair requires two independent writes. If these writes go down independent I/O paths, then there is hardly any overhead from the 2nd write. If the writes go through a bandwidth-limited shared path then they will contend for that bandwidth and you will see much less write performance. As a simple test, you can temporarily remove the mirror device from the pool and see if the write performance dramatically improves. Before doing that, it is useful to see the output of 'iostat -x 30' while under heavy write load to see if one device shows a much higher svc_t value than the other. > expectations, I was seeing 65-70mb/s. Note: again, exact same > hardware, exact same disks attached to the exact same controller. In > my knowledge, Solaris/OpenSolaris has an issue where they have to > automatically disable disk cache if ZFS is used on top of partitions > instead of raw disks, but to my knowledge (I recall reading this from > multiple reputable sources) this issue does not affect FreeBSD. Does FreeBSD always enable the disk write caches? Bob -- Bob Friesenhahn bfriesen@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/