From owner-freebsd-performance@FreeBSD.ORG Wed Jan 29 00:18:51 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EC3262D9 for ; Wed, 29 Jan 2014 00:18:51 +0000 (UTC) Received: from mail.ssimicro.com (mail.ssimicro.com [64.247.129.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 59836194F for ; Wed, 29 Jan 2014 00:18:50 +0000 (UTC) Received: from markham.ssimicro.com (markham.ssimicro.com [64.247.130.99]) by mail.ssimicro.com (8.14.7/8.14.7) with ESMTP id s0T02VC9023344 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NOT) for ; Tue, 28 Jan 2014 17:02:32 -0700 (MST) Message-ID: <52E84517.9010606@ssimicro.com> Date: Tue, 28 Jan 2014 17:02:31 -0700 From: markham breitbach User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 To: freebsd-performance@freebsd.org Subject: ZFS read performance X-Enigmail-Version: 1.6 MIME-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jan 2014 00:18:52 -0000 Hi, I'm trying to figure out a ZFS read performance issue that I am seeing on FreeBSD9.2 (amd64). CPU: Intel(R) Xeon(R) CPU E5405 @ 2.00GHz (2000.11-MHz K8-class CPU) real memory = 4294967296 (4096 MB) avail memory = 4059762688 (3871 MB) I have an LSI 9240-8i controller. (8ports @ 6GB/s ea, x8 PCIE 2.0) with the drives installed JBOD. These are intended for use as a storage pool. The main system drives are using the onboard SATA controller for the system. With a single drive formatted UFS, I can achieve ~ 175MBps Read and Write speeds using dd as follows: WRITE: dd if=/dev/zero of=/vol0/test bs=1m count=10000 READ dd if=/vol0/test of=/dev/null When I create a zfs pool with a single drive: zpool create vol0 /dev/mfisyspd0 using dd again, I can write ~175MBps, but my read speed is only about 60MBps. I have set vfs.zfs.prefetch_disable = 0 and this improves reads to about 70MBps, but no much beyond that. Now, this gets even more interesting. If I add a drive to the pool: zpool add vol0 /dev/mfisyspd1 My write speeds increase to ~280MBps, but my read speed is still about 60MBps. Using gstat, I can see that the load is shared pretty equally amongst the drives. This trend continues with the addition of a 3rd and 4th drive with write speeds ramping up to about 500MBps (I didn't test past that), but read speeds stuck around 60MBps. I have done similar testing under 9.1 making sure to align 4K sectors using the gnop trick (I am using WD RED 3TB AF drives) with no change. I have looked through the wiki and handbook for tuning and done some googling around, but I'm pretty much out of ideas now. Everything I have read seems to indicate that ZFS should run fine on 4GB without any tuning, and at this point I haven't even begun any serious stress testing. Any ideas of where to go from here would be greatly appreciated. Thanks, -Markham --- [1]Markham Breitbach Network Operations SSi People, Ideas, Technology - - - - - - - - - - - - - - - - - - - - - +1 867 669 7500 work +1 867 669 7510 fax [2]markham_breitbach@ssimicro.com [3]www.ssimicro.com 356B Old Airport Road Yellowknife , NT X1A 3T4 Canada - - - - - - - - - - - - - - - - - - - - - Visit some of our other networks [4]www.qiniq.com & [5]www.airware.ca References 1. http://www.ssimicro.com/ 2. mailto:markham_breitbach@ssimicro.com 3. http://www.ssimicro.com/ 4. http://www.qiniq.com/ 5. http://www.airware.ca/