From owner-freebsd-performance@FreeBSD.ORG Wed Jan 29 00:18:51 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EC3262D9 for ; Wed, 29 Jan 2014 00:18:51 +0000 (UTC) Received: from mail.ssimicro.com (mail.ssimicro.com [64.247.129.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 59836194F for ; Wed, 29 Jan 2014 00:18:50 +0000 (UTC) Received: from markham.ssimicro.com (markham.ssimicro.com [64.247.130.99]) by mail.ssimicro.com (8.14.7/8.14.7) with ESMTP id s0T02VC9023344 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NOT) for ; Tue, 28 Jan 2014 17:02:32 -0700 (MST) Message-ID: <52E84517.9010606@ssimicro.com> Date: Tue, 28 Jan 2014 17:02:31 -0700 From: markham breitbach User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 To: freebsd-performance@freebsd.org Subject: ZFS read performance X-Enigmail-Version: 1.6 MIME-Version: 1.0 Content-Type: text/plain; charset="ISO-8859-1" X-Content-Filtered-By: Mailman/MimeDel 2.1.17 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jan 2014 00:18:52 -0000 Hi, I'm trying to figure out a ZFS read performance issue that I am seeing on FreeBSD9.2 (amd64). CPU: Intel(R) Xeon(R) CPU E5405 @ 2.00GHz (2000.11-MHz K8-class CPU) real memory = 4294967296 (4096 MB) avail memory = 4059762688 (3871 MB) I have an LSI 9240-8i controller. (8ports @ 6GB/s ea, x8 PCIE 2.0) with the drives installed JBOD. These are intended for use as a storage pool. The main system drives are using the onboard SATA controller for the system. With a single drive formatted UFS, I can achieve ~ 175MBps Read and Write speeds using dd as follows: WRITE: dd if=/dev/zero of=/vol0/test bs=1m count=10000 READ dd if=/vol0/test of=/dev/null When I create a zfs pool with a single drive: zpool create vol0 /dev/mfisyspd0 using dd again, I can write ~175MBps, but my read speed is only about 60MBps. I have set vfs.zfs.prefetch_disable = 0 and this improves reads to about 70MBps, but no much beyond that. Now, this gets even more interesting. If I add a drive to the pool: zpool add vol0 /dev/mfisyspd1 My write speeds increase to ~280MBps, but my read speed is still about 60MBps. Using gstat, I can see that the load is shared pretty equally amongst the drives. This trend continues with the addition of a 3rd and 4th drive with write speeds ramping up to about 500MBps (I didn't test past that), but read speeds stuck around 60MBps. I have done similar testing under 9.1 making sure to align 4K sectors using the gnop trick (I am using WD RED 3TB AF drives) with no change. I have looked through the wiki and handbook for tuning and done some googling around, but I'm pretty much out of ideas now. Everything I have read seems to indicate that ZFS should run fine on 4GB without any tuning, and at this point I haven't even begun any serious stress testing. Any ideas of where to go from here would be greatly appreciated. Thanks, -Markham --- [1]Markham Breitbach Network Operations SSi People, Ideas, Technology - - - - - - - - - - - - - - - - - - - - - +1 867 669 7500 work +1 867 669 7510 fax [2]markham_breitbach@ssimicro.com [3]www.ssimicro.com 356B Old Airport Road Yellowknife , NT X1A 3T4 Canada - - - - - - - - - - - - - - - - - - - - - Visit some of our other networks [4]www.qiniq.com & [5]www.airware.ca References 1. http://www.ssimicro.com/ 2. mailto:markham_breitbach@ssimicro.com 3. http://www.ssimicro.com/ 4. http://www.qiniq.com/ 5. http://www.airware.ca/ From owner-freebsd-performance@FreeBSD.ORG Wed Jan 29 01:34:24 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BE803B71 for ; Wed, 29 Jan 2014 01:34:24 +0000 (UTC) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 599DF1EB6 for ; Wed, 29 Jan 2014 01:34:24 +0000 (UTC) Received: from r2d2 ([82.69.141.170]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50007980652.msg for ; Wed, 29 Jan 2014 01:34:10 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 29 Jan 2014 01:34:10 +0000 (not processed: message from valid local sender) X-MDDKIM-Result: neutral (mail1.multiplay.co.uk) X-MDRemoteIP: 82.69.141.170 X-Return-Path: prvs=110664dd82=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk X-MDaemon-Deliver-To: freebsd-performance@freebsd.org Message-ID: <2131FB3E08A74989B7A508D0B364A7E8@multiplay.co.uk> From: "Steven Hartland" To: "markham breitbach" , References: <52E84517.9010606@ssimicro.com> Subject: Re: ZFS read performance Date: Wed, 29 Jan 2014 01:34:06 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Jan 2014 01:34:24 -0000 First off avoid using mfi its a RAID card which adds a extra layer you dont want even in jbob mode. Whats the mfi max_cmd set to? If you havent already set it to -1 /boot/loader.conf (hw.mfi.max_cmds=-1) which means controller max. Next 4GB on a machine avoid it if possible, thats whats disabling prefetch by default, I'd recommend a min of 8Gb more if posslbe. When you "add" a disk your creating a stripe which is bad news if any single disk fails, so avoid that at all costs. Finally your issue with read performance is likely your block size, you set it for write to 1mb (which is reasonable) but you didn't on read so will be 512b which will likely cpu limit in the dd process instead of bottlenecking on the disk / FS performance. Regards Steve ----- Original Message ----- From: "markham breitbach" To: Sent: Wednesday, January 29, 2014 12:02 AM Subject: ZFS read performance > Hi, > I'm trying to figure out a ZFS read performance issue that I am seeing > on FreeBSD9.2 (amd64). > CPU: Intel(R) Xeon(R) CPU E5405 @ 2.00GHz (2000.11-MHz > K8-class CPU) > real memory = 4294967296 (4096 MB) > avail memory = 4059762688 (3871 MB) > I have an LSI 9240-8i controller. (8ports @ 6GB/s ea, x8 PCIE 2.0) > with the drives installed JBOD. These are intended for use as a > storage pool. The main system drives are using the onboard SATA > controller for the system. > > With a single drive formatted UFS, I can achieve ~ 175MBps Read and > Write speeds using dd as follows: > WRITE: > dd if=/dev/zero of=/vol0/test bs=1m count=10000 > READ > dd if=/vol0/test of=/dev/null > When I create a zfs pool with a single drive: > zpool create vol0 /dev/mfisyspd0 > using dd again, I can write ~175MBps, but my read speed is only about > 60MBps. > I have set > vfs.zfs.prefetch_disable = 0 > and this improves reads to about 70MBps, but no much beyond that. > Now, this gets even more interesting. > If I add a drive to the pool: > zpool add vol0 /dev/mfisyspd1 > My write speeds increase to ~280MBps, but my read speed is still about > 60MBps. > Using gstat, I can see that the load is shared pretty equally amongst > the drives. > This trend continues with the addition of a 3rd and 4th drive with > write speeds ramping up to about 500MBps (I didn't test past that), but > read speeds stuck around 60MBps. > I have done similar testing under 9.1 making sure to align 4K sectors > using the gnop trick (I am using WD RED 3TB AF drives) with no change. > I have looked through the wiki and handbook for tuning and done some > googling around, but I'm pretty much out of ideas now. Everything I > have read seems to indicate that ZFS should run fine on 4GB without any > tuning, and at this point I haven't even begun any serious stress > testing. > Any ideas of where to go from here would be greatly appreciated. > Thanks, > -Markham > > --- > [1]Markham Breitbach > Network Operations > SSi People, Ideas, Technology > - - - - - - - - - - - - - - - - - - - - - > +1 867 669 7500 work > +1 867 669 7510 fax > [2]markham_breitbach@ssimicro.com > [3]www.ssimicro.com > 356B Old Airport Road > Yellowknife , NT X1A 3T4 > Canada > - - - - - - - - - - - - - - - - - - - - - > Visit some of our other networks > [4]www.qiniq.com & [5]www.airware.ca > > References > > 1. http://www.ssimicro.com/ > 2. mailto:markham_breitbach@ssimicro.com > 3. http://www.ssimicro.com/ > 4. http://www.qiniq.com/ > 5. http://www.airware.ca/ > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk. From owner-freebsd-performance@FreeBSD.ORG Thu Jan 30 17:45:31 2014 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6B53687F for ; Thu, 30 Jan 2014 17:45:31 +0000 (UTC) Received: from mail.ssimicro.com (mail.ssimicro.com [64.247.129.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 07E901356 for ; Thu, 30 Jan 2014 17:45:30 +0000 (UTC) Received: from markham.ssimicro.com (markham.ssimicro.com [64.247.130.99]) by mail.ssimicro.com (8.14.7/8.14.7) with ESMTP id s0UHjKpm043214 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NOT); Thu, 30 Jan 2014 10:45:22 -0700 (MST) Message-ID: <52EA8FB0.7060106@ssimicro.com> Date: Thu, 30 Jan 2014 10:45:20 -0700 From: markham breitbach User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: Steven Hartland , freebsd-performance@freebsd.org Subject: Re: ZFS read performance References: <52E84517.9010606@ssimicro.com> <2131FB3E08A74989B7A508D0B364A7E8@multiplay.co.uk> In-Reply-To: <2131FB3E08A74989B7A508D0B364A7E8@multiplay.co.uk> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Jan 2014 17:45:31 -0000 Hi Steve, At this point, I am just doing some evaluation with what's on-hand. Ultimately I would add RAM and redundancy, etc. bs was definitely a big part of the issue. (*forehead slap*) hw-mfi.max_cmds=-1 also make a significant impact. Onwards to some more strenuous testing... Thanks for the help! -Markham On 14-01-28 6:34 PM, Steven Hartland wrote: > First off avoid using mfi its a RAID card which adds a extra layer > you dont want even in jbob mode. > > Whats the mfi max_cmd set to? If you havent already set it to -1 > /boot/loader.conf (hw.mfi.max_cmds=-1) which means controller max. > > Next 4GB on a machine avoid it if possible, thats whats disabling > prefetch by default, I'd recommend a min of 8Gb more if posslbe. > > When you "add" a disk your creating a stripe which is bad news if > any single disk fails, so avoid that at all costs. > > Finally your issue with read performance is likely your block size, > you set it for write to 1mb (which is reasonable) but you didn't > on read so will be 512b which will likely cpu limit in the dd > process instead of bottlenecking on the disk / FS performance. > > Regards > Steve > ----- Original Message ----- From: "markham breitbach" > To: > Sent: Wednesday, January 29, 2014 12:02 AM > Subject: ZFS read performance > > >> Hi, >> I'm trying to figure out a ZFS read performance issue that I am seeing >> on FreeBSD9.2 (amd64). >> CPU: Intel(R) Xeon(R) CPU E5405 @ 2.00GHz (2000.11-MHz >> K8-class CPU) >> real memory = 4294967296 (4096 MB) >> avail memory = 4059762688 (3871 MB) >> I have an LSI 9240-8i controller. (8ports @ 6GB/s ea, x8 PCIE 2.0) >> with the drives installed JBOD. These are intended for use as a >> storage pool. The main system drives are using the onboard SATA >> controller for the system. >> >> With a single drive formatted UFS, I can achieve ~ 175MBps Read and >> Write speeds using dd as follows: >> WRITE: >> dd if=/dev/zero of=/vol0/test bs=1m count=10000 >> READ >> dd if=/vol0/test of=/dev/null >> When I create a zfs pool with a single drive: >> zpool create vol0 /dev/mfisyspd0 >> using dd again, I can write ~175MBps, but my read speed is only about >> 60MBps. >> I have set >> vfs.zfs.prefetch_disable = 0 >> and this improves reads to about 70MBps, but no much beyond that. >> Now, this gets even more interesting. >> If I add a drive to the pool: >> zpool add vol0 /dev/mfisyspd1 >> My write speeds increase to ~280MBps, but my read speed is still about >> 60MBps. >> Using gstat, I can see that the load is shared pretty equally amongst >> the drives. >> This trend continues with the addition of a 3rd and 4th drive with >> write speeds ramping up to about 500MBps (I didn't test past that), but >> read speeds stuck around 60MBps. >> I have done similar testing under 9.1 making sure to align 4K sectors >> using the gnop trick (I am using WD RED 3TB AF drives) with no change. >> I have looked through the wiki and handbook for tuning and done some >> googling around, but I'm pretty much out of ideas now. Everything I >> have read seems to indicate that ZFS should run fine on 4GB without any >> tuning, and at this point I haven't even begun any serious stress >> testing. >> Any ideas of where to go from here would be greatly appreciated. >> Thanks, >> -Markham >> >> --- >> [1]Markham Breitbach >> Network Operations >> SSi People, Ideas, Technology >> - - - - - - - - - - - - - - - - - - - - - >> +1 867 669 7500 work >> +1 867 669 7510 fax >> [2]markham_breitbach@ssimicro.com >> [3]www.ssimicro.com >> 356B Old Airport Road >> Yellowknife , NT X1A 3T4 >> Canada >> - - - - - - - - - - - - - - - - - - - - - >> Visit some of our other networks >> [4]www.qiniq.com & [5]www.airware.ca >> >> References >> >> 1. http://www.ssimicro.com/ >> 2. mailto:markham_breitbach@ssimicro.com >> 3. http://www.ssimicro.com/ >> 4. http://www.qiniq.com/ >> 5. http://www.airware.ca/ >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >> > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or > entity to whom it is addressed. In the event of misdirection, the recipient is > prohibited from using, copying, printing or otherwise disseminating it or any > information contained in it. > In the event of misdirection, illegible or incomplete transmission please telephone +44 > 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. >