Date: Thu, 30 Jan 2014 10:45:20 -0700 From: markham breitbach <markham_breitbach@ssimicro.com> To: Steven Hartland <killing@multiplay.co.uk>, freebsd-performance@freebsd.org Subject: Re: ZFS read performance Message-ID: <52EA8FB0.7060106@ssimicro.com> In-Reply-To: <2131FB3E08A74989B7A508D0B364A7E8@multiplay.co.uk> References: <52E84517.9010606@ssimicro.com> <2131FB3E08A74989B7A508D0B364A7E8@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Steve, At this point, I am just doing some evaluation with what's on-hand. Ultimately I would add RAM and redundancy, etc. bs was definitely a big part of the issue. (*forehead slap*) hw-mfi.max_cmds=-1 also make a significant impact. Onwards to some more strenuous testing... Thanks for the help! -Markham On 14-01-28 6:34 PM, Steven Hartland wrote: > First off avoid using mfi its a RAID card which adds a extra layer > you dont want even in jbob mode. > > Whats the mfi max_cmd set to? If you havent already set it to -1 > /boot/loader.conf (hw.mfi.max_cmds=-1) which means controller max. > > Next 4GB on a machine avoid it if possible, thats whats disabling > prefetch by default, I'd recommend a min of 8Gb more if posslbe. > > When you "add" a disk your creating a stripe which is bad news if > any single disk fails, so avoid that at all costs. > > Finally your issue with read performance is likely your block size, > you set it for write to 1mb (which is reasonable) but you didn't > on read so will be 512b which will likely cpu limit in the dd > process instead of bottlenecking on the disk / FS performance. > > Regards > Steve > ----- Original Message ----- From: "markham breitbach" <markham_breitbach@ssimicro.com> > To: <freebsd-performance@freebsd.org> > Sent: Wednesday, January 29, 2014 12:02 AM > Subject: ZFS read performance > > >> Hi, >> I'm trying to figure out a ZFS read performance issue that I am seeing >> on FreeBSD9.2 (amd64). >> CPU: Intel(R) Xeon(R) CPU E5405 @ 2.00GHz (2000.11-MHz >> K8-class CPU) >> real memory = 4294967296 (4096 MB) >> avail memory = 4059762688 (3871 MB) >> I have an LSI 9240-8i controller. (8ports @ 6GB/s ea, x8 PCIE 2.0) >> with the drives installed JBOD. These are intended for use as a >> storage pool. The main system drives are using the onboard SATA >> controller for the system. >> >> With a single drive formatted UFS, I can achieve ~ 175MBps Read and >> Write speeds using dd as follows: >> WRITE: >> dd if=/dev/zero of=/vol0/test bs=1m count=10000 >> READ >> dd if=/vol0/test of=/dev/null >> When I create a zfs pool with a single drive: >> zpool create vol0 /dev/mfisyspd0 >> using dd again, I can write ~175MBps, but my read speed is only about >> 60MBps. >> I have set >> vfs.zfs.prefetch_disable = 0 >> and this improves reads to about 70MBps, but no much beyond that. >> Now, this gets even more interesting. >> If I add a drive to the pool: >> zpool add vol0 /dev/mfisyspd1 >> My write speeds increase to ~280MBps, but my read speed is still about >> 60MBps. >> Using gstat, I can see that the load is shared pretty equally amongst >> the drives. >> This trend continues with the addition of a 3rd and 4th drive with >> write speeds ramping up to about 500MBps (I didn't test past that), but >> read speeds stuck around 60MBps. >> I have done similar testing under 9.1 making sure to align 4K sectors >> using the gnop trick (I am using WD RED 3TB AF drives) with no change. >> I have looked through the wiki and handbook for tuning and done some >> googling around, but I'm pretty much out of ideas now. Everything I >> have read seems to indicate that ZFS should run fine on 4GB without any >> tuning, and at this point I haven't even begun any serious stress >> testing. >> Any ideas of where to go from here would be greatly appreciated. >> Thanks, >> -Markham >> >> --- >> [1]Markham Breitbach >> Network Operations >> SSi People, Ideas, Technology >> - - - - - - - - - - - - - - - - - - - - - >> +1 867 669 7500 work >> +1 867 669 7510 fax >> [2]markham_breitbach@ssimicro.com >> [3]www.ssimicro.com >> 356B Old Airport Road >> Yellowknife , NT X1A 3T4 >> Canada >> - - - - - - - - - - - - - - - - - - - - - >> Visit some of our other networks >> [4]www.qiniq.com & [5]www.airware.ca >> >> References >> >> 1. http://www.ssimicro.com/ >> 2. mailto:markham_breitbach@ssimicro.com >> 3. http://www.ssimicro.com/ >> 4. http://www.qiniq.com/ >> 5. http://www.airware.ca/ >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >> > > ================================================ > This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or > entity to whom it is addressed. In the event of misdirection, the recipient is > prohibited from using, copying, printing or otherwise disseminating it or any > information contained in it. > In the event of misdirection, illegible or incomplete transmission please telephone +44 > 845 868 1337 > or return the E.mail to postmaster@multiplay.co.uk. >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?52EA8FB0.7060106>