Date: Fri, 4 Jan 2013 15:06:06 +0100 From: Fleuriot Damien <ml@my.gd> To: "O. Hartmann" <ohartman@zedat.fu-berlin.de> Cc: Current FreeBSD <freebsd-current@FreeBSD.org> Subject: Re: ZFS/RAIDZ and SAMBA: abyssimal performance Message-ID: <1ADC2ECB-70FF-4DDD-9D62-16E2EEECDD8B@my.gd> In-Reply-To: <50E6DE91.7010404@zedat.fu-berlin.de> References: <50E6DE91.7010404@zedat.fu-berlin.de>
next in thread | previous in thread | raw e-mail | index | archive | help
On Jan 4, 2013, at 2:52 PM, "O. Hartmann" <ohartman@zedat.fu-berlin.de> = wrote: > I use a small testing server. The hardware is most modern Intel = hardware > (i3-3220, Z77 chipset), 16GB RAM. The OS is FreeBSD 10.0-CURRENT #1 > r245036M: Fri Jan 4 12:48:53 CET 2013. >=20 > The ZFS subsystem is comprised by 3 Western Digital 3 TB harddrives = (WDC > WD30EZRX-00DC0B0 80.00A80> ATA-9 SATA 3.x device), setup as a ZFS = RAIDZ: >=20 > --- > root@gate [etc] zpool status > pool: ASGARD00 > state: ONLINE > scan: scrub repaired 0 in 1h45m with 0 errors on Sat Dec 1 20:59:44 = 2012 > config: >=20 > NAME STATE READ > WRITE CKSUM > ASGARD00 ONLINE 0 > 0 0 > raidz1-0 ONLINE 0 > 0 0 > gptid/1e716118-1492-11e2-b828-90f6526a24d6 ONLINE 0 > 0 0 > gptid/294a6798-1492-11e2-b828-90f6526a24d6 ONLINE 0 > 0 0 > gptid/30c813f8-1492-11e2-b828-90f6526a24d6 ONLINE 0 > 0 0 > logs > ada0p1 ONLINE 0 > 0 0 > cache > ada0p2 ONLINE 0 > 0 0 >=20 > errors: No known data errors > --- >=20 > The "logs" and "cache" device is a single SAMSUNG 830 SSD, 60 GB > capacity, GPT partinioned, logs (ZIL) has 5GB, cache has ~55 GB. >=20 > I think its not the optimal setup using the very same SSD for both > caching/L2ARC and ZIL, but without the cache device the performance > doen't differ much at the moment. Luckliy, with ZFS I can change the > arrangement as I like. >=20 > The ZFS volumes created on the pool named ASGARD00 are standard, only > options sharenfs/sharesmb/checksum are set to yes. Everthing elese is > set to the defaults. >=20 > In /boot/loader.conf I set the following parameters according to many > (and confusing!) help and suggestions on the web: >=20 > # ZFS > #vfs.zfs.cache_flush_disable=3D1 > # > #vfs.zfs.write_limit_override=3D1073741824 # 1GB > vfs.zfs.l2arc_noprefetch=3D0 > vfs.zfs.l2arc_headroom=3D6 >=20 > The NFSv4 performance (client is also FreeBSD 10.0-CURRENT of the same > date) is moderate to disapointing and doesn't exceed 45 - 55 MB/s > sustained, but here are sometimes "spikes" I can watch with "systat = -vm > 1" reporting 120 MB/s per drive (ada2/ada3/ada4, the 3x 3TB WD drives = in > RAIDZ). I still benchmark via iozone. Both server and client use JUMBO > frames (MTU=3D6120), which gives better throughput compared to the > standard MTU=3D1500. >=20 > The local performance on the server itself is slightly better, but > iozone reports some strange numbers. The benchmark "writes" (using 4 > threads, 4k blocksizes, writing four times files of size 1G to the ZFS > volume reports sometimes 150 MB/s throughput, and then 70 MB/s and > re-writes is then 1/10 of the "write" throughput and according to the > manual of iozone, re-write is considered to have higher values due to > the lack of writing the meta data again. But I'm still testing this = case. >=20 > Well, the ZFS volumes are also shared as SAMBA CIFS volumes and here I > experience something that is simply described as "abyssimal" > performance! =46rom both a dedicated Windows 7 Pro client and a = VirtualBox > 4.2.6-client access to folders in a share, say my local home, can take > ages! Opening files takes eons, if possible, in most cases windows > reports "can not open ...". Copying files from Windows to the SAMBA > share doesn't work or take ages, the throughput visible on the server > side watched by "systat -vm 1" reports spiking 0.48 MB/s, with a = hiatus > of several seconds. >=20 > Well, the SAMBA setup is straightforward, for two weeks now I have > permutated nearly every parameter suggested on all the web's help = sites > and I simply took the well configuration from one of our lab's FreeBSD > 9.1-STABLE SAMBA servers and changed the local settings for IP and > domain names etc. The working server (FreeBSD 9.1-STABLE) in question > has a single ZFS drive and is exporting this also via NFSv4. It = doesn't > have RAIDZ setup! >=20 > Before I start benchmarking further with iozone I need to know whether > there is an unresolved problem in FreeBSD 10.0 with ZFS/RAIDZ and = SAMBA > or whether I'm mislead and have overseen an important setup option. > Before exposing all of my setups here I need to clearify. >=20 > I didn't find so far any issues on the web regarding SAMBA, NFSv4 and > ZFS/RAIDZ. >=20 > Thanks in advance, > Oliver >=20 >=20 I experienced the same performance problem, then followed Jeremy = Chadwick's advice and tuned variables a bit, I'm getting excellent = performance now. /boot/loader.conf # Tune ZFS somewhat aye ? vm.kmem_size=3D"3072M" vfs.zfs.arc_min=3D"128M" vfs.zfs.arc_max=3D"2048M" # Decrease ZFS txg timeout value from 30 (default) to 5 seconds. This # should increase throughput and decrease the "bursty" stalls that # happen during immense I/O with ZFS. # = http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html # = http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html vfs.zfs.txg.timeout=3D"5" And network cards: # Up a bit our intel cards parameters hw.em.txd=3D4096 hw.em.rxd=3D4096 hw.em.tx_int_delay=3D512 hw.em.rx_int_delay=3D512 hw.em.tx_abs_int_delay=3D1024 hw.em.rx_abs_int_delay=3D1024 And this is under [global] in /usr/local/etc/smb.conf: min receivefile size =3D 16384 aio read size =3D 16384 aio write size =3D 16384 aio write behind =3D yes
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1ADC2ECB-70FF-4DDD-9D62-16E2EEECDD8B>