From owner-freebsd-current@FreeBSD.ORG Fri Jan 4 14:06:05 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9C3C2AD6 for ; Fri, 4 Jan 2013 14:06:05 +0000 (UTC) (envelope-from ml@my.gd) Received: from mail-wi0-f180.google.com (mail-wi0-f180.google.com [209.85.212.180]) by mx1.freebsd.org (Postfix) with ESMTP id 24AB9371 for ; Fri, 4 Jan 2013 14:06:04 +0000 (UTC) Received: by mail-wi0-f180.google.com with SMTP id hj13so9262381wib.1 for ; Fri, 04 Jan 2013 06:06:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:content-type:mime-version:subject:from:in-reply-to:date :cc:content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=usF0XiTu/77nrMRCIusTyWNqsfXD5sEH2HJJI9iiZ8E=; b=PAx3++D/iGr4sOzCExwAru9ty2hHOgzczcS+0hIC6Z/+wh9bkVShFPTlbYmZSGhEXJ iPzL7/hMG1DLWnIiUmG4KxEg2NDSr3ZpzZhvh3JCpriqpkTQviHoTYkrvbOOT5lOlmKv X4E6lOStLyibGnIah8C6QBfo9FiNkerXvald6z7p4YyEhQh6jM+nVdAsj+gfdnx/Nca5 lMXR2/QsSsNF5OcmMlxGgwxIDwHkY58YiICllG3YS2MvZT3s0HlP6pbSj5AwXgP0ctrd w3v3lWC/jqeG8CELyEJjGRpqyVEUpp2xr4JEog14vGdynIi19SrPBYZp7/oVwYo8Xeuk ATZQ== X-Received: by 10.194.88.98 with SMTP id bf2mr84253307wjb.49.1357308363130; Fri, 04 Jan 2013 06:06:03 -0800 (PST) Received: from [10.75.0.66] ([83.167.62.196]) by mx.google.com with ESMTPS id g2sm90400225wiy.0.2013.01.04.06.06.00 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 04 Jan 2013 06:06:02 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: ZFS/RAIDZ and SAMBA: abyssimal performance From: Fleuriot Damien In-Reply-To: <50E6DE91.7010404@zedat.fu-berlin.de> Date: Fri, 4 Jan 2013 15:06:06 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <1ADC2ECB-70FF-4DDD-9D62-16E2EEECDD8B@my.gd> References: <50E6DE91.7010404@zedat.fu-berlin.de> To: "O. Hartmann" X-Mailer: Apple Mail (2.1499) X-Gm-Message-State: ALoCoQn4k1aq5+wXXdMKcK4aagQZrjLPaGft7cLRxjEwo1B64xohRWa/VU47bqWDqqVTNu9Xk+wd Cc: Current FreeBSD X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jan 2013 14:06:05 -0000 On Jan 4, 2013, at 2:52 PM, "O. Hartmann" = wrote: > I use a small testing server. The hardware is most modern Intel = hardware > (i3-3220, Z77 chipset), 16GB RAM. The OS is FreeBSD 10.0-CURRENT #1 > r245036M: Fri Jan 4 12:48:53 CET 2013. >=20 > The ZFS subsystem is comprised by 3 Western Digital 3 TB harddrives = (WDC > WD30EZRX-00DC0B0 80.00A80> ATA-9 SATA 3.x device), setup as a ZFS = RAIDZ: >=20 > --- > root@gate [etc] zpool status > pool: ASGARD00 > state: ONLINE > scan: scrub repaired 0 in 1h45m with 0 errors on Sat Dec 1 20:59:44 = 2012 > config: >=20 > NAME STATE READ > WRITE CKSUM > ASGARD00 ONLINE 0 > 0 0 > raidz1-0 ONLINE 0 > 0 0 > gptid/1e716118-1492-11e2-b828-90f6526a24d6 ONLINE 0 > 0 0 > gptid/294a6798-1492-11e2-b828-90f6526a24d6 ONLINE 0 > 0 0 > gptid/30c813f8-1492-11e2-b828-90f6526a24d6 ONLINE 0 > 0 0 > logs > ada0p1 ONLINE 0 > 0 0 > cache > ada0p2 ONLINE 0 > 0 0 >=20 > errors: No known data errors > --- >=20 > The "logs" and "cache" device is a single SAMSUNG 830 SSD, 60 GB > capacity, GPT partinioned, logs (ZIL) has 5GB, cache has ~55 GB. >=20 > I think its not the optimal setup using the very same SSD for both > caching/L2ARC and ZIL, but without the cache device the performance > doen't differ much at the moment. Luckliy, with ZFS I can change the > arrangement as I like. >=20 > The ZFS volumes created on the pool named ASGARD00 are standard, only > options sharenfs/sharesmb/checksum are set to yes. Everthing elese is > set to the defaults. >=20 > In /boot/loader.conf I set the following parameters according to many > (and confusing!) help and suggestions on the web: >=20 > # ZFS > #vfs.zfs.cache_flush_disable=3D1 > # > #vfs.zfs.write_limit_override=3D1073741824 # 1GB > vfs.zfs.l2arc_noprefetch=3D0 > vfs.zfs.l2arc_headroom=3D6 >=20 > The NFSv4 performance (client is also FreeBSD 10.0-CURRENT of the same > date) is moderate to disapointing and doesn't exceed 45 - 55 MB/s > sustained, but here are sometimes "spikes" I can watch with "systat = -vm > 1" reporting 120 MB/s per drive (ada2/ada3/ada4, the 3x 3TB WD drives = in > RAIDZ). I still benchmark via iozone. Both server and client use JUMBO > frames (MTU=3D6120), which gives better throughput compared to the > standard MTU=3D1500. >=20 > The local performance on the server itself is slightly better, but > iozone reports some strange numbers. The benchmark "writes" (using 4 > threads, 4k blocksizes, writing four times files of size 1G to the ZFS > volume reports sometimes 150 MB/s throughput, and then 70 MB/s and > re-writes is then 1/10 of the "write" throughput and according to the > manual of iozone, re-write is considered to have higher values due to > the lack of writing the meta data again. But I'm still testing this = case. >=20 > Well, the ZFS volumes are also shared as SAMBA CIFS volumes and here I > experience something that is simply described as "abyssimal" > performance! =46rom both a dedicated Windows 7 Pro client and a = VirtualBox > 4.2.6-client access to folders in a share, say my local home, can take > ages! Opening files takes eons, if possible, in most cases windows > reports "can not open ...". Copying files from Windows to the SAMBA > share doesn't work or take ages, the throughput visible on the server > side watched by "systat -vm 1" reports spiking 0.48 MB/s, with a = hiatus > of several seconds. >=20 > Well, the SAMBA setup is straightforward, for two weeks now I have > permutated nearly every parameter suggested on all the web's help = sites > and I simply took the well configuration from one of our lab's FreeBSD > 9.1-STABLE SAMBA servers and changed the local settings for IP and > domain names etc. The working server (FreeBSD 9.1-STABLE) in question > has a single ZFS drive and is exporting this also via NFSv4. It = doesn't > have RAIDZ setup! >=20 > Before I start benchmarking further with iozone I need to know whether > there is an unresolved problem in FreeBSD 10.0 with ZFS/RAIDZ and = SAMBA > or whether I'm mislead and have overseen an important setup option. > Before exposing all of my setups here I need to clearify. >=20 > I didn't find so far any issues on the web regarding SAMBA, NFSv4 and > ZFS/RAIDZ. >=20 > Thanks in advance, > Oliver >=20 >=20 I experienced the same performance problem, then followed Jeremy = Chadwick's advice and tuned variables a bit, I'm getting excellent = performance now. /boot/loader.conf # Tune ZFS somewhat aye ? vm.kmem_size=3D"3072M" vfs.zfs.arc_min=3D"128M" vfs.zfs.arc_max=3D"2048M" # Decrease ZFS txg timeout value from 30 (default) to 5 seconds. This # should increase throughput and decrease the "bursty" stalls that # happen during immense I/O with ZFS. # = http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007343.html # = http://lists.freebsd.org/pipermail/freebsd-fs/2009-December/007355.html vfs.zfs.txg.timeout=3D"5" And network cards: # Up a bit our intel cards parameters hw.em.txd=3D4096 hw.em.rxd=3D4096 hw.em.tx_int_delay=3D512 hw.em.rx_int_delay=3D512 hw.em.tx_abs_int_delay=3D1024 hw.em.rx_abs_int_delay=3D1024 And this is under [global] in /usr/local/etc/smb.conf: min receivefile size =3D 16384 aio read size =3D 16384 aio write size =3D 16384 aio write behind =3D yes