From owner-freebsd-current@FreeBSD.ORG Fri Jan 4 13:52:33 2013 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3BBB18BC for ; Fri, 4 Jan 2013 13:52:33 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id CE5CA303 for ; Fri, 4 Jan 2013 13:52:32 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) for freebsd-current@FreeBSD.org with esmtp (envelope-from ) id <1Tr7hB-0046p8-0D>; Fri, 04 Jan 2013 14:52:25 +0100 Received: from e178029108.adsl.alicedsl.de ([85.178.29.108] helo=thor.walstatt.dyndns.org) by inpost2.zedat.fu-berlin.de (Exim 4.69) for freebsd-current@FreeBSD.org with esmtpsa (envelope-from ) id <1Tr7hA-0007pe-S4>; Fri, 04 Jan 2013 14:52:24 +0100 Message-ID: <50E6DE91.7010404@zedat.fu-berlin.de> Date: Fri, 04 Jan 2013 14:52:17 +0100 From: "O. Hartmann" User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: Current FreeBSD Subject: ZFS/RAIDZ and SAMBA: abyssimal performance X-Enigmail-Version: 1.4.6 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig0C265D08FBE00A13812E55FE" X-Originating-IP: 85.178.29.108 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jan 2013 13:52:33 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig0C265D08FBE00A13812E55FE Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: quoted-printable I use a small testing server. The hardware is most modern Intel hardware (i3-3220, Z77 chipset), 16GB RAM. The OS is FreeBSD 10.0-CURRENT #1 r245036M: Fri Jan 4 12:48:53 CET 2013. The ZFS subsystem is comprised by 3 Western Digital 3 TB harddrives (WDC WD30EZRX-00DC0B0 80.00A80> ATA-9 SATA 3.x device), setup as a ZFS RAIDZ: --- root@gate [etc] zpool status pool: ASGARD00 state: ONLINE scan: scrub repaired 0 in 1h45m with 0 errors on Sat Dec 1 20:59:44 20= 12 config: NAME STATE READ WRITE CKSUM ASGARD00 ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 gptid/1e716118-1492-11e2-b828-90f6526a24d6 ONLINE 0 0 0 gptid/294a6798-1492-11e2-b828-90f6526a24d6 ONLINE 0 0 0 gptid/30c813f8-1492-11e2-b828-90f6526a24d6 ONLINE 0 0 0 logs ada0p1 ONLINE 0 0 0 cache ada0p2 ONLINE 0 0 0 errors: No known data errors --- The "logs" and "cache" device is a single SAMSUNG 830 SSD, 60 GB capacity, GPT partinioned, logs (ZIL) has 5GB, cache has ~55 GB. I think its not the optimal setup using the very same SSD for both caching/L2ARC and ZIL, but without the cache device the performance doen't differ much at the moment. Luckliy, with ZFS I can change the arrangement as I like. The ZFS volumes created on the pool named ASGARD00 are standard, only options sharenfs/sharesmb/checksum are set to yes. Everthing elese is set to the defaults. In /boot/loader.conf I set the following parameters according to many (and confusing!) help and suggestions on the web: # ZFS #vfs.zfs.cache_flush_disable=3D1 # #vfs.zfs.write_limit_override=3D1073741824 # 1GB vfs.zfs.l2arc_noprefetch=3D0 vfs.zfs.l2arc_headroom=3D6 The NFSv4 performance (client is also FreeBSD 10.0-CURRENT of the same date) is moderate to disapointing and doesn't exceed 45 - 55 MB/s sustained, but here are sometimes "spikes" I can watch with "systat -vm 1" reporting 120 MB/s per drive (ada2/ada3/ada4, the 3x 3TB WD drives in RAIDZ). I still benchmark via iozone. Both server and client use JUMBO frames (MTU=3D6120), which gives better throughput compared to the standard MTU=3D1500. The local performance on the server itself is slightly better, but iozone reports some strange numbers. The benchmark "writes" (using 4 threads, 4k blocksizes, writing four times files of size 1G to the ZFS volume reports sometimes 150 MB/s throughput, and then 70 MB/s and re-writes is then 1/10 of the "write" throughput and according to the manual of iozone, re-write is considered to have higher values due to the lack of writing the meta data again. But I'm still testing this case.= Well, the ZFS volumes are also shared as SAMBA CIFS volumes and here I experience something that is simply described as "abyssimal" performance! From both a dedicated Windows 7 Pro client and a VirtualBox 4.2.6-client access to folders in a share, say my local home, can take ages! Opening files takes eons, if possible, in most cases windows reports "can not open ...". Copying files from Windows to the SAMBA share doesn't work or take ages, the throughput visible on the server side watched by "systat -vm 1" reports spiking 0.48 MB/s, with a hiatus of several seconds. Well, the SAMBA setup is straightforward, for two weeks now I have permutated nearly every parameter suggested on all the web's help sites and I simply took the well configuration from one of our lab's FreeBSD 9.1-STABLE SAMBA servers and changed the local settings for IP and domain names etc. The working server (FreeBSD 9.1-STABLE) in question has a single ZFS drive and is exporting this also via NFSv4. It doesn't have RAIDZ setup! Before I start benchmarking further with iozone I need to know whether there is an unresolved problem in FreeBSD 10.0 with ZFS/RAIDZ and SAMBA or whether I'm mislead and have overseen an important setup option. Before exposing all of my setups here I need to clearify. I didn't find so far any issues on the web regarding SAMBA, NFSv4 and ZFS/RAIDZ. Thanks in advance, Oliver --------------enig0C265D08FBE00A13812E55FE Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (FreeBSD) iQEcBAEBAgAGBQJQ5t6YAAoJEOgBcD7A/5N8mQ4H/2+kV0yTu8ItrjJYqANJ2w+z zS6k9Wa3lFo0lrz6hY8w2F7ZilSMJuYkgn5Ugif/g/vNxs8KVjTm+CcZ2pQ/fPdO /Wv9eu+NGHZ1QhaBvQ0sKG1ZKNH6KcTnWlIhBviB8B98OEFGMQF7C8TQ37NwO40y hbdJGiAu16Lbkn1TGuMmWOfrW0FJfrhjqWJfDDVigr+gDTlHhW/fh0GZ3NSbzdVX jIBZTv3SSCRAIWyZieqTxzNXzHjPZ3otWL97vUM3+gFMPHXVQQEcmfdZOYzdJnnA 9DMNFedFbShCxIMlkm/n5fS5OsrOHXKgKocYyVw4BKVhpGt59GMv9ysVVHX8xFw= =n/b2 -----END PGP SIGNATURE----- --------------enig0C265D08FBE00A13812E55FE--