Date: Fri, 04 Jan 2013 14:52:17 +0100 From: "O. Hartmann" <ohartman@zedat.fu-berlin.de> To: Current FreeBSD <freebsd-current@FreeBSD.org> Subject: ZFS/RAIDZ and SAMBA: abyssimal performance Message-ID: <50E6DE91.7010404@zedat.fu-berlin.de>
next in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 2440 and 3156)
--------------enig0C265D08FBE00A13812E55FE
Content-Type: text/plain; charset=ISO-8859-15
Content-Transfer-Encoding: quoted-printable
I use a small testing server. The hardware is most modern Intel hardware
(i3-3220, Z77 chipset), 16GB RAM. The OS is FreeBSD 10.0-CURRENT #1
r245036M: Fri Jan 4 12:48:53 CET 2013.
The ZFS subsystem is comprised by 3 Western Digital 3 TB harddrives (WDC
WD30EZRX-00DC0B0 80.00A80> ATA-9 SATA 3.x device), setup as a ZFS RAIDZ:
---
root@gate [etc] zpool status
pool: ASGARD00
state: ONLINE
scan: scrub repaired 0 in 1h45m with 0 errors on Sat Dec 1 20:59:44 20=
12
config:
NAME STATE READ
WRITE CKSUM
ASGARD00 ONLINE 0
0 0
raidz1-0 ONLINE 0
0 0
gptid/1e716118-1492-11e2-b828-90f6526a24d6 ONLINE 0
0 0
gptid/294a6798-1492-11e2-b828-90f6526a24d6 ONLINE 0
0 0
gptid/30c813f8-1492-11e2-b828-90f6526a24d6 ONLINE 0
0 0
logs
ada0p1 ONLINE 0
0 0
cache
ada0p2 ONLINE 0
0 0
errors: No known data errors
---
The "logs" and "cache" device is a single SAMSUNG 830 SSD, 60 GB
capacity, GPT partinioned, logs (ZIL) has 5GB, cache has ~55 GB.
I think its not the optimal setup using the very same SSD for both
caching/L2ARC and ZIL, but without the cache device the performance
doen't differ much at the moment. Luckliy, with ZFS I can change the
arrangement as I like.
The ZFS volumes created on the pool named ASGARD00 are standard, only
options sharenfs/sharesmb/checksum are set to yes. Everthing elese is
set to the defaults.
In /boot/loader.conf I set the following parameters according to many
(and confusing!) help and suggestions on the web:
# ZFS
#vfs.zfs.cache_flush_disable=3D1
#
#vfs.zfs.write_limit_override=3D1073741824 # 1GB
vfs.zfs.l2arc_noprefetch=3D0
vfs.zfs.l2arc_headroom=3D6
The NFSv4 performance (client is also FreeBSD 10.0-CURRENT of the same
date) is moderate to disapointing and doesn't exceed 45 - 55 MB/s
sustained, but here are sometimes "spikes" I can watch with "systat -vm
1" reporting 120 MB/s per drive (ada2/ada3/ada4, the 3x 3TB WD drives in
RAIDZ). I still benchmark via iozone. Both server and client use JUMBO
frames (MTU=3D6120), which gives better throughput compared to the
standard MTU=3D1500.
The local performance on the server itself is slightly better, but
iozone reports some strange numbers. The benchmark "writes" (using 4
threads, 4k blocksizes, writing four times files of size 1G to the ZFS
volume reports sometimes 150 MB/s throughput, and then 70 MB/s and
re-writes is then 1/10 of the "write" throughput and according to the
manual of iozone, re-write is considered to have higher values due to
the lack of writing the meta data again. But I'm still testing this case.=
Well, the ZFS volumes are also shared as SAMBA CIFS volumes and here I
experience something that is simply described as "abyssimal"
performance! From both a dedicated Windows 7 Pro client and a VirtualBox
4.2.6-client access to folders in a share, say my local home, can take
ages! Opening files takes eons, if possible, in most cases windows
reports "can not open ...". Copying files from Windows to the SAMBA
share doesn't work or take ages, the throughput visible on the server
side watched by "systat -vm 1" reports spiking 0.48 MB/s, with a hiatus
of several seconds.
Well, the SAMBA setup is straightforward, for two weeks now I have
permutated nearly every parameter suggested on all the web's help sites
and I simply took the well configuration from one of our lab's FreeBSD
9.1-STABLE SAMBA servers and changed the local settings for IP and
domain names etc. The working server (FreeBSD 9.1-STABLE) in question
has a single ZFS drive and is exporting this also via NFSv4. It doesn't
have RAIDZ setup!
Before I start benchmarking further with iozone I need to know whether
there is an unresolved problem in FreeBSD 10.0 with ZFS/RAIDZ and SAMBA
or whether I'm mislead and have overseen an important setup option.
Before exposing all of my setups here I need to clearify.
I didn't find so far any issues on the web regarding SAMBA, NFSv4 and
ZFS/RAIDZ.
Thanks in advance,
Oliver
--------------enig0C265D08FBE00A13812E55FE
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.19 (FreeBSD)
iQEcBAEBAgAGBQJQ5t6YAAoJEOgBcD7A/5N8mQ4H/2+kV0yTu8ItrjJYqANJ2w+z
zS6k9Wa3lFo0lrz6hY8w2F7ZilSMJuYkgn5Ugif/g/vNxs8KVjTm+CcZ2pQ/fPdO
/Wv9eu+NGHZ1QhaBvQ0sKG1ZKNH6KcTnWlIhBviB8B98OEFGMQF7C8TQ37NwO40y
hbdJGiAu16Lbkn1TGuMmWOfrW0FJfrhjqWJfDDVigr+gDTlHhW/fh0GZ3NSbzdVX
jIBZTv3SSCRAIWyZieqTxzNXzHjPZ3otWL97vUM3+gFMPHXVQQEcmfdZOYzdJnnA
9DMNFedFbShCxIMlkm/n5fS5OsrOHXKgKocYyVw4BKVhpGt59GMv9ysVVHX8xFw=
=n/b2
-----END PGP SIGNATURE-----
--------------enig0C265D08FBE00A13812E55FE--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50E6DE91.7010404>