Date: Thu, 18 Dec 2008 10:19:42 -0800 From: Matt Simerson <matt@corp.spry.com> To: freebsd-fs@freebsd.org Subject: ZFS performance gains real or imaginary? Message-ID: <22C8092E-210F-4E91-AA09-CFD38966975C@spry.com>
next in thread | raw e-mail | index | archive | help
Did I miss some major ZFS performance enhancements?
I upgraded the disks in my home file server to 1.5TB disks. Rather
than using gmirror as I did last time, I decided to use ZFS to mirror
them. The file server was running 7.0 and booted off a CF card so it
was simply a matter of adding in the extra disks, configuring them
with ZFS, and copying all the data over.
[root@storage] ~ # zpool status
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror ONLINE 0 0 0
ad11 ONLINE 0 0 0
ad13 ONLINE 0 0 0
ZFS under FreeBSD 7 is horrendously slow. It took almost two days to
copy 600GB of data (a bunch of MP3s, movies, and UFS backups of my
servers in data centers) to the ZFS volume. Once completed, I removed
the old disks. The file system performance after switching to ZFS is
quite underwhelming. I notice it when doing any sort of writes to it.
This echoes my experience with ZFS on my production backup servers at
work. (all systems are multi-core Intel with 4GB+ RAM).
$ ssh back01 uname -a
FreeBSD back01.int.spry.com 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Fri
Aug 15 16:42:36 PDT 2008 root@back01.int.spry.com:/usr/obj/usr/src/
sys/BACK01 amd64
$ ssh back02 uname -a
FreeBSD back02.int.spry.com 8.0-CURRENT FreeBSD 8.0-CURRENT #1: Wed
Aug 13 13:57:19 PDT 2008 root@back02.int.spry.com:/usr/obj/usr/src/
sys/BACK02-HEAD amd64
On the two systems above (amd64 with 16GB of RAM and 24 1TB disks) I
get about 30 days of uptime before the system hangs with a ZFS error.
They write backups to disk 24x7 and never stop. I could not anything
near that level of stability with back03 (below) which was much older
hardware maxed out at 4GB of RAM. I finally resolved the stability
issues on back03 by ditching ZFS and using geom_stripe across the two
hardware RAID arrays.
$ ssh back03 uname -a
FreeBSD back03.int.spry.com 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Tue
Oct 28 16:54:22 PDT 2008 root@back03.int.spry.com:/usr/obj/usr/src/
sys/GENERIC amd64
Yesterday I did a cvsup to 8-HEAD and built a new kernel and world. I
installed the new kernel, and then paniced slightly when I booted off
the new kernel and the ZFS utilities proved completely worthless in
attempts to get /usr and /var mounted (which are both on ZFS). It took
a quick Google search to remember the solution:
mount -t zfs tank/usr /usr
mount -t zfs tank/var /var
After installing world and rebooting, the system is positively snappy.
File system interaction, which is lethargic on every ZFS system I've
installed seems to be much faster. I haven't benchmarked the IO
performance but something definitely changed. It's almost like the
latency has decreased. Would changes committed since mid-August (when
I built my last ZFS servers from -HEAD + the patch) and now explain
this?
If so, then I really should be upgrading my production ZFS servers to
the latest -HEAD.
Matt
PS: I am using compression and getting the following results:
[root@storage] ~ # zfs get compressratio
NAME PROPERTY VALUE SOURCE
tank compressratio 1.12x -
tank/usr compressratio 1.12x -
tank/usr/.snapshots compressratio 2.09x -
tank/var compressratio 2.13x -
In retrospect, I wouldn't bother with compression on /usr. But, /
usr/.snapshots is my rsnapshot based backups of my servers sitting in
remote data centers. Since the majority of changes between snapshots
is log files, the data is quite compressible and ZFS compressions is
quite effective. It's also quite effective on /var, as is shown. ZFS
compression is effectively getting me 1/3 more disk space off my 1.5TB
disks.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?22C8092E-210F-4E91-AA09-CFD38966975C>