Date: Mon, 19 Oct 2009 12:52:56 +0100 From: Christopher Key <cjk32@cam.ac.uk> To: freebsd-questions@freebsd.org Subject: ZFS: Strange performance issues Message-ID: <4ADC5318.2010706@cam.ac.uk>
next in thread | raw e-mail | index | archive | help
Hello,
I'm running FreeBSD 7.2 amd64 on a system with 2GB RAM. I've a zfs pool
using raidz1 over five 2Tb SATA drives connected via a port multiplier
and a RR2314 card.
I can write to a filesystem on this pool at approx 20MB/s:
# dd if=/dev/urandom of=$FS/testdump bs=1m count=1k
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 47.096440 secs (22798790 bytes/sec)
and zpool iostat -v is consistent with this
capacity operations bandwidth
pool used avail read write read write
-------------- ----- ----- ----- ----- ----- -----
films 6.37T 2.69T 11 440 53.2K 23.0M
raidz1 6.37T 2.69T 11 440 53.2K 23.0M
da0 - - 1 214 88.2K 5.58M
da1 - - 1 209 88.6K 5.58M
da2 - - 1 211 76.0K 5.70M
da3 - - 1 213 88.6K 5.77M
da4 - - 1 213 88.6K 5.71M
-------------- ----- ----- ----- ----- ----- -----
However, the read behaviour is strange:
dd if=$FS/testdump of=/dev/null bs=1m count=1k
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 40.392055 secs (26582996 bytes/sec)
but here, zpool iostat -v is odd:
capacity operations bandwidth
pool used avail read write read write
-------------- ----- ----- ----- ----- ----- -----
films 6.37T 2.69T 1.52K 0 194M 0
raidz1 6.37T 2.69T 1.52K 0 194M 0
da0 - - 420 0 48.2M 0
da1 - - 274 0 24.0M 0
da2 - - 270 0 24.4M 0
da3 - - 418 0 48.4M 0
da4 - - 418 0 48.4M 0
-------------- ----- ----- ----- ----- ----- -----
Notice that dd was reading at ~27MB/s, but zfs is reading from the vdev
at ~200MB/s. Also odd is that fact the reduced read rates for da1, da2.
I'm struggling to understand what's happening to the extra data being
read. The most likely scenario seems to be that ZFS is inflating its
read size, knowing that it won't delay the transfer significantly, and
hoping to pull in some useful data to the cache. However, it's either
failing the cache this data correctly, or the file is highly
non-contiguous, and the extra data read doesn't contain anythign useful
to out read.
I'm also somewhat surpised by the poor performance of the pool. From
memory, when it was first configured (on identical hardware and
software), I could write at ~130MB/s and read at ~200MB/s.
Once conclusion is that the pool is suffering from something akin to
fragmentation, perhaps with files always being allocated from very small
blacks. The vast majority of the data comprises large (~1Gb) files,
that are written to one 'import' pool, moved to the main pool, then
never modified. There are however a lot (~5000) of small (<1k) files
that get rewritten half hourly, and I'm wondering if that might be
causing problems, and confusing ZFS's block sizing algorithm.
Can anyone shed any light on what might be going on, or how to further
diagnose this problem. Do any of my tentative conclusions make any sense?
Kind regards,
Christopher Key
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4ADC5318.2010706>