Date: Fri, 23 Jul 2010 12:28:00 +0200 From: Attila Nagy <bra@fsn.hu> To: fs@freebsd.org Subject: ZFS makes SSDs faster than memory! Message-ID: <4C496EB0.7050004@fsn.hu>
next in thread | raw e-mail | index | archive | help
Hi,
I've came across a strange issue. On a file server (ftp/http/rsync)
there is a dual SSD based L2ARC configured for a pool of 24 disks:
NAME STATE READ WRITE CKSUM
data ONLINE 0 0 0
raidz2 ONLINE 0 0 0
label/disk20-01 ONLINE 0 0 0
label/disk20-02 ONLINE 0 0 0
label/disk20-03 ONLINE 0 0 0
label/disk20-04 ONLINE 0 0 0
label/disk20-05 ONLINE 0 0 0
label/disk20-06 ONLINE 0 0 0
label/disk20-07 ONLINE 0 0 0
label/disk20-08 ONLINE 0 0 0
label/disk20-09 ONLINE 0 0 0
label/disk20-10 ONLINE 0 0 0
label/disk20-11 ONLINE 0 0 0
label/disk20-12 ONLINE 0 0 0
raidz2 ONLINE 0 0 0
label/disk21-01 ONLINE 0 0 0
label/disk21-02 ONLINE 0 0 0
label/disk21-03 ONLINE 0 0 0
label/disk21-04 ONLINE 0 0 0
label/disk21-05 ONLINE 0 0 0
label/disk21-06 ONLINE 0 0 0
label/disk21-07 ONLINE 0 0 0
label/disk21-08 ONLINE 0 0 0
label/disk21-09 ONLINE 0 0 0
label/disk21-10 ONLINE 0 0 0
label/disk21-11 ONLINE 0 0 0
label/disk21-12 ONLINE 0 0 0
cache
ad4s2 ONLINE 0 0 0
ad6s2 ONLINE 0 0 0
and there is about 6GB of ARC (in memory).
The strange thing is that when I fetch a file with HTTP, it comes from
disks:
fetch -o /dev/null -4
http://ftp.fsn.hu/pub/CDROM-Images/opensolaris/osol-0906-106a-ai-sparc.iso
/dev/null 100% of 493 MB 11 MBps
00m00s
a second fetch right after the above comes from in-memory ARC:
/dev/null 100% of 493 MB 28 MBps
00m00s
If I wait some minutes, the blocks are evicted from memory to SSD, so
the next fetch comes from the L2ARC on the two SSDs:
/dev/null 100% of 493 MB 43 MBps
00m00s
and if I re-fetch it right after the above (blocks will be served from
memory again), I get:
/dev/null 100% of 493 MB 27 MBps
00m00s
And this is very consistent. I watch L2ARC activity with gstat/zpool
iostat and when the bits are served from that, the throughput is
constantly higher with at least 10MBps, and the next download will
always be slower, no matter that it comes from memory.
The effect can't be seen on localhost, with dd:
# dd if=osol-0906-106a-ai-sparc.iso of=/dev/null bs=1M
493+1 records in
493+1 records out
516968448 bytes transferred in 3.310415 secs (156164240 bytes/sec)
(comes from L2ARC)
# dd if=osol-0906-106a-ai-sparc.iso of=/dev/null bs=1M
493+1 records in
493+1 records out
516968448 bytes transferred in 0.861610 secs (600002886 bytes/sec)
(comes from memory)
The daemons run in jails and they see the data through a read only
nullfs mount. However dd-ing from that nullfs mount also gives fast
throughput:
# dd if=osol-0906-106a-ai-sparc.iso of=/dev/null bs=1M
493+1 records in
493+1 records out
516968448 bytes transferred in 0.718028 secs (719983859 bytes/sec)
I'm lost. Anyone else can reproduce this?
BTW, I see another strange issue with ZFS. On the site there are sparse
files for testing clients' compatibility with big files.
Fetching these files are slower than the ones, which contains real data
and therefore moves the disks!
What should cause this? It seems very unnatural to fetch a zero byte
file with 10 MBps, which doesn't involve even a bit of disk IO, while
fetching real files with real disk IO faster...
Thanks,
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4C496EB0.7050004>