Date: Tue, 08 Jun 2010 08:20:06 -0700 From: "Bradley W. Dutton" <brad@duttonbros.com> To: Jeremy Chadwick <freebsd@jdc.parodius.com> Cc: freebsd-fs@freebsd.org Subject: Re: ZFS performance of various vdevs (long post) Message-ID: <20100608082006.5006764hokcpvzqe@duttonbros.com> In-Reply-To: <20100608044707.GA78147@icarus.home.lan> References: <20100607154256.941428ovaq2hha0g@duttonbros.com> <alpine.GSO.2.01.1006071811040.12887@freddy.simplesystems.org> <20100607173218.11716iopp083dbpu@duttonbros.com> <20100608044707.GA78147@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
Quoting Jeremy Chadwick <freebsd@jdc.parodius.com>: >> On Mon, Jun 07, 2010 at 05:32:18PM -0700, Bradley W. Dutton wrote: >> I know it's pretty simple but for checking throughput I thought it >> would be ok. I don't have compression on and based on the drive >> lights and gstat, the drives definitely aren't idle. > > Try disabling prefetch (you have it enabled) and try setting > vfs.zfs.txg.timeout="5". Some people have reported a "sweet spot" with > regards to the last parameter (needing to be adjusted if your disks are > extremely fast, etc.), as otherwise ZFS would be extremely "bursty" in > its I/O (stalling/deadlocking the system at set intervals). By > decreasing the value you essentially do disk writes more regularly (with > less data), and depending upon the load and controller, this may even > out performance. I tested some of these settings. With the timeout set to 5 not much changed write wise. (keep in mind these results are the Nvidia/WDRE2 combo): With txg=5 and prefetch disabled I saw read speeds go down considerably: # normal/jbod txg=5 no prefetch zpool create bench /dev/ad4 /dev/ad6 /dev/ad10 /dev/ad12 /dev/ad14 dd if=/bench/test.file of=/dev/null bs=1m 12582912000 bytes transferred in 59.330330 secs (212082286 bytes/sec) compared to 12582912000 bytes transferred in 34.668165 secs (362952928 bytes/sec) zpool create bench raidz /dev/ad4 /dev/ad6 /dev/ad10 /dev/ad12 /dev/ad14 dd if=/bench/test.file of=/dev/null bs=1m 12582912000 bytes transferred in 71.135696 secs (176886046 bytes/sec) compared to 12582912000 bytes transferred in 45.825533 secs (274582993 bytes/sec) Running the same tests on the raidz2 Supermicro/Hitachi setup didn't yield any difference in writes, the reads were slower: zpool create tank raidz2 /dev/da0 /dev/da1 /dev/da2 /dev/da3 /dev/da4 /dev/da5 /dev/da6 /dev/da7 dd if=/tank/test.file of=/dev/null bs=1m 12582912000 bytes transferred in 44.118409 secs (285207745 bytes/sec) compared to 12582912000 bytes transferred in 32.911291 secs (382328118 bytes/sec) I rebooted and reran these numbers just to make sure they were consistent. >> >The higher CPU usage might be due to the device driver or the >> >interface card being used. >> >> Definitely a plausible explanation. If this was the case would the 8 >> parallel dd processes exhibit the same behavior? or is the type of >> IO affecting how much CPU the driver is using? > > It would be the latter. > > Also, I believe this Supermicro controller has been discussed in the > past. I can't remember if people had outright failures/issues with it > or if people were complaining about sub-par performance. I could also > be remembering a different Supermicro controller. > > If I had to make a recommendation, it would be to reproduce the same > setup on a system using an Intel ICH9/ICH9R or ICH10/ICH10R controller > in AHCI mode (with ahci.ko loaded, not ataahci.ko) and see if things > improve. But start with the loader.conf tunables I mentioned above -- > segregate each test. > > I would also recommend you re-run your tests with a different blocksize > for dd. I don't know why people keep using 1m (Linux websites?). Test > the following increments: 4k, 8k, 16k, 32k, 64k, 128k, 256k. That's > about where you should stop. I tested with 8, 16, 32, 64, 128, 1m and the results all looked similar. As such I stuck with bs=1m because it's easier to change count. > Otherwise, consider installing ports/benchmarks/bonnie++ and try that. > That will also get you concurrent I/O tests, I believe. I may give this a shot but I'm most interested in less concurrency as I have larger files with only a couple of readers/writers. As Bob noted a bunch of mirrors in the pool would definitely be faster for concurrent IO. Thanks for the help, Brad
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20100608082006.5006764hokcpvzqe>