From owner-freebsd-fs@FreeBSD.ORG Thu Mar 29 09:09:04 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 321CF1065687 for ; Thu, 29 Mar 2012 09:09:04 +0000 (UTC) (envelope-from mailer@mail.firmos.at) Received: from mail.firmos.at (mail.firmos.at [80.123.225.49]) by mx1.freebsd.org (Postfix) with ESMTP id 392C88FC20 for ; Thu, 29 Mar 2012 09:09:02 +0000 (UTC) Received: from [91.114.28.42] (helo=BigMac.local) by mail.firmos.at with esmtpsa (TLSv1:CAMELLIA256-SHA:256) (Exim 4.76 (FreeBSD)) (envelope-from ) id 1SDB2W-000B2P-6P for freebsd-fs@freebsd.org; Thu, 29 Mar 2012 10:49:04 +0200 Message-ID: <4F74220E.9070707@firmos.at> Date: Thu, 29 Mar 2012 10:49:18 +0200 From: Franz Schober Organization: FirmOS Business Solutions Gmbh User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:10.0.2) Gecko/20120216 Thunderbird/10.0.2 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Sender: mailer@mail.firmos.at X-SA-Exim-Connect-IP: 91.114.28.42 X-SA-Exim-Mail-From: mailer@mail.firmos.at X-SA-Exim-Scanned: No (on mail.firmos.at); SAEximRunCond expanded to false Subject: ZFS Performance FreeBSD 9.0 vs. Openindiana X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Mar 2012 09:09:04 -0000 Hi, I am testing a new storage hardware (Xeon E5606, 24 GB Ram, LSI1068 controller in non-raid mode, 8 7200 SATA Disks, 2 OCZ SSD ZIL Mirror), for a NFS/iSCSI storage system. Currently I'm doing basic performance tests of the ZFS filesystem. We are using FreeBSD for a couple of years for various services in our software development company, so FreeBSD is our first choice. For performance tests, I use the iozone benchmark with a multi-streaming concurrency test, iozone -o -c -t 8 -r 128k -s 4G (Sync Mode, 8 concurrent workers, 128 k Recordsize, 4G working file for every worker to run not only in cache). ZPool ist a stripe of the 8 SATA Disks and a mirror of the two SSD Log disks (here the FreeBSD zpool status): pool: pools state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM pools ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 logs mirror-8 ONLINE 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 Comparing FreeBSD 9.0-RELEASE (amd64) to OpenIndiana 151a, I get a performance difference, where FreeBSD is slower in most tests, the pools is freshly created before each test: FreeBSD: Iozone: Performance Test of File I/O Version $Revision: 3.397 $ Compiled for 64 bit mode. Build: freebsd Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Dave Boone, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root, Fabrice Bacchella, Zhenghua Xue, Qin Li, Darren Sawyer. Ben England. Run began: Thu Mar 29 10:39:47 2012 SYNC Mode. Include close in write timing Record Size 128 KB File size set to 4194304 KB Command line used: iozone -o -c -t 8 -r 128k -s 4G Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Throughput test with 8 processes Each process writes a 4194304 Kbyte file in 128 Kbyte records Children see throughput for 8 initial writers = 150780.28 KB/sec Parent sees throughput for 8 initial writers = 150767.08 KB/sec Min throughput per process = 18846.89 KB/sec Max throughput per process = 18849.44 KB/sec Avg throughput per process = 18847.53 KB/sec Min xfer = 4193792.00 KB Children see throughput for 8 rewriters = 150051.28 KB/sec Parent sees throughput for 8 rewriters = 150048.26 KB/sec Min throughput per process = 18615.49 KB/sec Max throughput per process = 18837.79 KB/sec Avg throughput per process = 18756.41 KB/sec Min xfer = 4144896.00 KB Children see throughput for 8 readers = 1001472.78 KB/sec Parent sees throughput for 8 readers = 1001417.37 KB/sec Min throughput per process = 118845.12 KB/sec Max throughput per process = 138027.58 KB/sec Avg throughput per process = 125184.10 KB/sec Min xfer = 3611520.00 KB Children see throughput for 8 re-readers = 876152.78 KB/sec Parent sees throughput for 8 re-readers = 870610.71 KB/sec Min throughput per process = 106323.10 KB/sec Max throughput per process = 114136.91 KB/sec Avg throughput per process = 109519.10 KB/sec Min xfer = 3909120.00 KB Children see throughput for 8 reverse readers = 1148130.64 KB/sec Parent sees throughput for 8 reverse readers = 1142916.22 KB/sec Min throughput per process = 126814.09 KB/sec Max throughput per process = 172099.73 KB/sec Avg throughput per process = 143516.33 KB/sec Min xfer = 3117440.00 KB Children see throughput for 8 stride readers = 299141.80 KB/sec Parent sees throughput for 8 stride readers = 298399.38 KB/sec Min throughput per process = 29238.70 KB/sec Max throughput per process = 45995.27 KB/sec Avg throughput per process = 37392.73 KB/sec Min xfer = 2667264.00 KB Children see throughput for 8 random readers = 115979.46 KB/sec Parent sees throughput for 8 random readers = 115974.77 KB/sec Min throughput per process = 14312.92 KB/sec Max throughput per process = 14753.71 KB/sec Avg throughput per process = 14497.43 KB/sec Min xfer = 4068992.00 KB Children see throughput for 8 mixed workload = 188689.19 KB/sec Parent sees throughput for 8 mixed workload = 188673.96 KB/sec Min throughput per process = 12175.37 KB/sec Max throughput per process = 34954.12 KB/sec Avg throughput per process = 23586.15 KB/sec Min xfer = 1460992.00 KB Children see throughput for 8 random writers = 147947.42 KB/sec Parent sees throughput for 8 random writers = 147939.94 KB/sec Min throughput per process = 18492.69 KB/sec Max throughput per process = 18494.70 KB/sec Avg throughput per process = 18493.43 KB/sec Min xfer = 4193920.00 KB Children see throughput for 8 pwrite writers = 149780.00 KB/sec Parent sees throughput for 8 pwrite writers = 149772.91 KB/sec Min throughput per process = 18721.60 KB/sec Max throughput per process = 18723.81 KB/sec Avg throughput per process = 18722.50 KB/sec Min xfer = 4193792.00 KB Children see throughput for 8 pread readers = 1014824.33 KB/sec Parent sees throughput for 8 pread readers = 1014743.46 KB/sec Min throughput per process = 114311.04 KB/sec Max throughput per process = 150336.52 KB/sec Avg throughput per process = 126853.04 KB/sec Min xfer = 3189376.00 KB iozone test complete. OpenIndiana: Iozone: Performance Test of File I/O Version $Revision: 3.323 $ Compiled for 32 bit mode. Build: Solaris10cc-64 Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Gavin Brebner, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong, Joshua Root. Run began: Fri Mar 16 14:11:48 2012 SYNC Mode. Include close in write timing Record Size 128 KB File size set to 4194304 KB Command line used: /usr/benchmarks/iozone/iozone -o -c -t 8 -r 128k -s 4G Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. Throughput test with 8 processes Each process writes a 4194304 Kbyte file in 128 Kbyte records Children see throughput for 8 initial writers = 163315.97 KB/sec Parent sees throughput for 8 initial writers = 163238.81 KB/sec Min throughput per process = 20399.37 KB/sec Max throughput per process = 20423.74 KB/sec Avg throughput per process = 20414.50 KB/sec Min xfer = 4189440.00 KB Children see throughput for 8 rewriters = 163415.44 KB/sec Parent sees throughput for 8 rewriters = 163298.96 KB/sec Min throughput per process = 20266.25 KB/sec Max throughput per process = 20611.95 KB/sec Avg throughput per process = 20426.93 KB/sec Min xfer = 4123904.00 KB Children see throughput for 8 readers = 1180533.94 KB/sec Parent sees throughput for 8 readers = 1180090.09 KB/sec Min throughput per process = 121104.02 KB/sec Max throughput per process = 198012.62 KB/sec Avg throughput per process = 147566.74 KB/sec Min xfer = 2565376.00 KB Children see throughput for 8 re-readers = 951965.89 KB/sec Parent sees throughput for 8 re-readers = 948751.85 KB/sec Min throughput per process = 118049.84 KB/sec Max throughput per process = 120618.28 KB/sec Avg throughput per process = 118995.74 KB/sec Min xfer = 4113536.00 KB Children see throughput for 8 reverse readers = 3024070.31 KB/sec Parent sees throughput for 8 reverse readers = 2867010.77 KB/sec Min throughput per process = 318376.59 KB/sec Max throughput per process = 557625.75 KB/sec Avg throughput per process = 378008.79 KB/sec Min xfer = 2563968.00 KB Children see throughput for 8 stride readers = 335398.05 KB/sec Parent sees throughput for 8 stride readers = 331540.81 KB/sec Min throughput per process = 32520.95 KB/sec Max throughput per process = 62529.34 KB/sec Avg throughput per process = 41924.76 KB/sec Min xfer = 2210048.00 KB Children see throughput for 8 random readers = 120422.72 KB/sec Parent sees throughput for 8 random readers = 120418.39 KB/sec Min throughput per process = 14456.03 KB/sec Max throughput per process = 15568.85 KB/sec Avg throughput per process = 15052.84 KB/sec Min xfer = 3894528.00 KB Children see throughput for 8 mixed workload = 191391.66 KB/sec Parent sees throughput for 8 mixed workload = 191249.48 KB/sec Min throughput per process = 12364.50 KB/sec Max throughput per process = 35442.05 KB/sec Avg throughput per process = 23923.96 KB/sec Min xfer = 1463296.00 KB Children see throughput for 8 random writers = 163556.06 KB/sec Parent sees throughput for 8 random writers = 163218.36 KB/sec Min throughput per process = 20411.16 KB/sec Max throughput per process = 20484.72 KB/sec Avg throughput per process = 20444.51 KB/sec Min xfer = 4179328.00 KB iozone test complete. Now my question: Which parameters in the ZFS Subsystem of FreeBSD are tuneable to reach the same performance in FreeBSD, especially pushing the synchronous write performance ? Thx a lot, Franz Schober