From owner-freebsd-fs@FreeBSD.ORG Fri Jul 23 13:16:57 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6026A106566C; Fri, 23 Jul 2010 13:16:57 +0000 (UTC) (envelope-from bra@fsn.hu) Received: from people.fsn.hu (people.fsn.hu [195.228.252.137]) by mx1.freebsd.org (Postfix) with ESMTP id A90E98FC16; Fri, 23 Jul 2010 13:16:56 +0000 (UTC) Received: by people.fsn.hu (Postfix, from userid 1001) id B305839B102; Fri, 23 Jul 2010 15:16:54 +0200 (CEST) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.2 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MF-ACE0E1EA [pR: 8.4695] X-CRM114-CacheID: sfid-20100723_15155_1BBBEC24 X-CRM114-Status: Good ( pR: 8.4695 ) X-DSPAM-Result: Whitelisted X-DSPAM-Processed: Fri Jul 23 15:16:53 2010 X-DSPAM-Confidence: 0.9910 X-DSPAM-Probability: 0.0000 X-DSPAM-Signature: 4c499644562341620720745 X-DSPAM-Factors: 27, From*Attila Nagy , 0.00119, wrote, 0.00274, wrote, 0.00274, wrote+>, 0.00304, wrote+>, 0.00304, >+On, 0.00943, this+>, 0.01000, CPUs, 0.01000, I+understand, 0.01000, hardware, 0.01000, benchmark, 0.01000, saying, 0.01000, understand+your, 0.01000, Nagy, 0.01000, checksum, 0.01000, on+it, 0.01000, the+machine, 0.01000, it's+not, 0.01000, /dev/null, 0.01000, /dev/null, 0.01000, 246, 0.01000, I've, 0.01076, >>+>>, 0.01253, >>+>>, 0.01253, >+>>, 0.01408, >+>>, 0.01408, X-Spambayes-Classification: ham; 0.00 Received: from japan.t-online.private (japan.t-online.co.hu [195.228.243.99]) by people.fsn.hu (Postfix) with ESMTPSA id 7CF2A39B0EB; Fri, 23 Jul 2010 15:15:35 +0200 (CEST) Message-ID: <4C4995F7.2080107@fsn.hu> Date: Fri, 23 Jul 2010 15:15:35 +0200 From: Attila Nagy User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.10) Gecko/20100629 Thunderbird/3.0.5 MIME-Version: 1.0 To: Ivan Voras References: <4C496EB0.7050004@fsn.hu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-2; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: ZFS makes SSDs faster than memory! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Jul 2010 13:16:57 -0000 On 07/23/10 14:15, Ivan Voras wrote: > On 07/23/10 12:28, Attila Nagy wrote: > >> Hi, >> >> I've came across a strange issue. On a file server (ftp/http/rsync) >> there is a dual SSD based L2ARC configured for a pool of 24 disks: >> > >> fetch -o /dev/null -4 >> http://ftp.fsn.hu/pub/CDROM-Images/opensolaris/osol-0906-106a-ai-sparc.iso >> /dev/null 100% of 493 MB 11 MBps >> > If I understand your setup and your benchmark correctly, you are saying > you have achieved 11 megabytes / s performance out of a volume of 24 > RAIDZ2 drives split into two parts (so it's like RAID 60). Doesn't this > number seem extremely low to you, considering that (if recent models) > each of your drives can probably pull at least 70 MB/s? > First of all, it's not an isolated system, there are active users on it. But you are right, 11 MiBps is far from being the max out of this hardware, even considering that the CPUs are somewhat old (2xAMD Opteron 246, 2GHz). When pulling this amount of data out of the machine, the disks aren't saturated, they are at around 10-20% of utilization according to gstat. BTW, remember that two RAIDZ2 in stripe isn't RAID60. In RAIDZ2 every read involves a full stripe (er, block) read for checksum validation, which means at a 128 kiB blocksize and with 12 disks in a RAIDZ2 pool, all disks provide their part of that 128k read. That's why a RAIDZ2 pool's IO performance equals of one disk's. The disks in a normal 20-30 MiBps network load do about 30-40 read IOPS, you are right that they are capable of more (around 100-120).