From owner-freebsd-performance@FreeBSD.ORG Wed Jul 7 14:12:47 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 17157106564A for ; Wed, 7 Jul 2010 14:12:47 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 95A2C8FC14 for ; Wed, 7 Jul 2010 14:12:46 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1OWVMg-0007uz-Q5 for freebsd-performance@freebsd.org; Wed, 07 Jul 2010 16:12:42 +0200 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 07 Jul 2010 16:12:42 +0200 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 07 Jul 2010 16:12:42 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Wed, 07 Jul 2010 16:12:45 +0200 Lines: 73 Message-ID: References: <4C33BDCC.1020004@softhammer.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.9) Gecko/20100518 Thunderbird/3.0.4 In-Reply-To: <4C33BDCC.1020004@softhammer.net> X-Enigmail-Version: 1.0.1 Subject: Re: More Controllers != Higher Through Put X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2010 14:12:47 -0000 On 07/07/10 01:35, Stephen Sanders wrote: > I'm wondering if anyone has heard of this. > > I've a system with a 3ware 9650 servicing 4 7200RPM Segate 1TB drives > and the motherboard servicing 2 7200 RPM Segate 1TB drives. So far so good. > The 4 disk array is RAID 6 while the 2 disk array is RAID 1. The drives > should deliver about 100MB/s. Ok, so you've reduced the 4-drive array's write performance nearly to equivalent of 2 drives and the 2-drive array to 1 drive. It should be even worse for random IOs. Since FreeBSD doesn't support RAID-6 I guess you are using hardware RAID? For the 2-disk RAID-1 : you are probably using software RAID, right? (on-board "SATA RAID" controllers usually are just software implementations). > 1. The most the 4 disk array is developing is 250MB/s write performance This is too much. It almost looks like something is caching what shouldn't be cached. How did you get this result? I'd expect less than 200 MB/s sequential writes on a 4-drive RAID-6 with 100 MB/s drives. > while the 2 disk array is coming in at 90MB/s write performance. This is as expected - write performance of any size RAID-1 is equivalent of 1 drive or less. > The 4 disk array seems slow. Nope - the contrary should be true. It looks like you are doing something you shouldn't if you get that much performance, or your test is overly simplistic (e.g. you're testing cache). > 2. Attempting to write to both arrays simultaneously causes the rate on > the 4 disk array to drop to 150MB/s and the 2 disk array drops to 60MB/s Are you running on an Atom CPU? What kind of system are you using? > I'd expect the 4 disk array should look more like 300+MB/s while the 2 > disk array is about right. No, you cannot get 300 MB/s from simple RAID6 of 4 drives in any direction. Think about it: all the data needs to be a) written as-is to 2 of the drives, then b) parity/ECCs calculated and c) the same amount of data written to 2 more drives. You cannot get write performance of more than 2 drives equivalent in this scheme, and will probably be worse. For reads, only if your RAID controller is very, very smart (meaning: it probably isn't), you can recover some performance by using this parity/ECC data to reconstruct more data than is read from the two "plain" drives. I think ZFS does this in a limited way. > I don't get why there should be a 'coupling' between the rates on > separate controllers. This is the only thing which is puzzling a bit. I > The system is running FreeBSD 8.0, has 16GB of RAM in the system, and > the test program is using O_DIRECT for writes in order to avoid the page > daemon. You should use some benchmark which knows how to deal with OS cache, for example bonnie++ from the ports. Use a benchmark with random IO to see just how horrible your RAID-6 performance will be for random writes.