From owner-freebsd-performance@FreeBSD.ORG Wed Jul 7 14:30:25 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D730C1065676 for ; Wed, 7 Jul 2010 14:30:25 +0000 (UTC) (envelope-from ssanders@softhammer.net) Received: from smtp-hq2.opnet.com (smtp-hq2.opnet.com [192.104.65.247]) by mx1.freebsd.org (Postfix) with ESMTP id 992698FC20 for ; Wed, 7 Jul 2010 14:30:25 +0000 (UTC) Received: from [172.16.12.251] (wtn12251.opnet.com [172.16.12.251]) by smtp.opnet.com (Postfix) with ESMTPSA id B7F29211009C; Wed, 7 Jul 2010 10:30:24 -0400 (EDT) Message-ID: <4C348F80.7040706@softhammer.net> Date: Wed, 07 Jul 2010 10:30:24 -0400 From: Stephen Sanders User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.10) Gecko/20100512 Lightning/1.0b1 Thunderbird/3.0.5 MIME-Version: 1.0 To: Ivan Voras References: <4C33BDCC.1020004@softhammer.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: More Controllers != Higher Through Put X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Jul 2010 14:30:26 -0000 Thanks for the response. Comments in inline --- On 7/7/2010 10:12 AM, Ivan Voras wrote: > On 07/07/10 01:35, Stephen Sanders wrote: > >> I'm wondering if anyone has heard of this. >> >> I've a system with a 3ware 9650 servicing 4 7200RPM Segate 1TB drives >> and the motherboard servicing 2 7200 RPM Segate 1TB drives. >> > So far so good. > > >> The 4 disk array is RAID 6 while the 2 disk array is RAID 1. The drives >> should deliver about 100MB/s. >> > Ok, so you've reduced the 4-drive array's write performance nearly to > equivalent of 2 drives and the 2-drive array to 1 drive. It should be > even worse for random IOs. > > Since FreeBSD doesn't support RAID-6 I guess you are using hardware RAID? > > For the 2-disk RAID-1 : you are probably using software RAID, right? > (on-board "SATA RAID" controllers usually are just software > implementations). > > The motherboard is an Intel 5000PSL with 2 - 3Ghz Xeon 5450 CPUs. The 3ware controller is a hardware raid implementation. The 2 disk RAID is probably done in software, its a BIOS configuration item. I should research that. An additional data point is that we've modified the kernel to use 128KB writes rather than the default of 64KB. >> 1. The most the 4 disk array is developing is 250MB/s write performance >> > This is too much. It almost looks like something is caching what > shouldn't be cached. How did you get this result? > > I'd expect less than 200 MB/s sequential writes on a 4-drive RAID-6 with > 100 MB/s drives. > > >> while the 2 disk array is coming in at 90MB/s write performance. >> > This is as expected - write performance of any size RAID-1 is equivalent > of 1 drive or less. > > >> The 4 disk array seems slow. >> > Nope - the contrary should be true. It looks like you are doing > something you shouldn't if you get that much performance, or your test > is overly simplistic (e.g. you're testing cache). > > The test program writes 32MB blocks to disk at a fixed rate with O_DIRECT set on the output file. I'll try this with bonnie and see what I get. >> 2. Attempting to write to both arrays simultaneously causes the rate on >> the 4 disk array to drop to 150MB/s and the 2 disk array drops to 60MB/s >> > Are you running on an Atom CPU? What kind of system are you using? > > 2 Xeon 3Ghz E5450 >> I'd expect the 4 disk array should look more like 300+MB/s while the 2 >> disk array is about right. >> > No, you cannot get 300 MB/s from simple RAID6 of 4 drives in any direction. > > Think about it: all the data needs to be a) written as-is to 2 of the > drives, then b) parity/ECCs calculated and c) the same amount of data > written to 2 more drives. You cannot get write performance of more than > 2 drives equivalent in this scheme, and will probably be worse. For > reads, only if your RAID controller is very, very smart (meaning: it > probably isn't), you can recover some performance by using this > parity/ECC data to reconstruct more data than is read from the two > "plain" drives. I think ZFS does this in a limited way. > > >> I don't get why there should be a 'coupling' between the rates on >> separate controllers. >> > This is the only thing which is puzzling a bit. I > > >> The system is running FreeBSD 8.0, has 16GB of RAM in the system, and >> the test program is using O_DIRECT for writes in order to avoid the page >> daemon. >> > You should use some benchmark which knows how to deal with OS cache, for > example bonnie++ from the ports. > > Use a benchmark with random IO to see just how horrible your RAID-6 > performance will be for random writes. > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > >