From owner-freebsd-performance@FreeBSD.ORG Wed Jul 21 01:12:31 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 202831065676 for ; Wed, 21 Jul 2010 01:12:31 +0000 (UTC) (envelope-from ssanders@softhammer.net) Received: from oproxy1-pub.bluehost.com (oproxy1-pub.bluehost.com [66.147.249.253]) by mx1.freebsd.org (Postfix) with SMTP id E62C98FC12 for ; Wed, 21 Jul 2010 01:12:30 +0000 (UTC) Received: (qmail 31980 invoked by uid 0); 21 Jul 2010 00:13:21 -0000 Received: from unknown (HELO host358.hostmonster.com) (66.147.240.158) by oproxy1.bluehost.com.bluehost.com with SMTP; 21 Jul 2010 00:13:21 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=softhammer.net; h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:References:In-Reply-To:X-Enigmail-Version:Content-Type:Content-Transfer-Encoding:X-Identified-User; b=r6KpAKw8T2+Nap0mZtDP66rmBAqz7EEe9EKP1I+BGBpcakJvQdIl9ghLokrrI9n+fw3cvgLsbt1HAAzazbrA7kSoyGP+uTuPyDrUITycC+2RUaTC6H8Tg50yeIrbfQz4; Received: from pool-74-96-233-244.washdc.fios.verizon.net ([74.96.233.244] helo=onyx.softhammer.net) by host358.hostmonster.com with esmtpa (Exim 4.69) (envelope-from ) id 1ObNrJ-0001v5-Ul for freebsd-performance@freebsd.org; Tue, 20 Jul 2010 19:12:30 -0600 Message-ID: <4C46497D.7030207@softhammer.net> Date: Tue, 20 Jul 2010 21:12:29 -0400 From: Stephen Sanders User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.1.9) Gecko/20100430 Fedora/3.0.4-2.fc12 Lightning/1.0b2pre Thunderbird/3.0.4 MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <4C33BDCC.1020004@softhammer.net> In-Reply-To: X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Identified-User: {2492:host358.hostmonster.com:softhamm:softhammer.net} {sentby:smtp auth 74.96.233.244 authed with ssanders@softhammer.net} Subject: Re: More Controllers != Higher Through Put X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Jul 2010 01:12:31 -0000 We found that in order to get the dual controller case up to speed we had to set vfs.lorunningspace=1MB and vfs.hirunningspace=8MB. On 07/07/2010 10:12 AM, Ivan Voras wrote: > On 07/07/10 01:35, Stephen Sanders wrote: > >> I'm wondering if anyone has heard of this. >> >> I've a system with a 3ware 9650 servicing 4 7200RPM Segate 1TB drives >> and the motherboard servicing 2 7200 RPM Segate 1TB drives. >> > So far so good. > > >> The 4 disk array is RAID 6 while the 2 disk array is RAID 1. The drives >> should deliver about 100MB/s. >> > Ok, so you've reduced the 4-drive array's write performance nearly to > equivalent of 2 drives and the 2-drive array to 1 drive. It should be > even worse for random IOs. > > Since FreeBSD doesn't support RAID-6 I guess you are using hardware RAID? > > For the 2-disk RAID-1 : you are probably using software RAID, right? > (on-board "SATA RAID" controllers usually are just software > implementations). > > >> 1. The most the 4 disk array is developing is 250MB/s write performance >> > This is too much. It almost looks like something is caching what > shouldn't be cached. How did you get this result? > > I'd expect less than 200 MB/s sequential writes on a 4-drive RAID-6 with > 100 MB/s drives. > > >> while the 2 disk array is coming in at 90MB/s write performance. >> > This is as expected - write performance of any size RAID-1 is equivalent > of 1 drive or less. > > >> The 4 disk array seems slow. >> > Nope - the contrary should be true. It looks like you are doing > something you shouldn't if you get that much performance, or your test > is overly simplistic (e.g. you're testing cache). > > >> 2. Attempting to write to both arrays simultaneously causes the rate on >> the 4 disk array to drop to 150MB/s and the 2 disk array drops to 60MB/s >> > Are you running on an Atom CPU? What kind of system are you using? > > >> I'd expect the 4 disk array should look more like 300+MB/s while the 2 >> disk array is about right. >> > No, you cannot get 300 MB/s from simple RAID6 of 4 drives in any direction. > > Think about it: all the data needs to be a) written as-is to 2 of the > drives, then b) parity/ECCs calculated and c) the same amount of data > written to 2 more drives. You cannot get write performance of more than > 2 drives equivalent in this scheme, and will probably be worse. For > reads, only if your RAID controller is very, very smart (meaning: it > probably isn't), you can recover some performance by using this > parity/ECC data to reconstruct more data than is read from the two > "plain" drives. I think ZFS does this in a limited way. > > >> I don't get why there should be a 'coupling' between the rates on >> separate controllers. >> > This is the only thing which is puzzling a bit. I > > >> The system is running FreeBSD 8.0, has 16GB of RAM in the system, and >> the test program is using O_DIRECT for writes in order to avoid the page >> daemon. >> > You should use some benchmark which knows how to deal with OS cache, for > example bonnie++ from the ports. > > Use a benchmark with random IO to see just how horrible your RAID-6 > performance will be for random writes. > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > >