From owner-freebsd-stable@FreeBSD.ORG Wed Mar 5 15:08:22 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10B7A1065671 for ; Wed, 5 Mar 2008 15:08:22 +0000 (UTC) (envelope-from erik@tefre.com) Received: from mta1-filtered.netlife.no (mail.netlife.no [213.187.191.68]) by mx1.freebsd.org (Postfix) with ESMTP id B7FCF8FC23 for ; Wed, 5 Mar 2008 15:08:21 +0000 (UTC) (envelope-from erik@tefre.com) Received: from localhost (unknown [10.0.68.2]) by mta1-filtered.netlife.no (Postfix) with ESMTP id 663C42875A for ; Wed, 5 Mar 2008 16:08:20 +0100 (CET) X-Virus-Scanned: amavisd-new at netlife.no Received: from mta1.netlife.no ([10.0.68.2]) by localhost (amavis.netlife.no [10.0.68.2]) (amavisd-new, port 10024) with ESMTP id Bjkmas5Kciot for ; Wed, 5 Mar 2008 15:08:14 +0000 (UTC) Received: from baviandesktop.netlife.no (kontor.netlife.no [217.13.28.50]) by mta1.netlife.no (Postfix) with ESMTP id 72AB928763 for ; Wed, 5 Mar 2008 16:08:14 +0100 (CET) Message-ID: <47CEB75E.7080906@tefre.com> Date: Wed, 05 Mar 2008 16:08:14 +0100 From: Erik Stian Tefre User-Agent: Thunderbird 2.0.0.9 (X11/20080127) MIME-Version: 1.0 To: freebsd-stable@freebsd.org References: <915419.88748.qm@web50505.mail.re2.yahoo.com> <47CE816A.6050903@tefre.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: 7.0-Release and 3ware 9550SXU w/BBU - horrible write performance X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Mar 2008 15:08:22 -0000 Ivan Voras wrote: > Erik Stian Tefre wrote: > >> Results from bonnie++ with default settings on my box (7.0-RELEASE, >> amd64, 480 MB system memory, BBU, write cache enabled, 7 drive raid-5, >> storsave = balance, NCQ on): >> >> Version 1.93d ------Sequential Output------ --Sequential Input- >> --Random- >> Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- >> --Seeks-- >> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP >> /sec %CP >> xxxxxxxxxxxxxxxx 1G 275 83 72440 23 24990 6 644 83 76548 15 >> 460.6 6 > > Was the array busy when you did the test? 72-76 MB MB/s is very slow for > a 7-drive array in RAID-5. FreeBSD's disk IO can be sluggish but I doubt > it would get that slow. On a RAID10 array with 6 drives I can get ~~ 200 > MB/s in both directions (though on a different controller). RAID5 is > basically striped so your setup should on average yield close to 6 times > the single drive performance, something in the area of 350 MB/s. It was completely idle. Changing vfs.read_max to 80 triples the sequential read performance, see bonnie++ output below (run on the same box, nothing changed except vfs.read_max). I tried setting it to 256 as recommended by 3ware, but 80 seems to be the sweet spot. 3ware performance tips for FreeBSD: http://www.3ware.com/KB/article.aspx?id=14852 I have measured similar write performance on 2 other servers with the same controller model and the same number of disks. By the way, the newer 9650SE controllers seem to be a lot faster, I remember getting around 350 MB/s of sequential reads/writes from an idle 16 drive raid-6 array. (Sorry, I have no idle 9650SE to run bonnie++ on at the moment.) bonnie++ with vfs_max=80: Version 1.93d ------Sequential Output------ --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP xxxxxxxxxxxxxxxx 1G 285 83 88193 28 26858 6 690 89 218642 59 406.5 5 Latency 44353us 557ms 542ms 90795us 209ms 361ms Version 1.93d ------Sequential Create------ --------Random Create-------- xxxxxxxxxxxxxxxxxxx -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 8204 19 +++++ +++ 22163 36 9925 25 +++++ +++ +++++ +++ Latency 585ms 571us 3844us 257ms 24015us 768us -- Erik