From owner-freebsd-performance@FreeBSD.ORG Thu Sep 30 12:58:46 2010 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 85D8D106566B for ; Thu, 30 Sep 2010 12:58:46 +0000 (UTC) (envelope-from ssanders@softhammer.net) Received: from oproxy2-pub.bluehost.com (oproxy2-pub.bluehost.com [67.222.39.60]) by mx1.freebsd.org (Postfix) with SMTP id 53C078FC13 for ; Thu, 30 Sep 2010 12:58:46 +0000 (UTC) Received: (qmail 26968 invoked by uid 0); 30 Sep 2010 12:58:45 -0000 Received: from unknown (HELO host358.hostmonster.com) (66.147.240.158) by oproxy2.bluehost.com with SMTP; 30 Sep 2010 12:58:45 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=default; d=softhammer.net; h=Received:Message-ID:Date:From:User-Agent:MIME-Version:To:Subject:References:In-Reply-To:Content-Type:Content-Transfer-Encoding:X-Identified-User; b=4SvrUM7RqfjjYS8xLWxTghWAfEiXkAyl09rl9Mob2LuQ2SdikaLfuG72WvQtVg4h9eDP/+jNOQ7hORYCWKhdSJe0AWRJRAfZDhAh2hkViDm/GOm0KzGiC+rM7ipS1V/i; Received: from pool-74-96-233-244.washdc.fios.verizon.net ([74.96.233.244] helo=[192.168.1.54]) by host358.hostmonster.com with esmtpa (Exim 4.69) (envelope-from ) id 1P1Iij-0000Th-IV for freebsd-performance@freebsd.org; Thu, 30 Sep 2010 06:58:45 -0600 Message-ID: <4CA48984.5030403@softhammer.net> Date: Thu, 30 Sep 2010 08:58:44 -0400 From: Stephen Sanders User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.9) Gecko/20100921 Fedora/3.1.4-1.fc13 Thunderbird/3.1.4 MIME-Version: 1.0 To: freebsd-performance@freebsd.org References: <4CA180FD.9050002@FreeBSD.org> In-Reply-To: <4CA180FD.9050002@FreeBSD.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Identified-User: {2492:host358.hostmonster.com:softhamm:softhammer.net} {sentby:smtp auth 74.96.233.244 authed with ssanders@softhammer.net} Subject: Re: gstripe small transaction size X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Sep 2010 12:58:46 -0000 On 09/28/2010 01:45 AM, Alexander Motin wrote: > Stephen Sanders wrote: >> I'm trying a disk throughput experiment where in two 3ware raid 6's are >> being put into a g_strip raid 0. >> >> The raid 6's are using 8 7200RPM disks. The disk transfer rate is >> ~80MB/s. Using a load generation tool that is using O_DIRECT for I/O, >> I've generated the following short output from iostat. Needless to say, >> the write performance is a lot less than I'm expecting. >> >> We've modified the kernel such that our KB/t figure is closer to 512KB/t >> per disk when measured without the g_strip. With g_strip turned on, the >> KB/t number is more like 60KB/t. >> >> The question is how do I get g_stripe to write larger transactions to >> the disk ? > How old is your system? There was bug, fixed 8-12 months ago, making > fast mode in gstripe not working on systems with increased MAXPHYS. As I > understand, it is what you've changed in your kernel. > Thanks for all of the suggestions. I merged all of the suggestions and got a good result. What worked in the end was to upgrade to FreeBSD 8.1 and : * MAXPHYS is upped to be at least stripe_size * number_of_drives (and possibly 2x that, to use "fast mode" of gstripe) * vfs.hirunningspace is > MAXPHYS * the driver supports IOs of at least your stripe_size * the file system doesn't introduce unnecessary choppiness in IO (i.e. use soft-updates)