From owner-freebsd-scsi@FreeBSD.ORG Mon May 26 14:10:07 2008 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23EB91065671 for ; Mon, 26 May 2008 14:10:07 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from noop.in-addr.com (in-addr.broker.freenet6.net [IPv6:2001:5c0:8fff:fffe::214d]) by mx1.freebsd.org (Postfix) with ESMTP id E72798FC20 for ; Mon, 26 May 2008 14:10:06 +0000 (UTC) (envelope-from gpalmer@freebsd.org) Received: from gjp by noop.in-addr.com with local (Exim 4.54 (FreeBSD)) id 1K0dOn-000MWz-CM; Mon, 26 May 2008 10:10:05 -0400 Date: Mon, 26 May 2008 10:10:05 -0400 From: Gary Palmer To: Alexander Goncharov Message-ID: <20080526141005.GC1142@in-addr.com> References: <5c29cc10805260541r1f90f516q8ac7d45d2bb15702@mail.gmail.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <5c29cc10805260541r1f90f516q8ac7d45d2bb15702@mail.gmail.com> Cc: freebsd-scsi@freebsd.org Subject: Re: Strange behavior of SCSI RAID 10 (FreeBSD, LSILogic MegaRAID) X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 26 May 2008 14:10:07 -0000 On Mon, May 26, 2008 at 11:41:56PM +1100, Alexander Goncharov wrote: > Hello world, I have faced with following issue on my dedicated server: > > 8x Opteron 885, 32gm RAM, 8x36 GM 15k rpm SCSI with RAID 10 > > FreeBSD 7.0-generic, 64 bit version > > > > IO performance and behavior is very strange: > > 1) No other process are running: > > Memory stat: > > Mem: 8796K Active, 9372K Inact, 80M Wired, 36K Cache, 12M Buf, 31G Free > > Copy 3GB file first time > > dd if=/home/3gb_file of=/home/3gb_file2 > 6291456+0 records in > 6291456+0 records out > 3221225472 bytes transferred in 138.842926 secs (23200501 bytes/sec) > > 20MBS is very poor? > > Memory stat now: > > Mem: 8940K Active, 5951M Inact, 287M Wired, 36K Cache, 214M Buf, 25G Free > > 2) Copy the same file again: > > dd if=/home/3gb_file of=/home/3gb_file2 > 6291456+0 records in > 6291456+0 records out > 3221225472 bytes transferred in 30.433515 secs (105844674 bytes/sec) > > 100MBs ? much better > The "Inact" (Inactive) went up - the 3GiB file is now cached in memory. So the second (and subsequent) runs are going from cached memory so your 100MiB/sec transfer is actually just testing write speed, not read/write speed. This is the same for your other tests too "Inact" is memory that has been used and is being kept around incase it is used again, in other words its caching file data in the "Inact" region in top. It'll be reused if something else needs the memory, but until then it sticks around. Remember - you are copying the file from and to the same filesystem - this is always going to appear slow relative to pure read or pure write tests. A pure write test is effectively what you have when you're getting your 100MiB/sec test result since its just writing out from cache memory. Regards, Gary