From owner-freebsd-stable@FreeBSD.ORG Wed Oct 25 21:37:19 2006 Return-Path: X-Original-To: freebsd-stable@freebsd.org Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3703716A40F for ; Wed, 25 Oct 2006 21:37:19 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id E64F743D5E for ; Wed, 25 Oct 2006 21:37:17 +0000 (GMT) (envelope-from freebsd-stable@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1GcqQr-000639-Pv for freebsd-stable@freebsd.org; Wed, 25 Oct 2006 23:37:05 +0200 Received: from 83-131-168-95.adsl.net.t-com.hr ([83.131.168.95]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 25 Oct 2006 23:37:05 +0200 Received: from ivoras by 83-131-168-95.adsl.net.t-com.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Wed, 25 Oct 2006 23:37:05 +0200 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-stable@freebsd.org From: Ivan Voras Date: Wed, 25 Oct 2006 23:37:00 +0200 Lines: 37 Message-ID: References: <20061025132455.GA52157@gvr.gvr.org> <200610251350.k9PDo442086250@lurza.secnetix.de> <20061025211153.GA58556@gvr.gvr.org> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@sea.gmane.org X-Gmane-NNTP-Posting-Host: 83-131-168-95.adsl.net.t-com.hr User-Agent: Thunderbird 1.5.0.7 (Windows/20060909) In-Reply-To: <20061025211153.GA58556@gvr.gvr.org> Sender: news Subject: Re: Gmirror performanc X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Oct 2006 21:37:19 -0000 Guido van Rooij wrote: > But a single-threaded process accessing a large file should also be > able to see a speed increase. I really do not see why a split or roundrobin > approach to a mirror would get only half the performance of a raw access > to a non-mirror. Somehow there must a limit of 128KB of outstanding requests. > That limit is software imposed and should IMHO scale with the number of > spindles in a mirror... I've recently toyed with creating RAID arrays from 2 USB memory sticks (for fun :) ), and this is what I've found out (I may create a web page with the results if I find the time): - diskinfo -t doesn't work on small drives (needs at least 1 GB drive, maybe larger) - maximum read data rate of my USB sticks is ~~ 7.5 MB/s (with dd), but it can be achieved only with large requests (64 KB/s, as seen by iostat) - the speed declines when using smaller requests, to ~~ 4 MB/s with 4 KB requests (CPU utilisation is still ~~ 0 - the slowdown is in the controller or the drive) - gmirror split balance algorithm works as advertised - requests larger than split size are delegated to multiple devices. This has the expected effect of achieving double throughput (15 MB/s) with large enough requests. - there's a (somewhat expected but still surprising) interaction with MAXPHYS (maximum IO size) in the kernel. MAXPHYS is 128 KB, which means requests from TO gmirror (and other GEOM devices) will be <= 128 KB, which gmirror will split in two, so we arrive to maximum size of 64 KB that's requested from the hardware. In case of a four-way mirror, maximum size to hardware would be 32 KB, etc. I don't know if this is what's slowing gmirror down, but it might, since current drives are apparently best tuned for large requests. It probably makes sense to use gmirror split size of around FS block size (16 KB) to make use of split access but not to make too small requests. I've done a round of testing with gstripe and it behaves as expected (and doesn't have the side-effect of splitting requests).