From owner-freebsd-stable@FreeBSD.ORG Wed Oct 25 15:43:16 2006 Return-Path: X-Original-To: freebsd-stable@FreeBSD.ORG Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BB64B16A415 for ; Wed, 25 Oct 2006 15:43:16 +0000 (UTC) (envelope-from michel@lpthe.jussieu.fr) Received: from shiva.jussieu.fr (shiva.jussieu.fr [134.157.0.129]) by mx1.FreeBSD.org (Postfix) with ESMTP id 05D5A43D5E for ; Wed, 25 Oct 2006 15:43:15 +0000 (GMT) (envelope-from michel@lpthe.jussieu.fr) Received: from parthe.lpthe.jussieu.fr (parthe.lpthe.jussieu.fr [134.157.10.1]) by shiva.jussieu.fr (8.13.7/jtpda-5.4) with ESMTP id k9PFhDiW024231 ; Wed, 25 Oct 2006 17:43:14 +0200 (CEST) X-Ids: 164 Received: from niobe.lpthe.jussieu.fr (niobe.lpthe.jussieu.fr [134.157.10.41]) by parthe.lpthe.jussieu.fr (Postfix) with ESMTP id 9D9019FE2C; Wed, 25 Oct 2006 17:43:13 +0200 (CEST) Received: by niobe.lpthe.jussieu.fr (Postfix, from userid 2005) id E97211D; Wed, 25 Oct 2006 17:43:34 +0200 (CEST) Date: Wed, 25 Oct 2006 17:43:34 +0200 From: Michel Talon To: freebsd-stable@FreeBSD.ORG, guido@gvr.org Message-ID: <20061025154334.GA45269@lpthe.jussieu.fr> Mail-Followup-To: Michel Talon , freebsd-stable@FreeBSD.ORG, guido@gvr.org Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.1i X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-2.0.2 (shiva.jussieu.fr [134.157.0.164]); Wed, 25 Oct 2006 17:43:14 +0200 (CEST) X-Virus-Scanned: ClamAV 0.88.2/2098/Wed Oct 25 15:14:20 2006 on shiva.jussieu.fr X-Virus-Status: Clean X-Miltered: at shiva.jussieu.fr with ID 453F8611.002 by Joe's j-chkmail (http://j-chkmail.ensmp.fr)! Cc: Subject: Re: Gmirror performanc X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Oct 2006 15:43:16 -0000 Oliver Fromme wrote: > > I tried with -b split -s , -b round-robin, -b load. > > (dd-ing as done with a bs of 1m; I see the transaction size is 128Kb, > > unless the split method is used, in which case the transaction size > > gies down. When round-robin is used, the transaction size is 128Kb/s, > > but the number of transaction per second goes down.). > > > > I cannot explain why I should not get a higher read speed. Anyone? > > dd is a sequential, single-threaded operation, so it will > only use one disk at a time. It's not really suitable as > a benchmark for real-world things. I see the same problem as Guido, that is gmirror on two disks is not faster that each disk separately and is even markedly slower. I disagree with your explanation, which moreover contradicts the definition of split, round-robin, etc. in the man page of gmirror. Experimentally, observing the disk throughput via iostat shows that both disks are involved in the IO. asmodee# dd if=/dev/mirror/gms1a of=/dev/null bs=256k 556+0 records in 556+0 records out 145752064 bytes transferred in 8.785839 secs (16589431 bytes/sec) (note the low throughput, the disks are individually able to do more than 20 MB/s, this is on an old machine), while iostat shows: asmodee% iostat -w 1 tty ad0 ad4 cpu tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 60 61.38 116 6.95 61.40 117 7.01 0 0 1 0 99 0 60 64.00 129 8.06 64.00 128 7.99 0 0 2 1 97 0 60 64.00 128 7.99 64.00 129 8.06 0 0 2 1 97 0 60 64.00 129 8.06 64.00 129 8.06 0 0 1 1 98 0 60 63.50 128 7.93 63.50 127 7.87 0 0 5 2 94 that is the transaction is evenly distributed on both disks ad0 and ad4 (which are on 2 separate channels). The problem is that each disk works at only 8MB/s while it is able of 3 times more. Looking at the kernel driver /usr/src/sys/geom/mirror/g_mirror.c, it seems that the load is split on the various disks in the function g_mirror_request_split() in a way which is simulatneous for all providers. How is it that after that the request proceeds so slowly, i don't know. But i doubt very much it will be any different wether you have a "real world load" or a simple dd. -- Michel TALON