From owner-freebsd-stable@FreeBSD.ORG Thu Apr 17 20:01:04 2008 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6F4431065674 for ; Thu, 17 Apr 2008 20:01:04 +0000 (UTC) (envelope-from petefrench@ticketswitch.com) Received: from angel.ticketswitch.com (angel.ticketswitch.com [IPv6:2002:57e0:1d4e::1]) by mx1.freebsd.org (Postfix) with ESMTP id 2ABD28FC0A for ; Thu, 17 Apr 2008 20:01:04 +0000 (UTC) (envelope-from petefrench@ticketswitch.com) Received: from [10.50.50.2] (helo=smaug.rattatosk) by angel.ticketswitch.com with esmtps (TLSv1:AES256-SHA:256) (Exim 4.69 (FreeBSD)) (envelope-from ) id 1JmaI1-000DHS-Ku; Thu, 17 Apr 2008 21:01:01 +0100 Received: from dilbert.rattatosk ([10.50.50.6] helo=dilbert.ticketswitch.com) by smaug.rattatosk with esmtp (Exim 4.69 (FreeBSD)) (envelope-from ) id 1JmaI1-000ARn-IT; Thu, 17 Apr 2008 21:01:01 +0100 Received: from petefrench by dilbert.ticketswitch.com with local (Exim 4.69 (FreeBSD)) (envelope-from ) id 1JmaI1-000GTm-He; Thu, 17 Apr 2008 21:01:01 +0100 To: zbeeble@gmail.com In-Reply-To: <5f67a8c40804171155o72b2ab1ctbc116510c39025f3@mail.gmail.com> Message-Id: From: Pete French Date: Thu, 17 Apr 2008 21:01:01 +0100 Cc: stable@freebsd.org Subject: Re: Dreadful gmirror performance, though each half works fine X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Apr 2008 20:01:04 -0000 > In the end we found that ggate was crashy after a week or two of heavy use, > too... dispite it's performance problems (which can be somewhat fixed by > telling gmirror to only read from the local disk) That last part interests me - how did you manage to make it do that ? I read the man page, and the 'prefer' balancing algorithm should let you tell it which disc to read from - but there is no mway to change the priority on a disc in amirror that I can see. It can only be set when inserting new drives. The ddefault is '0' and hence it's nbot possible to attach a new drive with a priority below that of the existing local drive. I tried using '-1' as a priority to fix this, but it came up as 255. > Certainly ZFS needs lots of memory --- most of my systems running ZFS have > 4G of RAM and are running in 64 bit mode. With the wiki's recomendation of > a large number of kernel pages, I havn't had a problem with crashing. I am > using ZFS RAIDZ as a large data store and ZFS mirroring (separately) on my > workstation as /usr, /var, and home directories. All out machine ateb64 bit with between 4 and 16 gig of RAm too, so I could try that. So you trust it then ? I;d be interested to know exactly which options from the wiki page you ended up using for both kernel pages and ZFS itself. That would be my ideal solution if it is stable enough. > efficient. Removing the read load from the ggated drive seems to help quite > a bit in overall performance. But even with this change, I still found that > ggate would crash after several days to a week of heavy use. Well, I upped the networking buffers and queue sizes to what I woulkd normally consider 'stupid' values, and now it seems to have settled down and is performing well (am using then 'load' balancing algorithm). Shall see if it stays that way for the next few weeks given what you have just said. I should probably try ZFS on it too, just for my own curiosity. cheers, -pete.