From owner-freebsd-stable@FreeBSD.ORG Thu Apr 17 18:55:22 2008 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 666F5106566B for ; Thu, 17 Apr 2008 18:55:22 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.31]) by mx1.freebsd.org (Postfix) with ESMTP id 1C40D8FC0A for ; Thu, 17 Apr 2008 18:55:21 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by yw-out-2324.google.com with SMTP id 2so121959ywt.13 for ; Thu, 17 Apr 2008 11:55:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=BFQc2KFhS1YDacds7jrpoTpf7VDhNisk49EwSuXeH4k=; b=Q7D7aa76lhcww4dBQsQkX3WWT2mICLUnzCgfaEwPn2D+EHbTMm2jubDov9015gTX6EL1pYLawFCq7LUpn8Fd0JM8mzypZA4lwzmruP32LaHfLs6lDE3aIT+be4sb2KkzfkNyytrSIP3IXPCI14mHb4Dnr5P2PV5EeZOeBWnGpek= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=oY7ijgKvwEL/dWC/6uOCgeS7NEeTw2fhImgR8wgxM/IoHnhb2OfUDI5Tw7zH7CmjvlyGRFb2kaB0va62G8Rvlr3CBW5Ez6eQ7tDJzgmj/mNPimRF/y74RES2rq9D9OkVzodB7xEMR9s0qFiN2WmiJO6c9Zx0sBieUt0B+gBPG+k= Received: by 10.150.189.9 with SMTP id m9mr2140971ybf.73.1208458512765; Thu, 17 Apr 2008 11:55:12 -0700 (PDT) Received: by 10.150.156.14 with HTTP; Thu, 17 Apr 2008 11:55:12 -0700 (PDT) Message-ID: <5f67a8c40804171155o72b2ab1ctbc116510c39025f3@mail.gmail.com> Date: Thu, 17 Apr 2008 14:55:12 -0400 From: "Zaphod Beeblebrox" To: "Pete French" In-Reply-To: MIME-Version: 1.0 References: <5f67a8c40804171109o264ad120wa442f21be8a4bb33@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: stable@freebsd.org Subject: Re: Dreadful gmirror performance, though each half works fine X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Apr 2008 18:55:22 -0000 On Thu, Apr 17, 2008 at 2:25 PM, Pete French wrote: > > I have experimented with this rather extensively and have operationally > > decided not to use ggated in combination with gmirror --- it doesn't > appear > > to work as well as one might expect. > > Ah, thats unmfortunate :-( I oroginally started off using the > iscsi initiator and target, which did work O.K., but when actually > ran live ended up locking up after several hours,a nd then panicing > the kernel. So not ideal - but when it was working it was fine. ggated > seems the opposite - doesnt crash, but performance is not suitable for any > kind of real use. > In the end we found that ggate was crashy after a week or two of heavy use, too... dispite it's performance problems (which can be somewhat fixed by telling gmirror to only read from the local disk) > I'm somewhat vaguely wondering if zfs with one local and one ggated disk > > will work well. > > I tried ZFS for a while myself, and it works O.K., but has a tendency > to panic if it wants memory which it can't get. Despite the many different > guides available, I never managed to get it to the point where I would > be happy to use it on a production system without worrying about it > suddenly becomming memory hungry and dieing. Certainly ZFS needs lots of memory --- most of my systems running ZFS have 4G of RAM and are running in 64 bit mode. With the wiki's recomendation of a large number of kernel pages, I havn't had a problem with crashing. I am using ZFS RAIDZ as a large data store and ZFS mirroring (separately) on my workstation as /usr, /var, and home directories. > Thanks for the inout though - I am doing some more experimentation > with ggate (basically raing some buffers as per a thread I found) and > seeing if that helps. > > BTW, I think ggate is the problem and not gmirror here - gmirror on top > of iscsi works fine as I said. > I would agree... save the fact that it may be an interaction between the two and/or UFS that is causing the problems. Certainly gmirror on local disks works fine (I've run gmirror/gstripe combinations for several years now as RAID 10 store with UFS on top). This is all going to be latency sensitive --- ggate needs to allow a larger number of oustanding transactions to be efficient. Removing the read load from the ggated drive seems to help quite a bit in overall performance. But even with this change, I still found that ggate would crash after several days to a week of heavy use.