From owner-freebsd-stable@FreeBSD.ORG Fri Apr 18 17:18:55 2008 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 57EC11065672 for ; Fri, 18 Apr 2008 17:18:55 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.29]) by mx1.freebsd.org (Postfix) with ESMTP id E163F8FC0A for ; Fri, 18 Apr 2008 17:18:54 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by yw-out-2324.google.com with SMTP id 2so392301ywt.13 for ; Fri, 18 Apr 2008 10:18:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=xwLfqhjzlH5GVmu4yuZoosATnQJY8KGfq7RqihkSvXM=; b=SdQ/KM/kHGyrhuAeZ0OMzCvynkjh0C39TZYmWvARa4fSruiFrqWzjBkehKRLg7Cj4XqVVX6TND/pc6o3xeHmyFa4+HHSh9PiGI/uUxMAHRcfCdzuAAlZzRTKay+Wa8elCp8Yv9DBnb8U9BLXlDk2WRSFehxHstPu4nwW1YB3TXc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=JCe0YxpAU8yfwQm9VXUuO+jLH0THk3c4CumPC/HqCZ4o4F5WftLwp2UwJ4hCzeIIZMIA/VFGjiQHPfL4r2EAhjvzsXIrmCMw/X3SXvc2R0MzIxK5eTbbYwpvCxbnHYDcsRiGOfCSEdUs9332bsq7QLr9IUnK0z2C0gPlbnAYPr0= Received: by 10.150.51.2 with SMTP id y2mr3927285yby.237.1208539123539; Fri, 18 Apr 2008 10:18:43 -0700 (PDT) Received: by 10.150.156.14 with HTTP; Fri, 18 Apr 2008 10:18:43 -0700 (PDT) Message-ID: <5f67a8c40804181018s48b9c652p1c08193eb7af342f@mail.gmail.com> Date: Fri, 18 Apr 2008 13:18:43 -0400 From: "Zaphod Beeblebrox" To: "Pete French" In-Reply-To: MIME-Version: 1.0 References: <5f67a8c40804171155o72b2ab1ctbc116510c39025f3@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: stable@freebsd.org Subject: Re: Dreadful gmirror performance, though each half works fine X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Apr 2008 17:18:55 -0000 On Thu, Apr 17, 2008 at 4:01 PM, Pete French wrote: > > In the end we found that ggate was crashy after a week or two of heavy > use, > > too... dispite it's performance problems (which can be somewhat fixed by > > telling gmirror to only read from the local disk) > > That last part interests me - how did you manage to make it do that ? > I read the man page, and the 'prefer' balancing algorithm should > let you tell it which disc to read from - but there is no mway to > change the priority on a disc in amirror that I can see. It can only > be set when inserting new drives. The ddefault is '0' and hence it's > nbot possible to attach a new drive with a priority below that of > the existing local drive. I tried using '-1' as a priority to fix > this, but it came up as 255. > I would suppose you might have to sync the mirror and then break off and forget the local copy and then sync again. In our case, I'm not sure --- it was awhile ago, but a number of them are also in the 'load' state --- as the higher latency network drives would normally show a higher load. > > Certainly ZFS needs lots of memory --- most of my systems running ZFS > have > > 4G of RAM and are running in 64 bit mode. With the wiki's recomendation > of > > a large number of kernel pages, I havn't had a problem with crashing. I > am > > using ZFS RAIDZ as a large data store and ZFS mirroring (separately) on > my > > workstation as /usr, /var, and home directories. > > All out machine ateb64 bit with between 4 and 16 gig of RAm too, so I > could > try that. So you trust it then ? I;d be interested to know exactly which > options from the wiki page you ended up using for both kernel pages and > ZFS > itself. That would be my ideal solution if it is stable enough. > Hmm. Trust is a funny thing. First the options. On my notebook: vm.kmem_size_max="1073741824" vm.kmem_size="1073741824" On the 32 bit version, I have options KVA_PAGES=512, but the same loader.conf settings above. My large fileserver is used as SMB and NFS filestore for large datasets generally ending in .avi or .iso. Not terribly stressed, I don't think. My laptop is used as a workstation. I don't think I'd run mysql or postgresql on zfs yet --- or if I did, I might run solaris. It's newer there. It would make me nervous at any rate. My laptop is a core-2-duo Extreme 7900 with 4 gig of ram. It will run "make -j8 world" quite quickly on zfs --- and even quicker the 2nd time. I've been running /usr, /var and home directories on zfs for several months now and I havn't had any problems. That said, I take regular (daily) snapshots and I "zfs send" the snapshots to the other zfs array for backup. This is a big advantage to zfs --- snapshots and snapshot backups are fast (and still checksum protected). Now as to trust: ZFS is copy-on-write. From my reading of problems people have, it seems that new data might be corrupted in some manner (there are posts even today about zfs and bittorrent) but the snapshots should not be affected. I hedge my bets further by keeping backups. > > efficient. Removing the read load from the ggated drive seems to help > quite > > a bit in overall performance. But even with this change, I still found > that > > ggate would crash after several days to a week of heavy use. > > Well, I upped the networking buffers and queue sizes to what I woulkd > normally consider 'stupid' values, and now it seems to have settled down > and is performing well (am using then 'load' balancing algorithm). Shall > see if it stays that way for the next few weeks given what you have just > said. I should probably try ZFS on it too, just for my own curiosity. 'load' should almost always prefer the local drive. Large buffers are required to compensate for network latency. Sounds about normal.