From owner-freebsd-stable@FreeBSD.ORG  Tue Mar  5 23:08:38 2013
Return-Path: <owner-freebsd-stable@FreeBSD.ORG>
Delivered-To: stable@freebsd.org
Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115])
 by hub.freebsd.org (Postfix) with ESMTP id 82FD510B;
 Tue,  5 Mar 2013 23:08:38 +0000 (UTC)
 (envelope-from daniel@digsys.bg)
Received: from smtp-sofia.digsys.bg (smtp-sofia.digsys.bg [193.68.21.123])
 by mx1.freebsd.org (Postfix) with ESMTP id DB92A26D;
 Tue,  5 Mar 2013 23:08:37 +0000 (UTC)
Received: from digsys200-136.pip.digsys.bg (digsys200-136.pip.digsys.bg
 [193.68.136.200]) (authenticated bits=0)
 by smtp-sofia.digsys.bg (8.14.6/8.14.6) with ESMTP id r25N8Wrb058505
 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO);
 Wed, 6 Mar 2013 01:08:33 +0200 (EET) (envelope-from daniel@digsys.bg)
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
Subject: Re: ZFS "stalls" -- and maybe we should be talking about defaults?
From: Daniel Kalchev <daniel@digsys.bg>
In-Reply-To: <CAOjFWZ7TDYgS=vvcuLRPr+fAv4bPk4ZU0CKbgf1BydxG1MeOYw@mail.gmail.com>
Date: Wed, 6 Mar 2013 01:08:28 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <22720B5D-7BCA-41BC-B1E8-A2ACB2ADB795@digsys.bg>
References: <513524B2.6020600@denninger.net>
 <1362449266.92708.8.camel@btw.pki2.com> <51355F64.4040409@denninger.net>
 <201303050540.r255ecEC083742@hergotha.csail.mit.edu>
 <20130305152252.GA52706@in-addr.com>
 <CAOjFWZ7TDYgS=vvcuLRPr+fAv4bPk4ZU0CKbgf1BydxG1MeOYw@mail.gmail.com>
To: Freddie Cash <fjwcash@gmail.com>
X-Mailer: Apple Mail (2.1499)
Cc: Garrett Wollman <wollman@hergotha.csail.mit.edu>, stable@freebsd.org,
 Steven Hartland <killing@multiplay.co.uk>
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.14
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 05 Mar 2013 23:08:38 -0000


On Mar 5, 2013, at 8:17 PM, Freddie Cash <fjwcash@gmail.com> wrote:

>=20
> ZFS send/recv would eventually complete, but what used to take 15-20
> minutes would take 6-8 hours to complete.
>=20
> I've reduced the ARC to only 32 GB, with arc_meta set to 28 GB, and =
things
> are running much smoother now (50-200 MB/s writes for 3-5 seconds =
every
> 10s), and send/recv is back down to 10-15 minutes.
>=20
> Who would have thought "too much RAM" would be an issue?
>=20
> Will play with this over the next couple of days with different ARC =
max
> settings to see where the problems start.  All of our ZFS boxes until =
this
> one had under 64 GB of RAM.  (And we had issues with dedupe enabled on
> boxes with too little RAM, as in under 32 GB.)

I have an archive box running very similar setup as yours, but with 72GB =
of RAM. I have set both arc_max and arc_meta_limit to 64GB, with no =
issues. I am still doing a very complex snapshot reordering between two =
pools. One of the pools has dedup enabled (which prompted me to add =
RAM), with dedup ratio f over 10x and there are still no issues or any =
stalling. The other pool has both dedup and compression for some =
filesystems.=20

My only issue is that replacing a drive in either pool takes few days =
(6-drive vdevs of 3TB drives).

Perhaps the memory indexing/search algorithms are inefficient?

Daniel=