From owner-freebsd-questions@freebsd.org Fri Jan 22 16:34:21 2016 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 61141A8D28D for ; Fri, 22 Jan 2016 16:34:21 +0000 (UTC) (envelope-from mason@blisses.org) Received: from phlegethon.blisses.org (phlegethon.blisses.org [50.56.97.101]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4A8F81B86 for ; Fri, 22 Jan 2016 16:34:20 +0000 (UTC) (envelope-from mason@blisses.org) Received: from blisses.org (cocytus.blisses.org [23.25.209.73]) by phlegethon.blisses.org (Postfix) with ESMTPSA id E22EF1493C3; Fri, 22 Jan 2016 11:34:18 -0500 (EST) Date: Fri, 22 Jan 2016 11:34:17 -0500 From: Mason Loring Bliss To: Warren Block , freebsd-questions@freebsd.org, kpneal@pobox.com, markham breitbach , Priyadarshan Subject: Re: ZFS performance help sought Message-ID: <20160122163417.GS4538@blisses.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1453455663.2364898.499431058.4EA85780@webmail.messagingengine.com> <56A24D25.70206@corp.ssimicro.com> <20160122135452.GC12085@neutralgood.org> <20160122140549.053fea0f@fabiankeil.de> User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jan 2016 16:34:21 -0000 On Thu, Jan 21, 2016 at 11:17:05PM -0700, Warren Block wrote: > I know there was some sort of ZFS resource exhaustion issue worked on > recently, but I don't recally exactly when or what the symptoms were. It > might only be in 10-stable so far. Best to ask on the freebsd-fs mailing > list. Thank you. I'm subscribing to that now. > It's conceivable that you are seeing the result of lock contention that > is slowing down arc_get_data_buf(), for details see: > https://www.fabiankeil.de/gehacktes/electrobsd/zfs-arc-tuning/ I will look at that and try the dtrace script soon. Thank you. > Even if it's a different issue, you may be able to work around it by > throttling the send/receive throughput with mbuffer or a similar tool. Most of the time I schedule the bulk moves for hours where I wouldn't notice the system being bogged down, so I'm more interested in identifying it than working around it, but that's worth noting, especially given that there is a very easy chokepoint to be had in the pipeline. On Fri, Jan 22, 2016 at 08:54:52AM -0500, kpneal@pobox.com wrote: > FWIW, I have a machine with a three way mirror, SAS 3Gb, 8GB RAM, and when > mutt saves a roughly 3GB mailbox it pushes the machine into swap. I'll move further correspondence to freebsd-fs once I've confirmed my subscription there, but I'd be curious to know if you can suggest a simple test case there I can reproduce. Is it one big mbox file, as opposed to maildir? Are you deleting a mail from the middle, or doing some other operation? On my end, I take a dataset with maybe 130G in it and I remove it from back-ups, and then re-ship it to tickle the issue. On Fri, Jan 22, 2016 at 08:39:17AM -0700, markham breitbach wrote: > There were some ZFS threads on freebsd-performance a while back too. It > might be worth throwing the question out that way too. Thank you. I will subscribe and look at archives there too. -- Mason Loring Bliss mason@blisses.org Ewige Blumenkraft! awake ? sleep : random() & 2 ? dream : sleep; -- Hamlet, Act III, Scene I