From owner-freebsd-stable@freebsd.org Thu Feb 4 01:22:34 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 75956A9A667 for ; Thu, 4 Feb 2016 01:22:34 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 659839F5 for ; Thu, 4 Feb 2016 01:22:34 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: by mailman.ysv.freebsd.org (Postfix) id 6295FA9A666; Thu, 4 Feb 2016 01:22:34 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 620A0A9A665 for ; Thu, 4 Feb 2016 01:22:34 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from ipmail04.adl6.internode.on.net (ipmail04.adl6.internode.on.net [150.101.137.141]) by mx1.freebsd.org (Postfix) with ESMTP id 013E19F1 for ; Thu, 4 Feb 2016 01:22:33 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from ppp121-45-70-178.lns20.adl6.internode.on.net (HELO leader.local) ([121.45.70.178]) by ipmail04.adl6.internode.on.net with ESMTP; 04 Feb 2016 11:45:59 +1030 Subject: Re: 10-STABLE hangups frequently To: Peter Jeremy References: <20160202183913.GG8270@graf.pompo.net> <56B1B1E9.2090707@ShaneWare.Biz> <20160203190323.GC78969@server.rulingia.com> Cc: stable@FreeBSD.org From: Shane Ambler Message-ID: <56B2A64C.3050107@ShaneWare.Biz> Date: Thu, 4 Feb 2016 11:45:56 +1030 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.4.0 MIME-Version: 1.0 In-Reply-To: <20160203190323.GC78969@server.rulingia.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Feb 2016 01:22:34 -0000 On 04/02/2016 05:33, Peter Jeremy wrote: > On 2016-Feb-03 18:23:13 +1030, Shane Ambler wrote: >> Any chance you get high wired allocations? > > A high wired allocation is normal for ZFS - ARC shows up as "wired" > memory. > >> Sometimes several times in a day I see the wired amount shown in top >> rise to over 6GB (of 8GB) bringing the system to a crawl. When wired >> gets over 7GB the system rarely recovers. > > The ARC limit defaults to 1GB less than physical RAM so 6GB wired on > an 8GB system isn't unexpected (my home system currently has 30GB > wired out of 32GB). If this is causing problems for your workload, it > sounds like you may need to explicitly reduce vfs.zfs.arc_max (note > that this is a soft limit). I have vfs.zfs.arc_max=2G - and now realise I also had set vfs.zfs.arc_min=500M some time ago. Don't recall where I got it but I also have vfs.zfs.dirty_data_max=200M Going by figures shown in top, ARC is usually in the 1500M to 2000M range but when wired gets over 6GB I often see ARC drop to 500MB which I now realise matches arc_min. When wired gets over 6GB apps start to stop responding, I then try to push out some wired by running a script that allocates 4G, 9 times out of 10 this drops wired to under 4GB and everything keeps working. If I notice wired over 7GB there's usually no recover. I don't always get to see the figures before it totally locks up. On desktop1 I leave a terminal running top and another ready to run my script to allocate some ram. When an app stops responding I go to desktop1 and depending on what I see I manually allocate some ram to flush things out. Currently I have an uptime of 9 days 9 hours - I have manually allocated 4GB 30 times in that time. -- FreeBSD - the place to B...Software Developing Shane Ambler