From owner-freebsd-current@freebsd.org Wed Jun 27 04:52:10 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8E24B102E9C5 for ; Wed, 27 Jun 2018 04:52:10 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 2913793D32 for ; Wed, 27 Jun 2018 04:52:10 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: by mailman.ysv.freebsd.org (Postfix) id D39AB102E9BC; Wed, 27 Jun 2018 04:52:09 +0000 (UTC) Delivered-To: current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2064102E9BB for ; Wed, 27 Jun 2018 04:52:09 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from ipmail06.adl2.internode.on.net (ipmail06.adl2.internode.on.net [150.101.137.129]) by mx1.freebsd.org (Postfix) with ESMTP id 0943B93D2E for ; Wed, 27 Jun 2018 04:52:08 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from ppp118-210-121-130.bras2.adl4.internode.on.net (HELO leader.local) ([118.210.121.130]) by ipmail06.adl2.internode.on.net with ESMTP; 27 Jun 2018 14:21:59 +0930 Subject: Re: numa involved in instability and swap usage despite RAM free? Cc: "current@freebsd.org" References: <20180624120329.Horde.HWORumQ7Ng1KAUeviJNtoc3@webmail.leidinger.net> <20180625182250.GA40651@troutmask.apl.washington.edu> <20180626222251.GA30202@troutmask.apl.washington.edu> From: Shane Ambler Message-ID: Date: Wed, 27 Jun 2018 14:21:57 +0930 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <20180626222251.GA30202@troutmask.apl.washington.edu> Content-Type: text/plain; charset=utf-8 Content-Language: en-AU Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jun 2018 04:52:10 -0000 On 27/06/2018 07:52, Steve Kargl wrote: > On Tue, Jun 26, 2018 at 02:39:27PM -0700, Adrian Chadd wrote: >> On Mon, 25 Jun 2018 at 11:23, Steve Kargl >> wrote: >>> >>> On Sun, Jun 24, 2018 at 12:03:29PM +0200, Alexander Leidinger wrote: >>>> >>>> I don't have hard evidence, but there is enough "smell" to open up a >>>> discussion... >>>> >>>> Short: >>>> Can it be that enabling numa in the kernel is the reason why some >>>> people see instability with zfs and usage of swap while a lot of free >>>> RAM is available? >>> >>> Interesting observation. I do have NUMA in my kernel, and swap >>> seems to be used instead of recycling freeing inactive memory. >>> Top shows >>> >>> Mem: 506M Active, 27G Inact, 98M Laundry, 2735M Wired, 1474M Buf, 1536M Free >>> Swap: 16G Total, 120M Used, 16G Free >From someone that has had memory issues since 10.1 (bug 194654), I have recently realised something that seems to make some sense to me. The arc_max setting is a limit of zfs arc and this ram gets wired to prevent it swapping, this makes sense. The vm.max_wired is a value that I had thought was ignored but now I see that these are two values of wired memory which are not connected. max_wired appears to default to 30% of kmem_size. Both of these values are added together to be reported in vm.stats.vm.v_wire_count which is the wired value shown by top. This appears to be the reason that I can see 9G wired when max_wired is at 5G The implications of this is that together (arc_max + max_wired) can be set to more than the physical installed ram. I can verify that with 8G installed and the two values add up to more than 7G you get no choice but a hard reset. Since upgrading to 16G I have been more vigilant and not allowed more than 10G to be wired so haven't had that problem in a year and a half. With the default arc_max usually set to ram minus 1G and max_wired at 5G it is easy to see that the current defaults are dangerous. I have not seen max_wired mentioned in relation to zfs but it seems that it should be considered when setting arc_max to prevent over wiring ram. Close to three weeks ago I applied review D7538 to my everyday desktop running stable/11. Until a few days ago I had no swap usage which is now at 9M. In the last few years of monitoring wired usage to try and find a solution I have not seen less than 1G of swap usage after an hour of uptime. If nothing else D7538 makes arc more willing to be released. -- FreeBSD - the place to B...Storing Data Shane Ambler