Date: Wed, 27 Jun 2018 14:21:57 +0930 From: Shane Ambler <FreeBSD@ShaneWare.Biz> Cc: "current@freebsd.org" <current@freebsd.org> Subject: Re: numa involved in instability and swap usage despite RAM free? Message-ID: <be613c69-589a-93e8-0056-b1400d0116a8@ShaneWare.Biz> In-Reply-To: <20180626222251.GA30202@troutmask.apl.washington.edu> References: <20180624120329.Horde.HWORumQ7Ng1KAUeviJNtoc3@webmail.leidinger.net> <20180625182250.GA40651@troutmask.apl.washington.edu> <CAJ-VmokW6bfTov8tvKQcpcVaKj=dxvPjmuTsejPVEh0mpKGrPg@mail.gmail.com> <20180626222251.GA30202@troutmask.apl.washington.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
On 27/06/2018 07:52, Steve Kargl wrote: > On Tue, Jun 26, 2018 at 02:39:27PM -0700, Adrian Chadd wrote: >> On Mon, 25 Jun 2018 at 11:23, Steve Kargl >> <sgk@troutmask.apl.washington.edu> wrote: >>> >>> On Sun, Jun 24, 2018 at 12:03:29PM +0200, Alexander Leidinger wrote: >>>> >>>> I don't have hard evidence, but there is enough "smell" to open up a >>>> discussion... >>>> >>>> Short: >>>> Can it be that enabling numa in the kernel is the reason why some >>>> people see instability with zfs and usage of swap while a lot of free >>>> RAM is available? >>> >>> Interesting observation. I do have NUMA in my kernel, and swap >>> seems to be used instead of recycling freeing inactive memory. >>> Top shows >>> >>> Mem: 506M Active, 27G Inact, 98M Laundry, 2735M Wired, 1474M Buf, 1536M Free >>> Swap: 16G Total, 120M Used, 16G Free >From someone that has had memory issues since 10.1 (bug 194654), I have recently realised something that seems to make some sense to me. The arc_max setting is a limit of zfs arc and this ram gets wired to prevent it swapping, this makes sense. The vm.max_wired is a value that I had thought was ignored but now I see that these are two values of wired memory which are not connected. max_wired appears to default to 30% of kmem_size. Both of these values are added together to be reported in vm.stats.vm.v_wire_count which is the wired value shown by top. This appears to be the reason that I can see 9G wired when max_wired is at 5G The implications of this is that together (arc_max + max_wired) can be set to more than the physical installed ram. I can verify that with 8G installed and the two values add up to more than 7G you get no choice but a hard reset. Since upgrading to 16G I have been more vigilant and not allowed more than 10G to be wired so haven't had that problem in a year and a half. With the default arc_max usually set to ram minus 1G and max_wired at 5G it is easy to see that the current defaults are dangerous. I have not seen max_wired mentioned in relation to zfs but it seems that it should be considered when setting arc_max to prevent over wiring ram. Close to three weeks ago I applied review D7538 to my everyday desktop running stable/11. Until a few days ago I had no swap usage which is now at 9M. In the last few years of monitoring wired usage to try and find a solution I have not seen less than 1G of swap usage after an hour of uptime. If nothing else D7538 makes arc more willing to be released. -- FreeBSD - the place to B...Storing Data Shane Ambler
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?be613c69-589a-93e8-0056-b1400d0116a8>