From owner-freebsd-stable@freebsd.org Thu Jun 7 09:35:01 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68D14FDD80B for ; Thu, 7 Jun 2018 09:35:01 +0000 (UTC) (envelope-from Shane@shaneware.biz) Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by mx1.freebsd.org (Postfix) with ESMTP id 04F1F87800 for ; Thu, 7 Jun 2018 09:34:39 +0000 (UTC) (envelope-from Shane@shaneware.biz) Received: from 124-169-213-169.dyn.iinet.net.au (HELO leader.local) ([124.169.213.169]) by ipmail07.adl2.internode.on.net with ESMTP; 07 Jun 2018 19:04:31 +0930 Subject: Re: ZFS+find(1) wiring all RAM To: Peter Jeremy , freebsd-stable@freebsd.org References: <20180607063924.GG1697@server.rulingia.com> From: Shane Ambler Message-ID: <1da884f4-1442-9bba-02d3-2cedadf97a06@ShaneWare.Biz> Date: Thu, 7 Jun 2018 19:04:29 +0930 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <20180607063924.GG1697@server.rulingia.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-AU Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jun 2018 09:35:01 -0000 On 07/06/2018 16:09, Peter Jeremy wrote: > I've noticed that 11-stable/amd64 has been wiring seemingly excessive > amounts of RAM for some time (the problem goes back at least 6 months). > This extends to getting ENOMEM errors from g_io_deliver() and out-of-swap > errors killing processes on a low-memory system. I'm not sure when it > started by it seems to hawe gotten worse between r331535 and r334494. Don't know if this will help you at all -- I have seen excess wired for a few years, since 10.1, I now run 11-stable, my experience has seen the severity varying over time. I first reported this 28/10/2014 related to heavy disk use on a zpool. https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194654 The inability to get solid repeatable steps to reproduce have prevented me from chasing this more. My desktop machine originally had 8G and when wired went over 7G there was no choice but to reset. I now have 16G and have a script running constantly to monitor the wired and react when needed, allocating a big chunk of ram normally causes wired to be released, this has kept wired under 10G for several months. I see arc usage sitting at the arc_max setting with wired going more than two times that. My vm.max_wired is at 5G (default) and I think this is a limit that is somehow bypassed, if it could be enforced we might be good. This could be the point to chase in a bug report. > 1) Significant amounts of wired memory are in use but I can't find anything > in "vmstat -mz" that would explain where it's going. I started logging my wired usage and have about 6 months of data, in the following sample, allocating 2G more than free caused wired to drop 4G in about 2 secs, you can also see that arc usage did not change in this time. This leads me to think it may be related to changes made to add support for zfs but I don't think zfs is the one allocating this wired amount. The values I am logging are from - vm.max_wired vm.stats.vm.v_wire_count kstat.zfs.misc.arcstats.size vm.stats.vm.v_free_count 2018/06/06_13:22:53 wired: 9779M max_wired: 5210M ARC: 4.1G Free:3.1G 2018/06/06_13:22:53 wired: 9781M max_wired: 5210M ARC: 4.1G Free:3.0G 2018/06/06_13:22:54 Excess wired detected. Allocating 4.9G to reduce wired. 2018/06/06_13:22:54 wired: 9787M max_wired: 5210M ARC: 4.1G Free:3.0G 2018/06/06_13:22:54 wired: 9796M max_wired: 5210M ARC: 4.1G Free:1.9G 2018/06/06_13:22:55 wired: 9809M max_wired: 5210M ARC: 4.1G Free:389.6M 2018/06/06_13:22:55 wired: 8169M max_wired: 5210M ARC: 4.1G Free:1.2G 2018/06/06_13:22:55 wired: 5917M max_wired: 5210M ARC: 4.1G Free:2.6G 2018/06/06_13:22:55 wired: 5566M max_wired: 5210M ARC: 4.1G Free:2.2G -- Shane Ambler Shane (at) ShaneWare (dot) Biz http://ShaneWare.Biz