From owner-freebsd-stable@freebsd.org Tue Jun 19 17:29:39 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D10E7100228C for ; Tue, 19 Jun 2018 17:29:39 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from resqmta-po-09v.sys.comcast.net (resqmta-po-09v.sys.comcast.net [IPv6:2001:558:fe16:19:96:114:154:168]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "resqmta-po-01v.sys.comcast.net", Issuer "COMODO RSA Organization Validation Secure Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 717EB6C499 for ; Tue, 19 Jun 2018 17:29:39 +0000 (UTC) (envelope-from jdc@koitsu.org) Received: from resomta-po-01v.sys.comcast.net ([96.114.154.225]) by resqmta-po-09v.sys.comcast.net with ESMTP id VKKSfoxj3EEmzVKRlf1x8L; Tue, 19 Jun 2018 17:29:37 +0000 Received: from koitsu.org ([71.198.44.84]) by resomta-po-01v.sys.comcast.net with ESMTPA id VKRkf7i1WVGSOVKRlf3Klk; Tue, 19 Jun 2018 17:29:37 +0000 Received: by icarus.home.lan (Postfix, from userid 1000) id 4193F1581FA9; Tue, 19 Jun 2018 10:29:36 -0700 (PDT) Date: Tue, 19 Jun 2018 10:29:36 -0700 From: Jeremy Chadwick To: freebsd-stable@freebsd.org Subject: Re: lightly loaded system eats swap space Message-ID: <20180619172936.GA24967@icarus.home.lan> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.10.0 (2018-05-17) X-CMAE-Envelope: MS4wfB6vY8/+jSMm74JIDhszizUnlu5cxTDgAnYTCODC6YRbLLqJ6A0JAqgBxQ42Ic8cqDsz20v4Wnv0CAbPZfE1d1YkEg6AFfE4A3a8iQ9uWyCrFs83cOhM tl7x7qyrKahEemszm3gcc1inCiFffr46kJXGDXVg55fRy5CAukNPVJp6 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jun 2018 17:29:40 -0000 (I am not subscribed to -stable, so please CC me, though I doubt I can help in any way/shape/form past this Email) Not the first time this has come up -- and every time it has, all that's heard is crickets in the threads. Recent proof: https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088727.html https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088728.html https://lists.freebsd.org/pipermail/freebsd-stable/2018-June/089094.html I sent private mail to Peter Jeremy about his issue. I will not disclose that Email here. However, I will disclose the commits I included in said Email that have touched ZFS ARC-related code: http://www.freshbsd.org/commit/freebsd/r332785 http://www.freshbsd.org/commit/freebsd/r332552 http://www.freshbsd.org/commit/freebsd/r332540 (may help give insights) http://www.freshbsd.org/commit/freebsd/r330061 http://www.freshbsd.org/commit/freebsd/r328235 http://www.freshbsd.org/commit/freebsd/r327491 http://www.freshbsd.org/commit/freebsd/r326619 http://www.freshbsd.org/commit/freebsd/r326427 (quota-related, maybe irrelevant) http://www.freshbsd.org/commit/freebsd/r323667 In short (and nebulous as hell; sorry, I cannot be more specific given the nature of the problem): there have been changes about ZFS's memory allocation/releasing decision-making scheme compared to ZFS on "older" FreeBSD (i.e. earlier 11.x, and definitely 10.x and 9.x). Recommendations like "limit your ARC" are nothing new in FreeBSD, but are still ridiculous kludges: tech-lists' system clearly has 105GB MRU (MRU = most recently used) in ARC, meaning there is memory that can be released back to the rest of the OS for general use (re: memory contention/pressure situation), but the OS is choosing to use swap instead, eventually exhausting it. That logic sounds broken, IMO. (And yes I did notice the size of bhyve process) ZFS-related kernel folks need to be involved in this conversation. For whatever reason, in the past several years, related committers are no longer participating in these type of discussions. The opposite was true back in the 7.x to 9.x days. The answers have to come from them. I don't know, today, a) how they prefer these problems get reported to them, or b) what exact information they want that can help narrow it down (tech-lists' provided data is, IMO, good and par for the course). -- | Jeremy Chadwick jdc@koitsu.org | | UNIX Systems Administrator http://jdc.koitsu.org/ | | Making life hard for others since 1977. PGP 4BD6C0CB |