From owner-freebsd-stable@freebsd.org Mon Aug 13 22:00:44 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E0C6F107EFEE for ; Mon, 13 Aug 2018 22:00:43 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 698138F46F for ; Mon, 13 Aug 2018 22:00:43 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: by mailman.ysv.freebsd.org (Postfix) id 2DF06107EFEB; Mon, 13 Aug 2018 22:00:43 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C011107EFEA for ; Mon, 13 Aug 2018 22:00:43 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 74F0C8F46B for ; Mon, 13 Aug 2018 22:00:42 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from amavis-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 41q8ml3Dk6z1B6; Tue, 14 Aug 2018 00:00:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla4; t=1534197618; x=1536789619; bh=jez 6acpdvdcvm6xkZpt0j0ZHX4hsW5AEoJuX9e/0Yeg=; b=IGs5QFJ9sWlx3heEOcX rCbu5FsR3Po6BEYDWZG1iD0ZVmruaocHF2zXeHoh/3RIeSaqPIElQu+DKqLKwveE jw/IjIk/PsldGeIF/tQ5wejJ1lDIPn+6pB/oGQ5Fhf8xGUfchfzXx8dImuEbjgd0 7Ae64EhI9QfqvoIf7PRNkl7I= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10026) with LMTP id Q1iHiSnKnQjQ; Tue, 14 Aug 2018 00:00:18 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP id 41q8mL1zwbz19j; Tue, 14 Aug 2018 00:00:18 +0200 (CEST) Received: from nabiralnik.ijs.si (nabiralnik.ijs.si [IPv6:2001:1470:ff80::80:16]) by mildred.ijs.si (Postfix) with ESMTP id 41q8mL0xG4zsl; Tue, 14 Aug 2018 00:00:18 +0200 (CEST) Received: from sleepy.ijs.si (2001:1470:ff80:e001::76) by webmail.ijs.si with HTTP (HTTP/1.1 POST); Tue, 14 Aug 2018 00:00:18 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Tue, 14 Aug 2018 00:00:18 +0200 From: Mark Martinec To: stable@freebsd.org Cc: Volodymyr Kostyrko Subject: Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64 Organization: Jozef Stefan Institute In-Reply-To: <8af780b9-1d8c-8279-dacd-eb1c8d199dec@b1t.name> References: <1a039af7758679ba1085934b4fb81b57@ijs.si> <8af780b9-1d8c-8279-dacd-eb1c8d199dec@b1t.name> Message-ID: <6cd192462e9c41d106cabb16bb61e0fe@ijs.si> X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.3.1 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 13 Aug 2018 22:00:44 -0000 2018-08-13 21:48, Volodymyr Kostyrko wrote: > I've been in the same situation. ZFS, only pool, no ZFS errors. > > I think the problem is rather between swapping and ZFS ARC. This host > has different load, sometimes it needs more active memory, somtimes > less... This means that active zone can expand and shrink like +-2G os > mem (I have 16Gb installed there). The problem is, when huge task is > idle it doesn't use much active memory and other activity is pushing > it's memory to the swap. When active runs low and ARC runs >50% of > memory it becomes very hard to make ARC give some memory back. My host > even was broght to the point when it couldn't get tasks back into > memory from swap, because while some pages were restored from swap the > time passes by and the other pages are instead stored to swap due to > zome ARC activity. Finally active zone shrinks so bad that the host > becomes unresponsive. > > Like 6 month ago I tried tweaking kernel and swap to make things go > other way. Currently I have `vm.swap_idle_enabled=1` in > /etc/loader.conf and looks like this solves my problem. The other > interesting things to look at are `vfs.zfs.arc_free_target`, > `vfs.zfs.arc_shrink_shift`, `vfs.zfs.arc_grow_retry`. > > Or you can take another route and plain limit current ARC size with > `vfs.zfs.arc_max`. What you describe is not the same problem as the one I described in this thread. In my case the ZFS malloc'ed memory ("solaris" zone) is growing, while the size of the ARC remains capped to a reasonably low value, and the ARC even shrinks as the "solaris" zone approaches the memory size. I too have been bitten previously by the ARC size being reluctant to shrink. Ths problem is described here, but only partially mitigated now in the 11.? version: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=187594 The usually suggested workaround is to limit the size of the ARC, although it would be nice to find a solution to handle ARC UMA shrinking automatically, like it worked well in FreeBSD 9 but broke in FreeBSD 10. Like I said, the problem I described in this thread is different. Mark