From owner-freebsd-stable@freebsd.org Tue Jun 19 17:58:01 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D46831004050 for ; Tue, 19 Jun 2018 17:58:00 +0000 (UTC) (envelope-from peixotocassiano@gmail.com) Received: from mail-qk0-x232.google.com (mail-qk0-x232.google.com [IPv6:2607:f8b0:400d:c09::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E2E96DBC0 for ; Tue, 19 Jun 2018 17:58:00 +0000 (UTC) (envelope-from peixotocassiano@gmail.com) Received: by mail-qk0-x232.google.com with SMTP id o2-v6so144349qkc.13 for ; Tue, 19 Jun 2018 10:58:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=qle7RfghUygJL+BPljsVEWWOfTb1AvTNttLxfhFj85k=; b=ir5NsLaJ4FFJKhueeNGXV0WPkG+05Asqfk6dj2swqOAduaiLP0L4FqW906W3+6Cub3 ij6Gma8VZBlXCv87gYb6cRcXGKjmBMEHtO+ES5GkWOI+fdnVZga1tp4nOSQQsNnLC7Rf ojpxI7OXWlwxs4F+NYzLA+9RW7x1R4O7tXsVcnTznsgGf5am/XSsSGs2plzYujB8YUaY MvczJiKdwWz58fv8gPtDUDnsDNrwsIsefbWBRZ/yfTMJCurUQYZhrPhvEfjSKagiX22l T375r8aK8TNwHwnAfIo+LUfHQw2Obkk5oeqv5eV+cZceghT5cLrRfTBifXquQGMfJiaR 9MMQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=qle7RfghUygJL+BPljsVEWWOfTb1AvTNttLxfhFj85k=; b=n1JYpwT556cfitxu1M08NPGGAc/EmO3YVrkMBjeGrHfgqdyS8YRfAx34MTn7d74Z64 I8DwA8KpfR3LDakBZarRJ3LkVjQ1u5N4I6Ya9oVr+GDNkH5DybEp4qC97qUyTpal26eh lp0HuzAE2xC/lPhNoq75/LZGqRyild+4ppQgGhBcNEs8pgmKqBDIFw9g/IPTUuQyEFLj QX3vCh/Q5vFZ/E2AGHixwTeEOI4sa41D+U1aH+r4a1DXGt3fhH6htm46+0IK9vASvY/a qobQ3w6EBbnPOxjoI89e97SXlYdI8lG5naVSB69DRe4WMiiGMH//y3e2Qc/Is1yY4tZa rs8Q== X-Gm-Message-State: APt69E2tdzlO9XwIVcX/4z33QLmRoPiltHg/kdLny3buoBXyf6FQSNTx iMPUmrA2E1ZEI413afo3KiK0TkpT/OwkR1wmLPSL7A== X-Google-Smtp-Source: ADUXVKJ6z2D+1E1hkTKrSpHXCxb6Aiq9l2OYBUBJ/jplb8WGWoWPNn0qh8/iIGSf6DIx6WOVLevRGXThrcmlPybOiU4= X-Received: by 2002:a37:88c2:: with SMTP id k185-v6mr14934771qkd.102.1529431080077; Tue, 19 Jun 2018 10:58:00 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:ac8:5399:0:0:0:0:0 with HTTP; Tue, 19 Jun 2018 10:57:59 -0700 (PDT) In-Reply-To: <20180619172936.GA24967@icarus.home.lan> References: <20180619172936.GA24967@icarus.home.lan> From: Cassiano Peixoto Date: Tue, 19 Jun 2018 14:57:59 -0300 Message-ID: Subject: Re: lightly loaded system eats swap space To: Jeremy Chadwick Cc: freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Jun 2018 17:58:01 -0000 Hi, I have the very same issue on many servers. Mainly on mail servers running qmail+spamassassin+clamav. I've tuned some variables on loader.conf: vfs.zfs.vdev.cache.size="2G" vfs.zfs.arc_min="614400000" vfs.zfs.arc_max="4915200000" But after some days, it begins eating swap and the system comes very very slow then I need to reboot it. My system config is: FreeBSD 11.1-STABLE #5 r321625M: Thu Sep 21 16:01:56 -03 2017 root@mail.com:/usr/obj/usr/src/sys/MAIL amd64 FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0) VT(vga): resolution 640x480 CPU: Intel(R) Atom(TM) CPU C2518 @ 1.74GHz (1750.04-MHz K8-class CPU) Origin="GenuineIntel" Id=0x406d8 Family=0x6 Model=0x4d Stepping=8 Features=0xbfebfbff Features2=0x43d8e3bf AMD Features=0x28100800 AMD Features2=0x101 Structured Extended Features=0x2282 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 8589934592 (8192 MB) avail memory = 8213245952 (7832 MB) It's configured with 4GB of swap. Let me know if I can help with any other information. Thanks. On Tue, Jun 19, 2018 at 2:29 PM, Jeremy Chadwick wrote: > (I am not subscribed to -stable, so please CC me, though I doubt I can > help in any way/shape/form past this Email) > > Not the first time this has come up -- and every time it has, all that's > heard is crickets in the threads. Recent proof: > > https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088727.html > https://lists.freebsd.org/pipermail/freebsd-stable/2018-April/088728.html > https://lists.freebsd.org/pipermail/freebsd-stable/2018-June/089094.html > > I sent private mail to Peter Jeremy about his issue. I will not > disclose that Email here. However, I will disclose the commits I > included in said Email that have touched ZFS ARC-related code: > > http://www.freshbsd.org/commit/freebsd/r332785 > http://www.freshbsd.org/commit/freebsd/r332552 > http://www.freshbsd.org/commit/freebsd/r332540 (may help give insights) > http://www.freshbsd.org/commit/freebsd/r330061 > http://www.freshbsd.org/commit/freebsd/r328235 > http://www.freshbsd.org/commit/freebsd/r327491 > http://www.freshbsd.org/commit/freebsd/r326619 > http://www.freshbsd.org/commit/freebsd/r326427 (quota-related, maybe > irrelevant) > http://www.freshbsd.org/commit/freebsd/r323667 > > In short (and nebulous as hell; sorry, I cannot be more specific given > the nature of the problem): there have been changes about ZFS's memory > allocation/releasing decision-making scheme compared to ZFS on "older" > FreeBSD (i.e. earlier 11.x, and definitely 10.x and 9.x). > > Recommendations like "limit your ARC" are nothing new in FreeBSD, but > are still ridiculous kludges: tech-lists' system clearly has 105GB MRU > (MRU = most recently used) in ARC, meaning there is memory that can be > released back to the rest of the OS for general use (re: memory > contention/pressure situation), but the OS is choosing to use swap > instead, eventually exhausting it. That logic sounds broken, IMO. (And > yes I did notice the size of bhyve process) > > ZFS-related kernel folks need to be involved in this conversation. For > whatever reason, in the past several years, related committers are no > longer participating in these type of discussions. The opposite was > true back in the 7.x to 9.x days. The answers have to come from them. > I don't know, today, a) how they prefer these problems get reported to > them, or b) what exact information they want that can help narrow it > down (tech-lists' provided data is, IMO, good and par for the course). > > -- > | Jeremy Chadwick jdc@koitsu.org | > | UNIX Systems Administrator http://jdc.koitsu.org/ | > | Making life hard for others since 1977. PGP 4BD6C0CB | > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >