From owner-freebsd-current@freebsd.org Tue Mar 6 10:14:55 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2D55DF46ED1 for ; Tue, 6 Mar 2018 10:14:55 +0000 (UTC) (envelope-from se@freebsd.org) Received: from mailout12.t-online.de (mailout12.t-online.de [194.25.134.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mailout00.t-online.de", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B87B96EAC4 for ; Tue, 6 Mar 2018 10:14:54 +0000 (UTC) (envelope-from se@freebsd.org) Received: from fwd15.aul.t-online.de (fwd15.aul.t-online.de [172.20.27.63]) by mailout12.t-online.de (Postfix) with SMTP id 809B541F9807 for ; Tue, 6 Mar 2018 11:04:51 +0100 (CET) Received: from Stefans-MBP-LAN.fritz.box (Ek4XYZZSYhlKOy1J5IjMfG0HceGCo75gAzMyATgbGJyENY5YCOhYqoSvtHaF95DZuW@[87.151.220.195]) by fwd15.t-online.de with (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384 encrypted) esmtp id 1et9Sh-3Ynzqy0; Tue, 6 Mar 2018 11:04:47 +0100 Subject: Re: Strange ARC/Swap/CPU on yesterday's -CURRENT To: freebsd-current@freebsd.org References: <20180305203918.jydrv7oelnh7sxp2@ler-imac.lerctr.org> From: Stefan Esser Message-ID: Date: Tue, 6 Mar 2018 11:04:46 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: <20180305203918.jydrv7oelnh7sxp2@ler-imac.lerctr.org> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit X-ID: Ek4XYZZSYhlKOy1J5IjMfG0HceGCo75gAzMyATgbGJyENY5YCOhYqoSvtHaF95DZuW X-TOI-MSGID: c712b518-916c-4003-a27d-538278dabd16 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Mar 2018 10:14:55 -0000 Am 05.03.18 um 21:39 schrieb Larry Rosenman: > Upgraded to: > > FreeBSD borg.lerctr.org 12.0-CURRENT FreeBSD 12.0-CURRENT #11 r330385: Sun Mar 4 12:48:52 CST 2018 root@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/VT-LER amd64 > +1200060 1200060 > > Yesterday, and I'm seeing really strange slowness, ARC use, and SWAP use and swapping. > > See http://www.lerctr.org/~ler/FreeBSD/Swapuse.png > > Ideas? I'm seeing the same, and currently work around this with a reasonably limited vfs.zfs.arc_max. Without such a limit I see (on a system with 24 GB RAM): CPU: 0.3% user, 0.0% nice, 0.9% system, 0.1% interrupt, 98.8% idle Mem: 14M Active, 1228K Inact, 32K Laundry, 23G Wired, 376M Free ARC: 19G Total, 3935M MFU, 14G MRU, 82M Anon, 223M Header, 876M Other 18G Compressed, 36G Uncompressed, 2.02:1 Ratio Swap: 24G Total, 888M Used, 23G Free, 3% Inuse, 8892K In, 5136K Out sysctl vfs.zfs.arc_max=15988656640 results in: Mem: 129M Active, 72M Inact, 36K Laundry, 18G Wired, 5149M Free ARC: 15G Total, 3997M MFU, 10G MRU, 40M Anon, 205M Header, 877M Other 13G Compressed, 28G Uncompressed, 2.08:1 Ratio Swap: 24G Total, 796M Used, 23G Free, 3% Inuse, 16K In The system was mostly idle at both times, just some Samba traffic and mail being checked by spamassassin. And I noticed it (this time) when the spamassassin processes were aborted due to a time limit. I think that this problem must have been introduced in the last few weeks, but cannot give a better estimate (do not reboot that often). But I had already applied the arc_max setting a week ago (and had not put it in sysctl.conf in the hope that the ARC growth was a temporary problem in the ZFS code, soon to be fixed ...). Regards, STefan