From owner-freebsd-current@freebsd.org Tue Mar 6 08:18:37 2018 Return-Path: Delivered-To: freebsd-current@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69E56F3E40D for ; Tue, 6 Mar 2018 08:18:37 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from smtp.fagskolen.gjovik.no (smtp.fagskolen.gjovik.no [IPv6:2001:700:1100:1:200:ff:fe00:b]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.fagskolen.gjovik.no", Issuer "Fagskolen i Gj??vik" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id D76A769A16 for ; Tue, 6 Mar 2018 08:18:36 +0000 (UTC) (envelope-from trond@fagskolen.gjovik.no) Received: from mail.fig.ol.no (localhost [127.0.0.1]) by mail.fig.ol.no (8.15.2/8.15.2) with ESMTPS id w268ITku058076 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Tue, 6 Mar 2018 09:18:29 +0100 (CET) (envelope-from trond@fagskolen.gjovik.no) Received: from localhost (trond@localhost) by mail.fig.ol.no (8.15.2/8.15.2/Submit) with ESMTP id w268ITfZ058072 for ; Tue, 6 Mar 2018 09:18:29 +0100 (CET) (envelope-from trond@fagskolen.gjovik.no) X-Authentication-Warning: mail.fig.ol.no: trond owned process doing -bs Date: Tue, 6 Mar 2018 09:18:29 +0100 (CET) From: =?ISO-8859-1?Q?Trond_Endrest=F8l?= Sender: Trond.Endrestol@fagskolen.gjovik.no To: FreeBSD current Subject: Re: Strange ARC/Swap/CPU on yesterday's -CURRENT In-Reply-To: <20180305203918.jydrv7oelnh7sxp2@ler-imac.lerctr.org> Message-ID: References: <20180305203918.jydrv7oelnh7sxp2@ler-imac.lerctr.org> User-Agent: Alpine 2.21 (BSF 202 2017-01-01) Organization: Fagskolen Innlandet OpenPGP: url=http://fig.ol.no/~trond/trond.key MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED autolearn=unavailable autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail.fig.ol.no X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Mar 2018 08:18:37 -0000 On Mon, 5 Mar 2018 14:39-0600, Larry Rosenman wrote: > Upgraded to: > > FreeBSD borg.lerctr.org 12.0-CURRENT FreeBSD 12.0-CURRENT #11 r330385: Sun Mar 4 12:48:52 CST 2018 root@borg.lerctr.org:/usr/obj/usr/src/amd64.amd64/sys/VT-LER amd64 > +1200060 1200060 > > Yesterday, and I'm seeing really strange slowness, ARC use, and SWAP use and swapping. > > See http://www.lerctr.org/~ler/FreeBSD/Swapuse.png I see these symptoms on stable/11. One of my servers has 32 GiB of RAM. After a reboot all is well. ARC starts to fill up, and I still have more than half of the memory available for user processes. After running the periodic jobs at night, the amount of wired memory goes sky high. /etc/periodic/weekly/310.locate is a particular nasty one. Limiting the ARC to, say, 16 GiB, has no effect of the high amount of wired memory. After a few more days, the kernel consumes virtually all memory, forcing processes in and out of the swap device. stable/10 never exhibited these symptoms, even with ZFS. I had hoped the kernel would manage its memory usage more wisely, but maybe it's time to set some hard limits on the kernel. Last year, I experienced deadlocks on stable/11 systems running ZFS with only 1 GiB of RAM. periodic(8) and clang jobs would never be rescheduled, they just sat there doing nothing halfway through their mission and with most of their pages on the swap device. I was lucky enough to be able to log in and reboot the damned servers. I installed 8 GiB of memory in each server and I never saw any deadlocks since. Maybe we should try and help by run (virtual) machines with low amounts of memory and high loads to weed out these bugs, if they still persist. -- Trond.