From owner-freebsd-stable@freebsd.org Mon Jun 18 12:23:22 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F29710037E6 for ; Mon, 18 Jun 2018 12:23:22 +0000 (UTC) (envelope-from dave+freebsd@fullard.ca) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2D7C57C333 for ; Mon, 18 Jun 2018 12:23:21 +0000 (UTC) (envelope-from dave+freebsd@fullard.ca) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 9297E20A9B for ; Mon, 18 Jun 2018 08:23:21 -0400 (EDT) Received: from web4 ([10.202.2.214]) by compute4.internal (MEProxy); Mon, 18 Jun 2018 08:23:21 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fullard.ca; h= content-transfer-encoding:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm3; bh=zFq0sBOw9LxNNrt4PQhLsgjBuzXPZ Qsy87OuWG7t66E=; b=WAL/XU5Xws3McNjq9zqMOWKPyOimYgvqKG33+cZXEp9oU T16094Pf+wgFxn2f4va9FhnAZOOhN8TLlrr5TPd9CqdRYyg9PoGo4PKFM9GprdAL kZUT6QA3z+xou1PFqCDjKhOd6Z5KoYmvyBs4o/5LYXTzc4r3Pbh3TD7BinAjGTTe I3L6EJJVCFz9v+5BRdsujjWM2JIjEmbHEWf23Wral15yPZPgLmJiWJhan2eF4TFF 9Bq9HqhVxYsLJaUISWNQpc47Mu2+933sUgqR5ehzBWSXkg+3dqODmu6IZd4RS/vy WC3N5aGd+J+P8w1pGF5zGQVe55f2qeci7iYCJW0Mw== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=zFq0sB Ow9LxNNrt4PQhLsgjBuzXPZQsy87OuWG7t66E=; b=HwAHSirhu+5NzteFZEwRay xW7+cQy+t6tpKs35vi3tI8LTyeuuj3dL665tsjXEja73AkLlbrMXoky+lJIbC65f Mf4XVwwVfs+/sj1EWObiqO9UdzBRZAvHWnADieOVGBEFACQ7hnjlDiT5pJy/biqi NdhQqR5Ci+SOnRemEHejxAOsuVzCYixk5usjWK72wn7zlctKYtrYwDa1jo7dxvZ7 2y7i3ORYy/3YMc8cc0BXmpaancu/h/UXtLqQtlbqWTgpD8omuffLjXp85j3cK7nt O4nwjqp+d3QFXAetT7SKkq9IP306xI1m3IFl80IubFcNx5OBPRCRmkPoFEjJ4jVQ == X-ME-Proxy: X-ME-Sender: Received: by mailuser.nyi.internal (Postfix, from userid 99) id 5C875BA43D; Mon, 18 Jun 2018 08:23:21 -0400 (EDT) Message-Id: <1529324601.427299.1411679552.7868D498@webmail.messagingengine.com> From: David Fullard To: freebsd-stable@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="utf-8" X-Mailer: MessagingEngine.com Webmail Interface - ajax-498d70f2 References: Date: Mon, 18 Jun 2018 08:23:21 -0400 In-Reply-To: Subject: Re: lightly loaded system eats swap space X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jun 2018 12:23:22 -0000 I've noticed you've got a rather large ZFS ARC. You could try limiting the ZFS max ARC size by setting the vfs.zfs.arc_max sysctl. On Sun, Jun 17, 2018, at 6:19 PM, tech-lists wrote: > Hello list, > > context is (server) > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > E5-2630 @2.3GHz, generic kernel. > > There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also > freebsd-11-stable) > > There have been no special options for zfs configuration on the server, > apart from several datasets having the compressed property set (lz4). > > The server runs nothing else really apart from sshd and it uses ntpd to > sync local time. > > How come such a lightly loaded server with plenty of resources is eating > up swap? If I run two bhyve instances, i.e. two of the same size as > indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace > errors in the daily logs: > > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > > Here's top, with one bhyve instance running: > > last pid: 49494; load averages: 0.12, 0.13, 0.88 > > > up 29+11:36:06 22:52:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle > Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, > 4123M Free > ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > 103G Compressed, 104G Uncompressed, 1.00:1 Ratio > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh > 32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve > 49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp > > From the looks of it, a huge amount of ram is wired. Why is that, and > how would I debug it? > > A server of similar spec which is running freebsd-current with seven > bhyve instances doesn't have this issue: > > last pid: 41904; load averages: 0.26, 0.19, 0.15 > > > up 17+01:06:11 23:14:13 > 27 processes: 1 running, 26 sleeping > CPU: 0.1% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.6% idle > Mem: 17G Active, 6951M Inact, 41G Laundry, 59G Wired, 1573M Buf, 1315M Free > ARC: 53G Total, 700M MFU, 52G MRU, 512K Anon, 182M Header, 958K Other > 53G Compressed, 69G Uncompressed, 1.30:1 Ratio, 122M Overhead > Swap: 35G Total, 2163M Used, 33G Free, 6% Inuse > > thanks, > -- > J. > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"