From owner-freebsd-questions@freebsd.org Sun Jun 17 22:19:05 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64422100CE41; Sun, 17 Jun 2018 22:19:05 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0DE6F8288F; Sun, 17 Jun 2018 22:19:04 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 7AC4E21AED; Sun, 17 Jun 2018 18:19:04 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Sun, 17 Jun 2018 18:19:04 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h=cc :content-transfer-encoding:content-type:date:from:message-id :mime-version:subject:to:x-me-sender:x-me-sender:x-sasl-enc; s= fm2; bh=V0+kaAtZ23Q6DOUuAawVEveOc2TCUacM11mHsFKjE/c=; b=k0L1JHLe Z7ezm3jpSYtmbZ2bI4kcoJT6WCLRda40/0F7s3Dj+XnlFPPqtoINrTc4ObGfxprF IX5eCFqWGBuJ0oLPutScg4XXN459gaLlO31Qi3faO6RDw8QP+7g+X+qAyEuTLCSA G39Ss2IYsAnV/tjSpRgP98GeIcqHM7ifDFuvQmQk8G052vkNfN/juFlK4vpZPwh5 wo+PEoqVGt8RiN8FpuEk4J7X3rNhxuuhTX/mLSsqVFWKMv7QhXnwXKj8pGJlH753 De+M8FrGjsySAhGdwe8LtrtnsdsJIStyp8sqqxTN094kw2Xc9MpyW6mU7N57DE2w x2/86pb7FqmkjA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:message-id:mime-version:subject:to:x-me-sender :x-me-sender:x-sasl-enc; s=fm3; bh=V0+kaAtZ23Q6DOUuAawVEveOc2TCU acM11mHsFKjE/c=; b=cL6+xZh2aOiYaM/vI812jJDQqXXzVYFPONLq2TNAQseQg h0nUHMkosT/KHL42ZJPUv7OfYeq11rcxDRO9WGJaPPUlD/24OzA0elMVCGBHW0uo 4Ij7sxkBOlveVKApGq1ZRHAbYwBNYDQFEcsKBdkepf+AC5AOJDKUzJ73kQSvqw5Z eW+/XpJVm/wA4JaMfzXncnXJp8cQsDWhyqpJHFL3HM3EKM/Sx12hebFvImQc6/WN 8xJS4TNk++QTce8KPgh++fCTZynBpHliA06E1Avq1nVKuJ6pu5/NRKDoIpHoh8+c DCGBe3sE2RcwAqev2oE4ip5O8P5PhaRVX6v7ACGww== X-ME-Proxy: X-ME-Sender: Received: from desktop.local (parsley.growveg.org [82.70.91.97]) by mail.messagingengine.com (Postfix) with ESMTPA id B0034E4115; Sun, 17 Jun 2018 18:19:03 -0400 (EDT) To: freebsd-stable@freebsd.org From: tech-lists Subject: lightly loaded system eats swap space Organization: none Cc: freebsd-questions@freebsd.org Message-ID: Date: Sun, 17 Jun 2018 23:19:02 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jun 2018 22:19:05 -0000 Hello list, context is (server) freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, E5-2630 @2.3GHz, generic kernel. There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also freebsd-11-stable) There have been no special options for zfs configuration on the server, apart from several datasets having the compressed property set (lz4). The server runs nothing else really apart from sshd and it uses ntpd to sync local time. How come such a lightly loaded server with plenty of resources is eating up swap? If I run two bhyve instances, i.e. two of the same size as indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace errors in the daily logs: +swap_pager_getswapspace(24): failed +swap_pager_getswapspace(24): failed +swap_pager_getswapspace(24): failed Here's top, with one bhyve instance running: last pid: 49494; load averages: 0.12, 0.13, 0.88 up 29+11:36:06 22:52:45 54 processes: 1 running, 53 sleeping CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, 4123M Free ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other 103G Compressed, 104G Uncompressed, 1.00:1 Ratio Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh 32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve 49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp From the looks of it, a huge amount of ram is wired. Why is that, and how would I debug it? A server of similar spec which is running freebsd-current with seven bhyve instances doesn't have this issue: last pid: 41904; load averages: 0.26, 0.19, 0.15 up 17+01:06:11 23:14:13 27 processes: 1 running, 26 sleeping CPU: 0.1% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.6% idle Mem: 17G Active, 6951M Inact, 41G Laundry, 59G Wired, 1573M Buf, 1315M Free ARC: 53G Total, 700M MFU, 52G MRU, 512K Anon, 182M Header, 958K Other 53G Compressed, 69G Uncompressed, 1.30:1 Ratio, 122M Overhead Swap: 35G Total, 2163M Used, 33G Free, 6% Inuse thanks, -- J.