From owner-freebsd-stable@freebsd.org Thu Nov 14 15:25:17 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 21A961AAE8A for ; Thu, 14 Nov 2019 15:25:17 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 47DQK75zV6z3NTM for ; Thu, 14 Nov 2019 15:25:15 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id xAEFP72I005392 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Thu, 14 Nov 2019 15:25:07 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id xAEFP4nV025379 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Thu, 14 Nov 2019 22:25:04 +0700 (+07) (envelope-from eugen@grosbein.net) To: FreeBSD stable From: Eugene Grosbein Subject: bhyve memory leak in stable/11 Message-ID: <7fddcea5-2188-afe1-3ea9-a53dffdbec32@grosbein.net> Date: Thu, 14 Nov 2019 22:24:59 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: 47DQK75zV6z3NTM X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.71 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[grosbein.net]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_THREE(0.00)[3]; IP_SCORE(-1.61)[ip: (-4.08), ipnet: 2a01:4f8::/29(-2.29), asn: 24940(-1.65), country: DE(-0.01)]; TO_DN_ALL(0.00)[]; R_SPF_PERMFAIL(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Nov 2019 15:25:17 -0000 Hi! After 1 day and 11 hours of host's uptime I observe significant leak of bhyve instances RSS: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 66668 root 34 20 0 60320M 58866M kqread 21 62.0H 692.35% bhyve: sappdev (bhyve) 5546 root 26 20 0 22573M 22088M kqread 12 310:39 6.41% bhyve: srdapp01 (bhyve) 3842 root 20 20 0 15946M 15862M kqread 13 340:36 14.03% bhyve: sdc01 (bhyve) 4729 root 20 20 0 13792M 13455M kqread 8 79:40 1.02% bhyve: skms01 (bhyve) 4283 root 28 20 0 12721M 12682M kqread 29 92:23 0.80% bhyve: sfile01 (bhyve) 8057 root 34 20 0 8512M 8183M kqread 25 186:52 17.02% bhyve: ssql01 (bhyve) 23336 root 34 20 0 16418M 7716M kqread 14 114:23 17.62% bhyve: solap (bhyve) The package vm-bhyve-1.2.3 is used here to manage bhyve instances and sappdev.conf has memory=40G limit. srdapp01 is limited to 12G, sdc01 to 4G, skms01 to 2G, sfile01 to 2G, ssql01 to 8G and solap to 16G. Most of are using much more memory than they should. All of them run different versions of Windows servers. The host is 11.3-STABLE/amd64 r354667 with 360G RAM and guest systems use ZFS ZVOLs and vfs.zfs.arc_max=160g. How do I debug this?