From owner-freebsd-virtualization@freebsd.org Thu Nov 30 23:28:42 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB91BDECA84 for ; Thu, 30 Nov 2017 23:28:42 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8D5327B325 for ; Thu, 30 Nov 2017 23:28:42 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from Ticonderoga.HML3.ScaleEngine.net (senat1-01.HML3.ScaleEngine.net [209.51.186.5]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id D93151440A for ; Thu, 30 Nov 2017 23:28:40 +0000 (UTC) Subject: Re: bhyve uses all available memory during IO-intensive operations To: freebsd-virtualization@freebsd.org References: From: Allan Jude Message-ID: Date: Thu, 30 Nov 2017 18:28:39 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Nov 2017 23:28:42 -0000 On 11/30/2017 18:15, Dustin Wenz wrote: > I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is also FreeBSD 11.1). Their sole purpose is to house some medium-sized Postgres databases (100-200GB). The host system has 64GB of real memory and 112GB of swap. I have configured each guest to only use 16GB of memory, yet while doing my initial database imports in the VMs, bhyve will quickly grow to use all available system memory and then be killed by the kernel: > > kernel: swap_pager: I/O error - pageout failed; blkno 1735,size 4096, error 12 > kernel: swap_pager: I/O error - pageout failed; blkno 1610,size 4096, error 12 > kernel: swap_pager: I/O error - pageout failed; blkno 1763,size 4096, error 12 > kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space > > The OOM condition seems related to doing moderate IO within the VM, though nothing within the VM itself shows high memory usage. This is the chyves config for one of them: > > bargs -A -H -P -S > bhyve_disk_type virtio-blk > bhyve_net_type virtio-net > bhyveload_flags > chyves_guest_version 0300 > cpu 4 > creation Created on Mon Oct 23 16:17:04 CDT 2017 by chyves v0.2.0 2016/09/11 using __create() > loader bhyveload > net_ifaces tap51 > os default > ram 16G > rcboot 0 > revert_to_snapshot > revert_to_snapshot_method off > serial nmdm51 > template no > uuid 8495a130-b837-11e7-b092-0025909a8b56 > > > I've also tried using different bhyve_disk_types, with no improvement. How is it that bhyve can use far more memory that I'm specifying? > > - .Dustin > Can you show 'top' output. What makes you think bhyve is using the memory? Are you using ZFS? Have you limited the vfs.zfs.arc_max to leave some free RAM for the bhyve instances? -- Allan Jude