From owner-freebsd-virtualization@freebsd.org Fri Dec 1 03:13:52 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAC91DF21E1 for ; Fri, 1 Dec 2017 03:13:52 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 98C6417C7 for ; Fri, 1 Dec 2017 03:13:52 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id DA15B14817; Fri, 1 Dec 2017 03:13:44 +0000 (UTC) Subject: Re: bhyve uses all available memory during IO-intensive operations To: Dustin Wenz Cc: freebsd-virtualization@freebsd.org References: From: Allan Jude Message-ID: Date: Thu, 30 Nov 2017 22:13:44 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 03:13:52 -0000 On 2017-11-30 22:10, Dustin Wenz wrote: > I am using a zvol as the storage for the VM, and I do not have any ARC > limits set. However, the bhyve process itself ends up grabbing the vast > majority of memory.  > > I’ll run a test tomorrow to get the exact output from top. > >    - .Dustin > > On Nov 30, 2017, at 5:28 PM, Allan Jude > wrote: > >> On 11/30/2017 18:15, Dustin Wenz wrote: >>> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest >>> OS is also FreeBSD 11.1). Their sole purpose is to house some >>> medium-sized Postgres databases (100-200GB). The host system has 64GB >>> of real memory and 112GB of swap. I have configured each guest to >>> only use 16GB of memory, yet while doing my initial database imports >>> in the VMs, bhyve will quickly grow to use all available system >>> memory and then be killed by the kernel: >>> >>>    kernel: swap_pager: I/O error - pageout failed; blkno 1735,size >>> 4096, error 12 >>>    kernel: swap_pager: I/O error - pageout failed; blkno 1610,size >>> 4096, error 12 >>>    kernel: swap_pager: I/O error - pageout failed; blkno 1763,size >>> 4096, error 12 >>>    kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >>> >>> The OOM condition seems related to doing moderate IO within the VM, >>> though nothing within the VM itself shows high memory usage. This is >>> the chyves config for one of them: >>> >>>    bargs                      -A -H -P -S >>>    bhyve_disk_type            virtio-blk >>>    bhyve_net_type             virtio-net >>>    bhyveload_flags >>>    chyves_guest_version       0300 >>>    cpu                        4 >>>    creation                   Created on Mon Oct 23 16:17:04 CDT 2017 >>> by chyves v0.2.0 2016/09/11 using __create() >>>    loader                     bhyveload >>>    net_ifaces                 tap51 >>>    os                         default >>>    ram                        16G >>>    rcboot                     0 >>>    revert_to_snapshot >>>    revert_to_snapshot_method  off >>>    serial                     nmdm51 >>>    template                   no >>>    uuid                       8495a130-b837-11e7-b092-0025909a8b56 >>> >>> >>> I've also tried using different bhyve_disk_types, with no >>> improvement. How is it that bhyve can use far more memory that I'm >>> specifying? >>> >>>    - .Dustin >>> >> >> Can you show 'top' output. What makes you think bhyve is using the >> memory? Are you using ZFS? Have you limited the vfs.zfs.arc_max to leave >> some free RAM for the bhyve instances? >> >> -- >> Allan Jude >> _______________________________________________ >> freebsd-virtualization@freebsd.org >> mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to >> "freebsd-virtualization-unsubscribe@freebsd.org >> " The default limit for the ARC is 99% of your ram, so you are definitely going to want to reduce that to something like: 90% of your 64GB of ram, less the total amount of RAM you have given to all VMs. -- Allan Jude