Date: Thu, 30 Nov 2017 22:13:44 -0500 From: Allan Jude <allanjude@freebsd.org> To: Dustin Wenz <dustinwenz@ebureau.com> Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve uses all available memory during IO-intensive operations Message-ID: <a0ecb7b5-feeb-fdfb-ae4e-f01c2973ad1d@freebsd.org> In-Reply-To: <F633EDFD-E9DA-47A3-AEFD-ED9C499FBD29@ebureau.com> References: <F4E35CB9-30F9-4C63-B4CC-F8ADC9947E3C@ebureau.com> <a324b830-b941-2767-2272-957da680d997@freebsd.org> <F633EDFD-E9DA-47A3-AEFD-ED9C499FBD29@ebureau.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 2017-11-30 22:10, Dustin Wenz wrote: > I am using a zvol as the storage for the VM, and I do not have any ARC > limits set. However, the bhyve process itself ends up grabbing the vast > majority of memory. > > I’ll run a test tomorrow to get the exact output from top. > > - .Dustin > > On Nov 30, 2017, at 5:28 PM, Allan Jude <allanjude@freebsd.org > <mailto:allanjude@freebsd.org>> wrote: > >> On 11/30/2017 18:15, Dustin Wenz wrote: >>> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest >>> OS is also FreeBSD 11.1). Their sole purpose is to house some >>> medium-sized Postgres databases (100-200GB). The host system has 64GB >>> of real memory and 112GB of swap. I have configured each guest to >>> only use 16GB of memory, yet while doing my initial database imports >>> in the VMs, bhyve will quickly grow to use all available system >>> memory and then be killed by the kernel: >>> >>> kernel: swap_pager: I/O error - pageout failed; blkno 1735,size >>> 4096, error 12 >>> kernel: swap_pager: I/O error - pageout failed; blkno 1610,size >>> 4096, error 12 >>> kernel: swap_pager: I/O error - pageout failed; blkno 1763,size >>> 4096, error 12 >>> kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >>> >>> The OOM condition seems related to doing moderate IO within the VM, >>> though nothing within the VM itself shows high memory usage. This is >>> the chyves config for one of them: >>> >>> bargs -A -H -P -S >>> bhyve_disk_type virtio-blk >>> bhyve_net_type virtio-net >>> bhyveload_flags >>> chyves_guest_version 0300 >>> cpu 4 >>> creation Created on Mon Oct 23 16:17:04 CDT 2017 >>> by chyves v0.2.0 2016/09/11 using __create() >>> loader bhyveload >>> net_ifaces tap51 >>> os default >>> ram 16G >>> rcboot 0 >>> revert_to_snapshot >>> revert_to_snapshot_method off >>> serial nmdm51 >>> template no >>> uuid 8495a130-b837-11e7-b092-0025909a8b56 >>> >>> >>> I've also tried using different bhyve_disk_types, with no >>> improvement. How is it that bhyve can use far more memory that I'm >>> specifying? >>> >>> - .Dustin >>> >> >> Can you show 'top' output. What makes you think bhyve is using the >> memory? Are you using ZFS? Have you limited the vfs.zfs.arc_max to leave >> some free RAM for the bhyve instances? >> >> -- >> Allan Jude >> _______________________________________________ >> freebsd-virtualization@freebsd.org >> <mailto:freebsd-virtualization@freebsd.org> mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to >> "freebsd-virtualization-unsubscribe@freebsd.org >> <mailto:freebsd-virtualization-unsubscribe@freebsd.org>" The default limit for the ARC is 99% of your ram, so you are definitely going to want to reduce that to something like: 90% of your 64GB of ram, less the total amount of RAM you have given to all VMs. -- Allan Jude
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?a0ecb7b5-feeb-fdfb-ae4e-f01c2973ad1d>