From owner-freebsd-virtualization@freebsd.org Sat Dec 2 05:23:39 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABCC2DF3BEF for ; Sat, 2 Dec 2017 05:23:39 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 86AEA21AF; Sat, 2 Dec 2017 05:23:39 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 08FB98444930; Fri, 1 Dec 2017 23:23:32 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KVwM3CeZqom9; Fri, 1 Dec 2017 23:23:31 -0600 (CST) Received: from [172.16.3.163] (unknown [67.148.93.60]) by internet06.ebureau.com (Postfix) with ESMTPSA id 04C0C844491A; Fri, 1 Dec 2017 23:23:30 -0600 (CST) Mime-Version: 1.0 (1.0) Subject: Re: bhyve uses all available memory during IO-intensive operations From: Dustin Wenz X-Mailer: iPhone Mail (15B202) In-Reply-To: Date: Fri, 1 Dec 2017 23:23:28 -0600 Cc: "freebsd-virtualization@freebsd.org" Message-Id: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> References: To: "K. Macy" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 05:23:39 -0000 I have noticed significant storage amplification for my zvols; that could ve= ry well be the reason. I would like to know more about why it happens.=20 Since the volblocksize is 512 bytes, I certainly expect extra cpu overhead (= and maybe an extra 1k or so worth of checksums for each 128k block in the vm= ), but how do you get a 10X expansion in stored data? What is the recommended zvol block size for a FreeBSD/ZFS guest? Perhaps 4k,= to match the most common mass storage sector size? - .Dustin > On Dec 1, 2017, at 9:18 PM, K. Macy wrote: >=20 > One thing to watch out for with chyves if your virtual disk is more > than 20G is the fact that it uses 512 byte blocks for the zvols it > creates. I ended up using up 1.4TB only half filling up a 250G zvol. > Chyves is quick and easy, but it's not exactly production ready. >=20 > -M >=20 >=20 >=20 >> On Thu, Nov 30, 2017 at 3:15 PM, Dustin Wenz wro= te: >> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is= also FreeBSD 11.1). Their sole purpose is to house some medium-sized Postgr= es databases (100-200GB). The host system has 64GB of real memory and 112GB o= f swap. I have configured each guest to only use 16GB of memory, yet while d= oing my initial database imports in the VMs, bhyve will quickly grow to use a= ll available system memory and then be killed by the kernel: >>=20 >> kernel: swap_pager: I/O error - pageout failed; blkno 1735,size 40= 96, error 12 >> kernel: swap_pager: I/O error - pageout failed; blkno 1610,size 40= 96, error 12 >> kernel: swap_pager: I/O error - pageout failed; blkno 1763,size 40= 96, error 12 >> kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >>=20 >> The OOM condition seems related to doing moderate IO within the VM, thoug= h nothing within the VM itself shows high memory usage. This is the chyves c= onfig for one of them: >>=20 >> bargs -A -H -P -S >> bhyve_disk_type virtio-blk >> bhyve_net_type virtio-net >> bhyveload_flags >> chyves_guest_version 0300 >> cpu 4 >> creation Created on Mon Oct 23 16:17:04 CDT 2017= by chyves v0.2.0 2016/09/11 using __create() >> loader bhyveload >> net_ifaces tap51 >> os default >> ram 16G >> rcboot 0 >> revert_to_snapshot >> revert_to_snapshot_method off >> serial nmdm51 >> template no >> uuid 8495a130-b837-11e7-b092-0025909a8b56 >>=20 >>=20 >> I've also tried using different bhyve_disk_types, with no improvement. Ho= w is it that bhyve can use far more memory that I'm specifying? >>=20 >> - .Dustin