From owner-freebsd-virtualization@freebsd.org Sat Dec 2 06:47:26 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 512D4DF5B59 for ; Sat, 2 Dec 2017 06:47:26 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-oi0-x22f.google.com (mail-oi0-x22f.google.com [IPv6:2607:f8b0:4003:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1986563A94 for ; Sat, 2 Dec 2017 06:47:26 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-oi0-x22f.google.com with SMTP id j17so8637628oih.3 for ; Fri, 01 Dec 2017 22:47:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=N4KRZpl1zRKfYTO04ldgATXSrE+XVPVH2jpjVAQarGg=; b=amdRoOacebgzeDXYSeFlfNnvMgI/ZyfYRuAGvIuXugXqs0+KV/F8nGLYBxzKlL4osa 99LvFomMItN8osbillKatWeKPmYvHf5wJ25Q6bn4fQQmkrZre1ym4nDJpXAHu0Xb15dL azj2gYQt0Z4u/BWhOpl9TdHMWyOt1T4cViqG/hQwyGxhQRNEMJZ+z4jz3RxD9WmEPX/M fss9Vc2I8HtiGr3YyPWfju+2nBGOjyYwYn0txl4vaJnsJqPiHFzwuQV/utzsArIRutkP 9lX1GKoil7lcire7aNGe8SLddT4TQzecAyQ8tADg/IeCZfhBoeJ43AuahXR7hLcsDyPs NuKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=N4KRZpl1zRKfYTO04ldgATXSrE+XVPVH2jpjVAQarGg=; b=bGj2CyGqjIcWR2EvUye6Knrzr0RlfIfQ/eUrx3WejUdsCToqPUSlH9Bi+UWzfbPK7L y1MnJZkQ5hVzmvRlp2a7NLR47oHNJIjqiEoaXYH1Sps5htWWTSghyPfTydBlWy7AnTVV FAgp2YmgH5bDDKalBOVcbDd3VcLPRQ0F2EGZmK4l4bCjMi16WGpjnXOZPb7Q0eRQF+7T DJYgd+JZbfpp/Cp+qF8f6TOHP5PngFZwg0FRgug5SZukP/FBKXNe33HpbTjm949VIxYt +af8BXJkiU4otG6ZBgN+ZEFjGFTSpCGb/g4qQsrNqXCjIfpHAPPUPvcqCuOOG/XqZqlD PH4Q== X-Gm-Message-State: AJaThX7daUHauMVzNZxRCSzzM5xOyNWzhX1WQssnkFcCdDoKVSY10DVj 6df6hga2wa56SKOnLhN0OmGRWIzOKWi4WeWaff9vbA== X-Google-Smtp-Source: AGs4zMbxm38YTK2u+CyUhIfOCdDCQsBJtl4Y7lmM0NRQNyLTlqoWwoA76cYlXSTsyoV8IBZj3kw4hn9S9qhCmihA7jE= X-Received: by 10.202.199.133 with SMTP id x127mr8857313oif.17.1512197245165; Fri, 01 Dec 2017 22:47:25 -0800 (PST) MIME-Version: 1.0 Sender: kmacybsd@gmail.com Received: by 10.157.66.238 with HTTP; Fri, 1 Dec 2017 22:47:24 -0800 (PST) In-Reply-To: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> References: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> From: "K. Macy" Date: Fri, 1 Dec 2017 22:47:24 -0800 X-Google-Sender-Auth: 2Rjh3-wG_jYcM6PfLLVEW2vXcQU Message-ID: Subject: Re: bhyve uses all available memory during IO-intensive operations To: Dustin Wenz Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 06:47:26 -0000 On Fri, Dec 1, 2017 at 9:23 PM, Dustin Wenz wrote: > I have noticed significant storage amplification for my zvols; that could > very well be the reason. I would like to know more about why it happens. > > Since the volblocksize is 512 bytes, I certainly expect extra cpu overhead > (and maybe an extra 1k or so worth of checksums for each 128k block in the > vm), but how do you get a 10X expansion in stored data? > > What is the recommended zvol block size for a FreeBSD/ZFS guest? Perhaps 4k, > to match the most common mass storage sector size? I would err somewhat larger, the benefits of shallower indirect block chains will outweigh the cost of RMW I would guess. And I think it should be your guest file system block size. I don't know what ext4 is, but ext2/3 was 16k by default IIRC. -M > > - .Dustin > > On Dec 1, 2017, at 9:18 PM, K. Macy wrote: > > One thing to watch out for with chyves if your virtual disk is more > than 20G is the fact that it uses 512 byte blocks for the zvols it > creates. I ended up using up 1.4TB only half filling up a 250G zvol. > Chyves is quick and easy, but it's not exactly production ready. > > -M > > > > On Thu, Nov 30, 2017 at 3:15 PM, Dustin Wenz wrote: > > I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is > also FreeBSD 11.1). Their sole purpose is to house some medium-sized > Postgres databases (100-200GB). The host system has 64GB of real memory and > 112GB of swap. I have configured each guest to only use 16GB of memory, yet > while doing my initial database imports in the VMs, bhyve will quickly grow > to use all available system memory and then be killed by the kernel: > > > kernel: swap_pager: I/O error - pageout failed; blkno 1735,size 4096, > error 12 > > kernel: swap_pager: I/O error - pageout failed; blkno 1610,size 4096, > error 12 > > kernel: swap_pager: I/O error - pageout failed; blkno 1763,size 4096, > error 12 > > kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space > > > The OOM condition seems related to doing moderate IO within the VM, though > nothing within the VM itself shows high memory usage. This is the chyves > config for one of them: > > > bargs -A -H -P -S > > bhyve_disk_type virtio-blk > > bhyve_net_type virtio-net > > bhyveload_flags > > chyves_guest_version 0300 > > cpu 4 > > creation Created on Mon Oct 23 16:17:04 CDT 2017 by > chyves v0.2.0 2016/09/11 using __create() > > loader bhyveload > > net_ifaces tap51 > > os default > > ram 16G > > rcboot 0 > > revert_to_snapshot > > revert_to_snapshot_method off > > serial nmdm51 > > template no > > uuid 8495a130-b837-11e7-b092-0025909a8b56 > > > > I've also tried using different bhyve_disk_types, with no improvement. How > is it that bhyve can use far more memory that I'm specifying? > > > - .Dustin