From owner-freebsd-virtualization@freebsd.org Sun Dec 3 01:21:29 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2C53CE6EA11 for ; Sun, 3 Dec 2017 01:21:29 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-ot0-x232.google.com (mail-ot0-x232.google.com [IPv6:2607:f8b0:4003:c0f::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E14F33CDE; Sun, 3 Dec 2017 01:21:28 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-ot0-x232.google.com with SMTP id b54so12031990otd.8; Sat, 02 Dec 2017 17:21:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-transfer-encoding; bh=BzA9DPuvJ+cL5IbV/btzQ4k5437smCnh1rMAAkKvbbk=; b=kgYRuSB+MtqzbLoYLaotQz2qcWc5hpxCkpZFJiwo7QWay6klzYF3YZB/kVNQHLFrdo 12hSHWCJ/HZEjgoGXbDoB6fw6XT2cRUPUnXq+l7oQxArQKhiiTcnAfG5cX0WUM3RnHAC 8fcjL9l7vJxfc8iDB5YS5cwtUPzU/JhBaB5NflIa/oSgFgTaPHANGGinDdcRbzz6zCAi +XzQ7okG6KQmZZgFb8V+DKaLDpIsQkmSnHuQu9jm3qZoRmqWew/Zuvro5oLU7B3GNfC8 29m+Lv/aMDXUt0NvqGPmjBVedZCPzHCy11fWKhlIPb4TvqKnEmvXOFLch3bfJSlHi62G caqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-transfer-encoding; bh=BzA9DPuvJ+cL5IbV/btzQ4k5437smCnh1rMAAkKvbbk=; b=YbFDEWVK7I+Vr1ET394+3ZFjoSlbpdLX57IhbJo2BYKSUg1CRowJgaMRmtzhOErPY4 sKnh22BAJVIqciQtN21ebvO9iu45jrIbb9ZkayGPMrfNxQ94ZCZoYujPNvNMr7r5Ao6u KHauMa8Zg4/P6HdN/cgOO3E40bjHB2mOtrRVYz2HhU+jwIYRtjAzLZWPKuIwDebOQWJL GeDGc3rgQwZvS6euHohJiRqFpaOXod71BoqL1ztjPgtpeDSciRCxtuXDoOTu/5q4Futk HrXbB6EKNtthtG3w52HLJ0tc7sWIy8vaGralNMQiekBH1d3Dg0LpAS4NC3kbdwjXD93y BYOw== X-Gm-Message-State: AJaThX4nMguRdxWW5Bi8EOEyRBIpOsSvjjaudJimY/bs9XkZG1oHFW72 0sWgOdGPOf6iGsOa85uV/y0NlAKZPooAdz+DpiwhOw== X-Google-Smtp-Source: AGs4zMYXQZ5tNrCY48ubfUKfkc8XByZFMItyHArt06m3BxX5j2gT7bgLk9J6EeqoiBBqCaJ14ttWD6LWllmU+yc8sCk= X-Received: by 10.157.45.161 with SMTP id g30mr11380674otb.122.1512264087914; Sat, 02 Dec 2017 17:21:27 -0800 (PST) MIME-Version: 1.0 Sender: kmacybsd@gmail.com Received: by 10.157.66.238 with HTTP; Sat, 2 Dec 2017 17:21:27 -0800 (PST) In-Reply-To: <11e6e55d-9802-c9fc-859c-37c026eaba2b@freebsd.org> References: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> <11e6e55d-9802-c9fc-859c-37c026eaba2b@freebsd.org> From: "K. Macy" Date: Sat, 2 Dec 2017 17:21:27 -0800 X-Google-Sender-Auth: kfEpn_c5IkcAcTkYbTs_tkqshuo Message-ID: Subject: Re: bhyve uses all available memory during IO-intensive operations To: Allan Jude Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Dec 2017 01:21:29 -0000 On Sat, Dec 2, 2017 at 5:16 PM, Allan Jude wrote: > On 12/02/2017 00:23, Dustin Wenz wrote: >> I have noticed significant storage amplification for my zvols; that coul= d very well be the reason. I would like to know more about why it happens. >> >> Since the volblocksize is 512 bytes, I certainly expect extra cpu overhe= ad (and maybe an extra 1k or so worth of checksums for each 128k block in t= he vm), but how do you get a 10X expansion in stored data? >> >> What is the recommended zvol block size for a FreeBSD/ZFS guest? Perhaps= 4k, to match the most common mass storage sector size? >> >> - .Dustin >> >>> On Dec 1, 2017, at 9:18 PM, K. Macy wrote: >>> >>> One thing to watch out for with chyves if your virtual disk is more >>> than 20G is the fact that it uses 512 byte blocks for the zvols it >>> creates. I ended up using up 1.4TB only half filling up a 250G zvol. >>> Chyves is quick and easy, but it's not exactly production ready. >>> >>> -M >>> >>> >>> >>>> On Thu, Nov 30, 2017 at 3:15 PM, Dustin Wenz = wrote: >>>> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS= is also FreeBSD 11.1). Their sole purpose is to house some medium-sized Po= stgres databases (100-200GB). The host system has 64GB of real memory and 1= 12GB of swap. I have configured each guest to only use 16GB of memory, yet = while doing my initial database imports in the VMs, bhyve will quickly grow= to use all available system memory and then be killed by the kernel: >>>> >>>> kernel: swap_pager: I/O error - pageout failed; blkno 1735,size= 4096, error 12 >>>> kernel: swap_pager: I/O error - pageout failed; blkno 1610,size= 4096, error 12 >>>> kernel: swap_pager: I/O error - pageout failed; blkno 1763,size= 4096, error 12 >>>> kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >>>> >>>> The OOM condition seems related to doing moderate IO within the VM, th= ough nothing within the VM itself shows high memory usage. This is the chyv= es config for one of them: >>>> >>>> bargs -A -H -P -S >>>> bhyve_disk_type virtio-blk >>>> bhyve_net_type virtio-net >>>> bhyveload_flags >>>> chyves_guest_version 0300 >>>> cpu 4 >>>> creation Created on Mon Oct 23 16:17:04 CDT 2= 017 by chyves v0.2.0 2016/09/11 using __create() >>>> loader bhyveload >>>> net_ifaces tap51 >>>> os default >>>> ram 16G >>>> rcboot 0 >>>> revert_to_snapshot >>>> revert_to_snapshot_method off >>>> serial nmdm51 >>>> template no >>>> uuid 8495a130-b837-11e7-b092-0025909a8b56 >>>> >>>> >>>> I've also tried using different bhyve_disk_types, with no improvement.= How is it that bhyve can use far more memory that I'm specifying? >>>> >>>> - .Dustin >> _______________________________________________ >> freebsd-virtualization@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fre= ebsd.org" >> > > Storage amplification usually has to do with ZFS RAID-Z padding. If your > ZVOL block size does not make sense with your disk sector size, and > RAID-Z level, you can get pretty silly numbers. That's not what I'm talking about here. If your volblocksize is too small you end up using (vastly) more space for indirect blocks than data blocks. -M