From owner-freebsd-virtualization@freebsd.org Mon Jun 24 22:01:01 2019 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1185E15DAA27 for ; Mon, 24 Jun 2019 22:01:01 +0000 (UTC) (envelope-from k@free.de) Received: from smtp.free.de (smtp.free.de [91.204.6.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7F3D98AF94 for ; Mon, 24 Jun 2019 22:00:59 +0000 (UTC) (envelope-from k@free.de) Received: from [127.0.0.1] (localhost [127.0.0.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.free.de (Postfix) with ESMTPSA id 1FF951162C for ; Tue, 25 Jun 2019 00:00:49 +0200 (CEST) To: freebsd-virtualization@freebsd.org From: Kai Gallasch Subject: ZVOL backed bhyve VMs running on UFS or ZFS - Best practices, pros and cons, performance Openpgp: preference=signencrypt Autocrypt: addr=k@free.de; prefer-encrypt=mutual; keydata= xsFNBFHV6PQBEAC46oZo3Nv4qBhAkFDUyIZH1g58rIMAMCdSFWcnorZiv2tCWXuKYxFY1P5K IjYw3Mnwpi7NneCtnxgPEbZbIEqxk9dnL2yYSdT5dPlfWw5KPvIcKhzOQZTf6mOC+dNsj9Ek UGPXW1cLnG5TOzKGEIFSaZPLECUcGl3temkEvcqW74A6ms2dOMIH5snY9mptGTEyqtiT6Rhd sFuRZuj3NA1IFxpVMwfWJ0iBk+YK/2s5HQVj/mhAfDhShcjfASxxZhLrDYn3Xgj+BRAxW6S3 BCqc97EL9Hayz3U9jEnbDCoSbZK/Fn651YZ5gVOQ/Os9nZ9z/rPRYeKGm11P0tyMFbsfuROz ZpPVRV07Ms4Cdxrwck6i4W51n+fIelVa1Pm71mtzhxBJdDx8fmuLRd7cMaGvYGiYPw4DOG8d q2ZlQbfYHgvo3zqmVJixv4+ZlGZspsZf1wvvqYqttLnVuepQrbpVX6vMWqii6p3dBasSnULx 2sulzQ943ow862hw2OcaLDkTIuEjE4ebnMgclGwvvGQOFfoZfdnf8DSSZM55+l4lBV19ueu+ ynKqtBNJj+GxBYMjy1dCqUpYXaXo47crPKGsvDCu1CWAu+W6kKjRPNZhLzRkEWzMfb3koxm2 tgaW1TPvGJfnLbxh+/X9cEpJUWArug11M9EQoBB1XJlxQbjW9wARAQABzRhLYWkgR2FsbGFz Y2ggPGtAZnJlZS5kZT7CwYAEEwEKACoCGwMFCwkIBwMFFQoJCAsFFgMCAQACHgECF4ACGQEF AllXbC8FCQtD6jYACgkQcGVNfE+x9Yj2Jw//fL0Tm6HX/yeyr5qumeSjI78ocxTLdkp7Lvo1 2JsmJ2uGZbasOjzgskEcz5+UPoq65H9pGxkpYyPw7Ps7nwPkle7kx90/ODD5BtwCVmfDY0fy NhJCVgwF/ekvF5tgX5d2GuIRWsxk/zKnPt3/djl7HV3XPeR5Vv9S53f2xPHHribeqdMs0ZaQ pHoKLC614X5JidySJXZw2zOV1BhbzYxDlPI5P0c+QboHkJh6KVMeuBc7cXZlumkoeLUhnAO2 jv1QabnfJY06jEIEPIJ7KRrkCY0HLnXSs7MuJuLEVF6ll1RKtuYJuhUAdWFlj8+L4JXS8cWg OFyGpXonC+Y/LD4V3T/CNUVEtlwNbDDVFjtd8R7A4ouQTchFTgw/dM3SxxS8l6UsXItAvOPo pTnUrl36YxWULFTuQh6AQa0piJilMiokBhfzTOfqfUs7iXRJe0ZFwi+fwWuuz3VHjOm3WUwX 3Rw7kB8RlzaGJp+LL8GhfnI1RoU2Bg87aEk+vm7+yivY4lbPK5nUwk4TJTMlcRCOqxwKzwWv NzZhYZHAJhwdhARmi4O27z/qtI/VeDhNvkesHqf9GUz9U0f/2MOvK41PyhBjcbm1Jui9Djdm WSBjgCR69P23uJOX7Iv9eOxPtBsbReHZAr7P5ngahtXQckFQlG69GNZ9k6mdKz7wGuuLFiTO wU0EUdXo9AEQANkUT0xdxeDDPh/U6+5lQmcBgDkKqd9L9QBTgW28vvY+5OzkjIz4LmnCZCpq boFZNEOvgRM5BLtL5YIeKGKADvYoe/iRQan5xAgO+zgamCY58KL73046P1YhstW3Ka0+yKw5 i4WfeWPF4vXTLkZZjtyU77wOx0Dn0gEFyvKispUwL5YZP2oI7jEqRhSouysuJMmUw5vYF+GM H4oJQhIqVzYKVyT2K96je14dWvbLuRbmUCqLH4lTYMkx5xs5dAOLy8PanMV0lCQRdZ6pIarq GO7+lxdnLx85JpmmQpnZmyxOSIgATrfL/vti6oUoTPyJYI5aPGtMuLuP348JjblUxZs1tCx+ oTHUzj+S0YMTrGkzqG7JW2oMQY9H9mEIM2A9MPNiFm2VIFiiZAT/siWp/IpT3cdAIKrF1c8f fl9CM4xorxk6jhdguLqh5RZYsAIRtLwBJWZFG4gpDdHIEZibkIkPPzi6Frz9PGwjMEsGP+nP pHdkTxlC4XG9DaO5djbJYiSQQmaxQf59g0ukpNMoM8gxfVtbnfZWgFlaYitJwp33GpbpgUBS aYOtKdATmyqMnD05vWHrjp+3UeSrR0hA6VPBVya8Slivmzo0ROVm0yeMKNSUdhoAqeVVkYq2 TMHx++yh2I7kVvhMI1SCMKwvyOVRiCDC/Mf3jMMRLFUTQGRrABEBAAHCwWUEGAEKAA8CGwwF AllXbvQFCQtD7PsACgkQcGVNfE+x9Yg5vA//ROIcyVX9RNK2IlklDPO4S5CWJx1ZH4E2SvvU fY2EJG3j6oXClscZhonrfWV/97+446QNFL734KcUf1ZW4oeEH+x3huT36H+OKCBsYjy+kNLn AkItHhzNPSrAjqbRHxhMVk0hrxwYDBQ50F9bqTynhhll8AiSWnbuLtiMyUsbjP4BPRun+7nG e/wFvWLxYUL8taAoLtW2lLpGjd+OEZvgDY7V1gx56FxaRK/QOCLI4otfm8Rgl/Z/lw0bR+P1 vYPS8b5pmw109bJBpdgKjgbhEawg6S63wVBnjwh6i6D1SqsXZ58NkD2i+QO+nw41DyEJpvfT nlZTfeAmMeeMAM3iiEZXs+hqhSyiCjO13MyGrP1B0gAqAueotvuXqemxJ1QTHRqjECxTkao3 hfXo+V7Gm2Eg61IQTpl57f1OyRzv7V+qb9LJVJMIShh0O4vq9eNa6loJ+Q8NDGMDOeRLdnNU mBY4fEdrdX822SlK3sz7cQMm29R3AxrlwMzixMaNczwzYw+A4BE3j6ZPrKxI+DIVM5I5dCzd rarz98nrDo+6UXjAtErUB89fC2X8sZUsOkWkWEHBSZd5TMe72GLojnfA/Lkl+pWuIWYCdpEU vbvAc4LlamA3msqexwt1kthAXPUbfEev3+COto26uxCkFc2btIq6uOA3JoH3WJS4/89V5d8= Message-ID: <91ce1f2e-7e5c-2e1f-51bf-64185b53798a@free.de> Date: Tue, 25 Jun 2019 00:00:49 +0200 Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: de-DE Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 7F3D98AF94 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of k@free.de designates 91.204.6.103 as permitted sender) smtp.mailfrom=k@free.de X-Spamd-Result: default: False [-6.54 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DMARC_NA(0.00)[free.de]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MX_GOOD(-0.01)[pop.free.de]; NEURAL_HAM_SHORT(-0.99)[-0.988,0]; RCVD_IN_DNSWL_NONE(0.00)[103.6.204.91.list.dnswl.org : 127.0.10.0]; IP_SCORE(-3.75)[ip: (-9.86), ipnet: 91.204.4.0/22(-4.93), asn: 31371(-3.94), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:31371, ipnet:91.204.4.0/22, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jun 2019 22:01:01 -0000 Hi. I am planning to convert several 11.2 FreeBSD Jails into bhyve VMs. My bhyve setup uses a ZVOL for each VM (currently Debian and Ubuntu Linux using ext3/4) and I want to keep it that way for the new FreeBSD VMs. I really like the idea of having ZFS as file system inside the FreeBSD guests, but wonder if there are any strong points against running ZFS on top of ZFS. - Is the I/O performance of a FreeBSD guest with ZFS much worse compared to a UFS only guest or only e.g. 1/3 slower. Any numbers? - Wasted RAM allocation: Do FreeBSD guests with ZFS make any sense for small memory allocations < 2 GB? (Because ZFS caches will possibly consume most of the memory) - I/O bloat? Will you waste performance on the physical disks on the host server when running most of your bhyve guests with ZFS? - What ZFS send/receive performance can be expected in and out of a ZFS guest running on top of a ZVOL? (Is there an I/O penalty?) - Are there any ZFS or ZPOOL parameters inside the ZFS guest that should be tuned to reflect the situation that they are virtualized on a ZVOL? (maybe to save precious RAM) - Zpool Scrubbing inside the VM should not be necessary, right? - Can the file system journaling of a ZVOL hosted FreeBSD guest running UFS cope with a crash of the host server? Will the UFS journaling keep the file guest system integrity even if an older ZVOL snapshot of the VM will be rolled back (VM not running; Auto-Snapshot from the VM in state "running") Regards, Kai.