From owner-freebsd-emulation@freebsd.org Wed Nov 16 17:17:57 2016 Return-Path: Delivered-To: freebsd-emulation@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0B66C459E7 for ; Wed, 16 Nov 2016 17:17:57 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from smtp.rlwinm.de (smtp.rlwinm.de [IPv6:2a01:4f8:201:31ef::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BE2E7196D for ; Wed, 16 Nov 2016 17:17:57 +0000 (UTC) (envelope-from crest@rlwinm.de) Received: from vader9.bultmann.eu (unknown [87.253.189.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.rlwinm.de (Postfix) with ESMTPSA id 1029410F28 for ; Wed, 16 Nov 2016 18:17:48 +0100 (CET) Subject: Re: bhyve: zvols for guest disk - yes or no? To: freebsd-emulation@freebsd.org References: From: Jan Bramkamp Message-ID: Date: Wed, 16 Nov 2016 18:17:47 +0100 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-emulation@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Development of Emulators of other operating systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Nov 2016 17:17:58 -0000 On 16/11/2016 15:47, Patrick M. Hausen wrote: > Hi, all, > > we are just starting a project that will run a couple > of Ubuntu guests on top of bhyve instead of ESXi > that we used in the past. > > As far as I could find out, more or less all bhyve > manager/wrapper tools use zvols as the backing > store for "raw" guest disk images. > > I looked at > > * chyves > * iohyve > * vm-bhyve > > So far so good. Yet, this blog article has some very > valid (IMHO) points against using them: > > http://jrs-s.net/2016/06/16/psa-snapshots-are-better-than-zvols Afaik this is only a problem if you create a reservation for your ZVOLs. By default ZFS does create ZVOLs with reservations matching their size. The problem is that there is no way to signal to a VM that the overcommited SAN just ran out of space and your stupid operator is looking for ways to increase free space *NOW* and the VM should just retry. Instead the VM gets a general write error. To keep this promise ZFS has to reserve enough space to change each bit in the ZVOL if you take a snapshot of a ZVOL. Other datasets are created without reservations by default because the POSIX file system API can report ENOSPC if the file system ran out of space. Of course you get the same problem if you storage VM images in a file system there is just no sane default setting protecting you, because you just outsmarted your poor operating system. > Another thing I'm pondering is: wouldn't it be better to > run on UFS so you can dedicate as much memory > as possible to VMs? ZFS isn't a memory eating boogie man. Yes ZFS uses a lot of memory and requires more memory than UFS for stable performance, but you get a lot in return: * End to end checksumming * Snapshots * Remote replication of snaphosts * Painless volume management * ... > So I'm a bit puzzled now on what to do. Any opinions, > experiences, war stories to share? ZFS saved my bacon more than once. Twice it detected and corrected datacorruption on dying hardware in time. With a normal FS both cases would have turned into bitrot spreading into the backups until it's too late. Without ZFS you would require a reliable hardware RAID controller (if such a magical creature exists) instead (or build a software RAID1+0 from gmirror and gstripe). IMO money is better invested into more RAM keeping ZFS and the admin happy. -- Jan Bramkamp