From owner-freebsd-virtualization@freebsd.org Tue Dec 5 16:14:44 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E5ECE6F4D4 for ; Tue, 5 Dec 2017 16:14:44 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1E3CF1081 for ; Tue, 5 Dec 2017 16:14:43 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 7113D139BE for ; Tue, 5 Dec 2017 16:14:42 +0000 (UTC) Subject: Re: Storage overhead on zvols To: freebsd-virtualization@freebsd.org References: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> From: Allan Jude Message-ID: <3496de79-8610-5640-0d2c-22031d7e3e5f@freebsd.org> Date: Tue, 5 Dec 2017 11:14:41 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 16:14:44 -0000 On 2017-12-05 10:20, Dustin Wenz wrote: > Thanks for linking that resource. The purpose of my posting was to increase the body of knowledge available to people who are running bhyve on zfs. It's a versatile way to deploy guests, but I haven't seen much practical advise about doing it efficiently. > > Allan's explanation yesterday of how allocations are padded is exactly the sort of breakdown I could have used when I first started provisioning VMs. I'm sure other people will find this conversation useful as well. > > - .Dustin > This subject is covered in detail in chapter 9 (Tuning) of "FreeBSD Mastery: Advanced ZFS", available from http://www.zfsbook.com/ or any finer book store. >> On Dec 4, 2017, at 9:37 PM, Adam Vande More wrote: >> >> On Mon, Dec 4, 2017 at 5:19 PM, Dustin Wenz wrote: >> I'm starting a new thread based on the previous discussion in "bhyve uses all available memory during IO-intensive operations" relating to size inflation of bhyve data stored on zvols. I've done some experimenting with this, and I think it will be useful for others. >> >> The zvols listed here were created with this command: >> >> zfs create -o volmode=dev -o volblocksize=Xk -V 30g vm00/chyves/guests/myguest/diskY >> >> The zvols were created on a raidz1 pool of four disks. For each zvol, I created a basic zfs filesystem in the guest using all default tuning (128k recordsize, etc). I then copied the same 8.2GB dataset to each filesystem. >> >> volblocksize size amplification >> >> 512B 11.7x >> 4k 1.45x >> 8k 1.45x >> 16k 1.5x >> 32k 1.65x >> 64k 1x >> 128k 1x >> >> The worst case is with a 512B volblocksize, where the space used is more than 11 times the size of the data stored within the guest. The size efficiency gains are non-linear as I continue from 4k and double the block sizes; 32k blocks being the second-worst. The amount of wasted space was minimized by using 64k and 128k blocks. >> >> It would appear that 64k is a good choice for volblocksize if you are using a zvol to back your VM, and the VM is using the virtual device for a zpool. Incidentally, I believe this is the default when creating VMs in FreeNAS. >> >> I'm not sure what your purpose is behind the posting, but if its simply a "why this behavior" you can find more detail here as well as some calculation leg work: >> >> https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or-how-i-learned-stop-worrying-and-love-raidz >> >> -- >> Adam > -- Allan Jude