From owner-freebsd-virtualization@freebsd.org Tue Dec 5 15:20:46 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 99B76E6D5B7 for ; Tue, 5 Dec 2017 15:20:46 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 722837EA98 for ; Tue, 5 Dec 2017 15:20:46 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 7F9DF846B770; Tue, 5 Dec 2017 09:20:44 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6M4XFHG0oKYr; Tue, 5 Dec 2017 09:20:44 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.21.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id 0BE9C846B763; Tue, 5 Dec 2017 09:20:44 -0600 (CST) From: Dustin Wenz Message-Id: <423F466A-732A-4B04-956E-3CC5F5C47390@ebureau.com> Content-Type: multipart/signed; boundary="Apple-Mail=_1D490CF7-77CD-464F-B3DB-17A2F86D455D"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: Storage overhead on zvols Date: Tue, 5 Dec 2017 09:20:43 -0600 In-Reply-To: Cc: FreeBSD virtualization To: Adam Vande More References: X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Dec 2017 15:20:46 -0000 --Apple-Mail=_1D490CF7-77CD-464F-B3DB-17A2F86D455D Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Thanks for linking that resource. The purpose of my posting was to = increase the body of knowledge available to people who are running bhyve = on zfs. It's a versatile way to deploy guests, but I haven't seen much = practical advise about doing it efficiently.=20 Allan's explanation yesterday of how allocations are padded is exactly = the sort of breakdown I could have used when I first started = provisioning VMs. I'm sure other people will find this conversation = useful as well. - .Dustin > On Dec 4, 2017, at 9:37 PM, Adam Vande More = wrote: >=20 > On Mon, Dec 4, 2017 at 5:19 PM, Dustin Wenz = wrote: > I'm starting a new thread based on the previous discussion in "bhyve = uses all available memory during IO-intensive operations" relating to = size inflation of bhyve data stored on zvols. I've done some = experimenting with this, and I think it will be useful for others. >=20 > The zvols listed here were created with this command: >=20 > zfs create -o volmode=3Ddev -o volblocksize=3DXk -V 30g = vm00/chyves/guests/myguest/diskY >=20 > The zvols were created on a raidz1 pool of four disks. For each zvol, = I created a basic zfs filesystem in the guest using all default tuning = (128k recordsize, etc). I then copied the same 8.2GB dataset to each = filesystem. >=20 > volblocksize size amplification >=20 > 512B 11.7x > 4k 1.45x > 8k 1.45x > 16k 1.5x > 32k 1.65x > 64k 1x > 128k 1x >=20 > The worst case is with a 512B volblocksize, where the space used is = more than 11 times the size of the data stored within the guest. The = size efficiency gains are non-linear as I continue from 4k and double = the block sizes; 32k blocks being the second-worst. The amount of wasted = space was minimized by using 64k and 128k blocks. >=20 > It would appear that 64k is a good choice for volblocksize if you are = using a zvol to back your VM, and the VM is using the virtual device for = a zpool. Incidentally, I believe this is the default when creating VMs = in FreeNAS. >=20 > I'm not sure what your purpose is behind the posting, but if its = simply a "why this behavior" you can find more detail here as well as = some calculation leg work: >=20 > = https://www.delphix.com/blog/delphix-engineering/zfs-raidz-stripe-width-or= -how-i-learned-stop-worrying-and-love-raidz >=20 > --=20 > Adam --Apple-Mail=_1D490CF7-77CD-464F-B3DB-17A2F86D455D Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEzDCCBMgw ggOwoAMCAQICAUEwDQYJKoZIhvcNAQELBQAwgZkxCzAJBgNVBAYTAlVTMRIwEAYDVQQIEwlNaW5u ZXNvdGExFDASBgNVBAcTC1NhaW50IENsb3VkMRAwDgYDVQQKEwdlQnVyZWF1MRQwEgYDVQQLEwtJ bnRlZ3JhdGlvbjEUMBIGA1UEAxMLZWJ1cmVhdS5jb20xIjAgBgkqhkiG9w0BCQEWE3N1cHBvcnRA ZWJ1cmVhdS5jb20wHhcNMTcwNTA1MTYxNjE1WhcNMjcwNTAzMTYxNjE1WjBKMQswCQYDVQQGEwJV UzEUMBIGA1UEAwwLRHVzdGluIFdlbnoxJTAjBgkqhkiG9w0BCQEWFmR1c3RpbndlbnpAZWJ1cmVh dS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQ/HJXe7JhUaexqEaxCNVifsue sUMgohgXLmi5YIcbAWhdxTr1PUzKYkeTkL9sYNjXU6uaI2tZMF3hA9gcFjxQIfkKSo31MrYOgMdU xQe0Q+t6Vd4pGAmtDQDwhAsrNGccADp3Yjy4eBtVfkDGdzz1Y8Lbc684TPFcW7i9+U/dDaXlcxeq fyDqiHZ5y8Lp/1M2Ot/Rz7eikJZTAuHOWKs/PEiJIM2JHuhPyNy+mL2oqEWeOcEsKMNzgn7HVt4k Xz2irBAG+cj4WAxWs418l46EEXgur4PvhBXZMl0LJg0TyaxOHbsUam4R4tbKnaZ3HhRkg79k2Had sb6DKbnCw9/1AgMBAAGjggFnMIIBYzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAnBgNVHSUEIDAe BggrBgEFBQcDAQYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBTLi/8HUHpbBEt9OtPqQoax AmpaNDCBzgYDVR0jBIHGMIHDgBRnpZeXB5rQYLgsUKqiiBcLIHyu6aGBn6SBnDCBmTELMAkGA1UE BhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoT B2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAG CSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbYIJAMwZcjAWAsWXMDAGCWCGSAGG+EIBBAQj FiFodHRwOi8vd3d3LmVidXJlYXUuY29tL2NhLWNybC5wZW0wDQYJKoZIhvcNAQELBQADggEBAHbO qVdB9raUKXCgZRA/nES5a60dlIaGnIlpgz+Y3SjFt0bcJxoUYhIzumBHk9yjyP4M1DubOphkQpJ4 LNZbAS01cjCxjnC0ZUq5V3FCeaDwrn1qPY+QJGoZPLlhWdJUNu17OpnR7ZfBWlp3/pRhvNU5PCbJ nmF7rnvsqxUFq9oeiV3SmqBux5lwJ7p2Uss5SHSW6g17K/KdTMK1roQr/+rWpxp2233qddDrLpOE xGRlvhEqSa/IZbGC9oiYmsiaG1PefQkadoob5IMIS5/MDpWHUgSHqAj1V/LwcCx0rbt73SazGMND EzHVWhsj+khepB/MG5QGfWP23IGFmvQYWWcxggOQMIIDjAIBATCBnzCBmTELMAkGA1UEBhMCVVMx EjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoTB2VCdXJl YXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAGCSqGSIb3 DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTAJBgUrDgMCGgUAoIIBxTAYBgkqhkiG9w0BCQMx CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNzEyMDUxNTIwNDNaMCMGCSqGSIb3DQEJBDEW BBSiUZAcewvpJ5w262TQiHjTQDFWiDCBsAYJKwYBBAGCNxAEMYGiMIGfMIGZMQswCQYDVQQGEwJV UzESMBAGA1UECBMJTWlubmVzb3RhMRQwEgYDVQQHEwtTYWludCBDbG91ZDEQMA4GA1UEChMHZUJ1 cmVhdTEUMBIGA1UECxMLSW50ZWdyYXRpb24xFDASBgNVBAMTC2VidXJlYXUuY29tMSIwIAYJKoZI hvcNAQkBFhNzdXBwb3J0QGVidXJlYXUuY29tAgFBMIGyBgsqhkiG9w0BCRACCzGBoqCBnzCBmTEL MAkGA1UEBhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAO BgNVBAoTB2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNv bTEiMCAGCSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTANBgkqhkiG9w0BAQEFAASC AQCBXotEZa74uWViaewdng8UcUseVUjhOLgQy+syE37Aa4I2zM252Ctqdwq5HuI5fpv9xf1Tt8hJ helNC/Z+nFapllc4EAzOrFSbYvxMVPWzEPg53dMJ8yw+GOV4/1fZ/cQUsdb+JbGeSVry06CaixTo sAxQXbedyJBdwDPD9IfJk5vGzguljAbUUuS1QVg8Wf/HuGmzKV/ToPt23wIpasqB6qd3smrXq1UL rjQdGfTOauFX0Mbed+kUKB6T+Jh1a08+nqsWqZuM0i5YSqEkDvf4qbRw638jQzatv38cI18x8RMP I4N2fsvWB4057MadSjm4X/c0BfH8QTC/F8rU19mIAAAAAAAA --Apple-Mail=_1D490CF7-77CD-464F-B3DB-17A2F86D455D--