From owner-freebsd-emulation@freebsd.org Thu Nov 17 10:36:21 2016 Return-Path: Delivered-To: freebsd-emulation@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DC63AC451B3 for ; Thu, 17 Nov 2016 10:36:21 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5D679107A for ; Thu, 17 Nov 2016 10:36:20 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id uAHAaIA5074624 for ; Thu, 17 Nov 2016 11:36:18 +0100 (CET) Received: from [217.29.44.220] ([217.29.44.220]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id uAHAaHTe081867 for ; Thu, 17 Nov 2016 11:36:18 +0100 (CET) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: bhyve: zvols for guest disk - yes or no? From: "Patrick M. Hausen" In-Reply-To: <5be68f57-c9c5-7c20-f590-1beed55fd6bb@rlwinm.de> Date: Thu, 17 Nov 2016 11:36:17 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <46273DBA-06F8-44F2-88AD-C4812E249ED8@punkt.de> References: <5be68f57-c9c5-7c20-f590-1beed55fd6bb@rlwinm.de> To: freebsd-emulation@freebsd.org X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-emulation@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Development of Emulators of other operating systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 Nov 2016 10:36:22 -0000 Hi, all, > Am 17.11.2016 um 11:16 schrieb Jan Bramkamp : > Bhyve doesn't support page deduplication but it doesn't wire down = guest memory unless you asked it to, but you have to wire down guest = memory to use PCI passthrough. If I were picking VM hosts today I would = go with LGA 2011 v3 boards having least eight DDR4 slots per socket. = Maybe use some nice >=3D 2TB NVMe SSDs and suddenly your limited by CPU = cycles and storage space instead of IOPS and RAM. Perhaps I'll describe the intention of this project. All your = suggestions are great for a heavy virtualisation platform. In our case we just want a cheap, yet "reliable enough" platform to host a handful of rocket.chat instances. While rocket.chat claims FreeBSD support that's way behind compared to running the product on Linux. We found Ubuntu to be a good choice in this case. Unfortunately the application's architecture doesn't easily fit running multiple instances on one kernel/OS. The absence of current FreeBSD support rules out jails. So a full-fledged hypervisor is called for. And we will use a system with two mirrored SATA disks to run no more than 10 Ubuntu guests with at most 4 GB each and 100 GB of guest disk space. 1 or 2 TB of spinning rust in a mirror, that's all. At least rocket.chat is not performance hungry. The only issue I saw is RAM (CPU supports at most 32 GB) and the ZVOL "issue" I initially asked about. We will now provision them = "sparse" with a small block size ... > An other thing I learned the hard way is that ZVOL are set in stone at = the ZVOL creation. You have to (cam)dd everything to change the block = size. The default ZVOL block size is 8K which isn't wrong but your = guests need to align their file systems (and swap) correctly or you'll = suffer from write amplification. And ZFS RAID-Z really sucks for such = small block sizes. Use mirrored VDEVs in your pools or you will suffer = from massive metadata overhead and disappointing IOPS. I attended Kirk's kernel class on last EuroBSDCon ;-) You are = essentially preaching to the choir - but for the list archives and later searches = sake, your work will probably do somebody else some good. Thanks again, Patrick --=20 punkt.de GmbH * Kaiserallee 13a * 76133 Karlsruhe Tel. 0721 9109 0 * Fax 0721 9109 100 info@punkt.de http://www.punkt.de Gf: J=C3=BCrgen Egeling AG Mannheim 108285