From owner-freebsd-virtualization@freebsd.org Sun Sep 3 03:52:00 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A1B7E17587 for ; Sun, 3 Sep 2017 03:52:00 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: from mail-yw0-x230.google.com (mail-yw0-x230.google.com [IPv6:2607:f8b0:4002:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D072E80A2F for ; Sun, 3 Sep 2017 03:51:59 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: by mail-yw0-x230.google.com with SMTP id c85so8338557ywa.0 for ; Sat, 02 Sep 2017 20:51:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=NwLxzMevKiRzf84drB1Wi/qjmi2KjP7PoQvqgQ+ezRE=; b=PI2/Op4WysWCyFhDTHgFxj8q9SAjV2Dj1BpSLLOlgBPBiVqODqt1tqR7Yhw4K9lXWg LcXPAgNTGdL2C+ho7MEX6DdREEtLKq0SugyIeAENRxh2dkFRxLf4ONRSeT1vQuW6o3/z wO0RFZXWSGwPPWzBbcYHP0fLkj2S+z4i11hRRfre+ArI3/SKz2VL2CpRwl+cQ0LOratr PLBYQH80uIwXyryq6vFcXc1ewsVXCIC1ZbuiA14BNkOdypFbDZDTD9ijZiF10tJJ7dXd qsEjropHGB+miFz1WASjQFl+wKPvvXjElPm1NOEeXjx/kHayB3Ku1/zD1SAoUHhxe5Er /0fw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=NwLxzMevKiRzf84drB1Wi/qjmi2KjP7PoQvqgQ+ezRE=; b=Cx0HF1RgTDyiT/HAfX1nmve/qoQubauSl6h4prPdkvtDhjaSFA4FOajHokxJekEUwI jqME4ihmoy39Oo7CW8Ak9wEogyK5TCvB0HphNwsnVzLDv0YWrw+4wha3ivQH9BpECFeH BiCKshYzaV4ZvJG2c+n7fg9qHZQ4eEcEo3kFKj0GIpARxI61JOUs5IHj1fPzODJ11foS TC7GU3/mh6fWhbGzIrhT/fxNdxNxd0UTRZuuZUIVbMuZEauF0kPMe1Dbv0TEvCb3KzCY A1X6zJXVe1+tj0J6Bt98HaRYbzyUIC3fqfl9s6CFx3ooKWHTFq0OPdGjhAOHr7sbgaID EF0g== X-Gm-Message-State: AHPjjUjVE12fdC6yLBan5Nq+PQ4+3VHiLE76IYasUlrAiz/HW6bp6mVO MDn2mlQY9061i1txsWkeBXelX+Y5QPaq X-Google-Smtp-Source: ADKCNb5oUkshyfHGCThXFuOerCrvHDKksROPETUGgeTSFTXJ7qnZmeatuNDX9lMjUf+0f1QGMoJQgZq5ZWeq7s9oV5g= X-Received: by 10.37.224.151 with SMTP id x145mr3078000ybg.74.1504410718506; Sat, 02 Sep 2017 20:51:58 -0700 (PDT) MIME-Version: 1.0 Sender: jtubnor@gmail.com Received: by 10.37.198.10 with HTTP; Sat, 2 Sep 2017 20:51:38 -0700 (PDT) In-Reply-To: <147e2841-bb29-1fe6-11cc-917f22bf8e87@osfux.nl> References: <147e2841-bb29-1fe6-11cc-917f22bf8e87@osfux.nl> From: Jason Tubnor Date: Sun, 3 Sep 2017 13:51:38 +1000 X-Google-Sender-Auth: GklOG4YiV8egVGCKISv91MhoLUI Message-ID: Subject: Re: bhyve VM drive size limit? To: Ruben Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Sep 2017 03:52:00 -0000 Hi Ruben, On 3 September 2017 at 00:52, Ruben wrote: > > Perhaps slightly off-topic, but might you have any recommendations > regarding those blocksizes? > I use 8k zvol block sizes. This seems to work consistently for me with OpenBSD and Windows Server 2016 guests. While you could probably go larger, it really depends on your use case and how the data intends to move around in the guest. Some people have had performance issues with 8k blocks, however, I am yet to experience this. Chyves by default used 512 byte block sizes, this is sub-optimal for a guest with > 9GB storage needs, so I change the default settings back to 8k. > > I'm not really that well versed in blocksizes / filesystems and am often > wondering where my 5TB zvol (bhyves into an ext4 volume for an ubuntu > guest) exactly disappears into :P For the above, I have experienced where block sizes were defined at 512 bytes, the associated block checksums would eat up the storage, just as quick as the data being committed to the zvol (even more so as the snapshots pile up). Increasing the zvol block size increased the storage to checksum ratio, reducing how much the checksum overhead had an affect on the storage. Keep in mind, this is with a zmirror, so mileage may vary with raidz or raidz2 setups. Another thing to keep in mind, as you increase the block size, this can have performance impacts on the guest as a 4k update in the guest filesystem, would but upwards pressure on array performance if you used say 32-64k blocks (rewriting > 32x the data+checksum calcs). I suggest trying different sizes for the same workload you plan to commit to the guest storage to see which has the best storage to performance ratio. Cheers, Jason. P.S. In the above, I wouldn't commit the 5TB to the guest in one hit. Either grow it out as you need it or add additional zvols to your guest as needed.