From owner-freebsd-virtualization@freebsd.org Thu Aug 27 10:10:47 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AE8629C3669 for ; Thu, 27 Aug 2015 10:10:47 +0000 (UTC) (envelope-from marieheleneka@gmail.com) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 44F4713B9 for ; Thu, 27 Aug 2015 10:10:47 +0000 (UTC) (envelope-from marieheleneka@gmail.com) Received: by widdq5 with SMTP id dq5so72931909wid.1 for ; Thu, 27 Aug 2015 03:10:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-type; bh=AoHIhFw2FvlO51tNsDwjsWuYPVlOOf1XhzqxiH1Zx2g=; b=E7nAiDLriFnhDuKcgtWb7FL88zDXKbsN/UierWsptzCoOggCdkqWskBMFZ7w4fuJjZ XWitG/reHn93GYHGky2Tq3ENDkU3VoeB8mzM97ZI6Rj4n0xUkVLsPQXEAER1HaVmysia YF0OtyuDBpD1X6q/fQaaoMfUKjxoO2JH9YYUtuqvPk1FQd1+B6j9Ttxf9gBo6o2TRqN8 OflFNrP4dJvl1s6Ult2mp00X6IA7AmNXzpKLhd2e29Vc1dHtP8VvyNZvQSHW/Xf4EA5u H89BOrg+TmxQzrl7NTF8tvby5R/LhbSNwFIxYU275IOF7oiRDpKijnU0elc4w15VrL14 qYrA== X-Received: by 10.194.83.70 with SMTP id o6mr4079681wjy.44.1440670245672; Thu, 27 Aug 2015 03:10:45 -0700 (PDT) MIME-Version: 1.0 References: <20150827061044.GA10221@blazingdot.com> <20150827062015.GA10272@blazingdot.com> <1a6745e27d184bb99eca7fdbdc90c8b5@SERVER.ad.usd-group.com> In-Reply-To: <1a6745e27d184bb99eca7fdbdc90c8b5@SERVER.ad.usd-group.com> From: Marie Date: Thu, 27 Aug 2015 10:10:36 +0000 Message-ID: Subject: Re: Options for zfs inside a VM backed by zfs on the host To: Matt Churchyard , Marcus Reid , Vick Khera Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Aug 2015 10:10:47 -0000 On Thu, Aug 27, 2015 at 11:42 AM Matt Churchyard via freebsd-virtualization wrote: > > On Wed, Aug 26, 2015 at 11:10:44PM -0700, Marcus Reid wrote: > > On Wed, Aug 26, 2015 at 05:25:52PM -0400, Vick Khera wrote: > > > > Opinions? Preferably well-reasoned ones. :) > > > > > > > However, having the ARC eating up lots of memory twice seems pretty > > > bletcherous. You can probably do some tuning to reduce that, but I > > > never liked tuning the ARC much. > > > I just realized that you can turn primarycache off per-dataset. Does it > make more sense to turn primarycache=none on the zvol on the host, or > on > the datasets in the vm? I'm thinking on the host, but it might be worth > experimenting. > > I'd be very wary of disabling ARC on the main host, it can have pretty > serious side effects. It could possibly be useful in the guest though, as > data should be cached already by ARC on the host, you're just going through > an extra step of reading through the virtual disk driver, and into host > ARC, instead of directly from the guest memory. Would need testing to know > what performance was like and if there are any side effects. > > I do agree that it doesn't seem unnecessary to have any redundancy in the > guest if the host pool is redundant. Save for any glaring bugs in the > virtual disk emulation, you wouldn't expect to get errors on the guest pool > if the host pool is already checksumming the data. > > It's also worth testing with guest ARC enabled but just limited to a > fairly small size, so you're not disabling it entirely, but doing at little > double-caching as possible. > > ZFS features seems perfect for virtual hosts, although it's not ideal that > you have to give up a big chunk of host RAM for ARC. You may also find that > you need to limit host ARC, then only use "MAX_RAM - MY_ARC_LIMIT" for > guests. Otherwise you'll have ZFS and VMs fighting for memory and enough of > us have seen what shouldn't, but unfortunately does happen in that > situation. > > Matt > - > > > Marcus > I've tried this in the past, and found the worst performance penalty was with ARC disabled in guest. I tried with ARC enabled on host and guest, only on host, only on guest. There was a significant performance penalty with either ARC disabled. I'd still recommend to experiment with it on your own to see if the hit is acceptable or not. Shameless plug: I'm working on a project (tunnelfs.io) which should be useful for this use case. :) Unfortunately, there is no ETA on usable code yet. -- Marie Helene Kvello-Aune