From nobody Fri Feb 16 18:00:13 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Tc08H4g3Fz53wwV for ; Fri, 16 Feb 2024 18:00:15 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx2.shrew.net (mx2.shrew.net [204.27.62.58]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Tc08H3hf2z3xW2 for ; Fri, 16 Feb 2024 18:00:15 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; none Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx2.shrew.net (8.17.1/8.17.1) with ESMTP id 41GI0DH3036228; Fri, 16 Feb 2024 12:00:13 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1708106413; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=7acgBZXoWjZKLVXFxT2S150WoyWhqM5Ex951cfRkd5g=; b=B2u76vRtq0FpOauZhYxJSXo1zkWtHKimJ83sDhFuTrFJepCh2Hi7tzqzpIcK9SRtsgYmDb ex+OmM9tj6Q8Jp7g20rbelDSjnauLaX5P0+cATHdgRMBz+Xv83ns4i4FDOn/JwQhXvoyu1 1gd9rwtK9SKzIciACYxq2sITAQbNk5YuF8+QFi/3Nh0vNLtFKqEaiTMquZ0atUxqk2yhxb eXFUvnDImuc30gTJWsmQ0xAm623BXW6Z0j1dXNZTQTLccyYubOzSx0ZKqMRLzySf6bd55U ZQjetNaWyHidKm6vLAxgIK1I68Rt+poLYXS4KckC3acItsffFm0NYp/LneDXTw== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id CB83D3B587; Fri, 16 Feb 2024 12:00:13 -0600 (CST) Content-Type: multipart/alternative; boundary="------------OUO03Djdqszf3XOd8PWHDNYK" Message-ID: <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> Date: Fri, 16 Feb 2024 12:00:13 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: Chuck Tuffli , FreeBSD virtualization References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> From: Matthew Grooms In-Reply-To: <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> X-Spamd-Bar: ---- X-Rspamd-Queue-Id: 4Tc08H3hf2z3xW2 X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US] This is a multi-part message in MIME format. --------------OUO03Djdqszf3XOd8PWHDNYK Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/16/24 11:42, Chuck Tuffli wrote: > On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: >> >> Hi All, >> >> >> I'm in the middle of a project that involves building out a handful >> of servers to host virtual Linux instances. Part of that includes >> testing bhyve to see how it performs. The intent is to compare host >> storage options such as raw vs zvol block devices and ufs vs zfs disk >> images using hardware raid vs zfs managed disks. It would also involve >> >> > … >> >> Here is a list of a few other things I'd like to try: >> >> >> 1) Wiring guest memory ( unlikely as it's 32G of 256G ) >> 2) Downgrading the host to 13.2-RELEASE > > FWIW we recently did a similar exercise and saw significant > performance differences on ZFS backed disk images when comparing 14.0 > and 13.2. We didn’t have time to root cause the difference, so it > could simply be some tuning difference needed for 14. Hi Chuck, That's very helpful feedback. I'll start by downgrading the host to 13.2 and report back here. Thanks again, -Matthew --------------OUO03Djdqszf3XOd8PWHDNYK Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit On 2/16/24 11:42, Chuck Tuffli wrote:
On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,


I'm in the middle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


…

Here is a list of a few other things I'd like to try:


1) Wiring guest memory ( unlikely as it's 32G of 256G )
2) Downgrading the host to 13.2-RELEASE

FWIW we recently did a similar exercise and saw significant performance differences on ZFS backed disk images when comparing 14.0 and 13.2. We didn’t have time to root cause the difference, so it could simply be some tuning difference needed for 14.

Hi Chuck,

That's very helpful feedback. I'll start by downgrading the host to 13.2 and report back here.

Thanks again,

-Matthew

--------------OUO03Djdqszf3XOd8PWHDNYK--