From nobody Sun Feb 18 15:21:10 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4Td8Wt3Znlz59xJt for ; Sun, 18 Feb 2024 15:21:14 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from mx1.shrew.net (mx1.shrew.net [204.27.62.57]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Td8Ws3mKvz46pl for ; Sun, 18 Feb 2024 15:21:13 +0000 (UTC) (envelope-from mgrooms@shrew.net) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=shrew.net header.s=default header.b=t+KSVmxs; spf=pass (mx1.freebsd.org: domain of mgrooms@shrew.net designates 204.27.62.57 as permitted sender) smtp.mailfrom=mgrooms@shrew.net; dmarc=none Received: from mail.shrew.net (mail1.shrew.prv [10.26.2.18]) by mx1.shrew.net (8.17.1/8.17.1) with ESMTP id 41IFLBht071852 for ; Sun, 18 Feb 2024 09:21:11 -0600 (CST) (envelope-from mgrooms@shrew.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=shrew.net; s=default; t=1708269671; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=mLckGg/35HCYCGmXjDSvMooeZYNmSYcHT94WvQeeyDo=; b=t+KSVmxs6VncGiqjLFO/wNMv3CLAoDIoGBcq3We8NyPu8aYtVLskyGk8Y+hJ3cupZSrUAx jsHCVqPuCiNpq2o08GQ/AtCZ0kvO6JzS+Nnb9QoCzMHiVax18lWC32BojIJuaySje/BRRq dMVSgp+jN96t5G6SEv6VeA93fpabd1CvXr0mMa1Fb3RZH0ChOoNr4Owjw2OjVVM91cTLiI hopN9s0Jh+ahBC2mznyLXM2na+x7RgA3DYpNFwBCvAjB99xXp0oxbeMwLJ1BTBNNmim6d1 SgIxW1SwpVsaAL9do9WfJ9v1ejMxj22ORTJHFxWsyEB+avBF8+dOoh41lCYu3w== Received: from [10.22.200.32] (unknown [136.62.156.42]) by mail.shrew.net (Postfix) with ESMTPSA id 4B6193B553 for ; Sun, 18 Feb 2024 09:21:11 -0600 (CST) Content-Type: multipart/alternative; boundary="------------aSmCDs7yhrXrJYYlTx0uyb8U" Message-ID: <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net> Date: Sun, 18 Feb 2024 09:21:10 -0600 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: bhyve disk performance issue Content-Language: en-US To: virtualization@freebsd.org References: <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> From: Matthew Grooms In-Reply-To: <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> X-Rspamd-Queue-Id: 4Td8Ws3mKvz46pl X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.49 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; R_SPF_ALLOW(-0.20)[+mx]; R_DKIM_ALLOW(-0.20)[shrew.net:s=default]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; XM_UA_NO_VERSION(0.01)[]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; ASN(0.00)[asn:19969, ipnet:204.27.56.0/21, country:US]; RCPT_COUNT_ONE(0.00)[1]; MIME_TRACE(0.00)[0:+,1:+,2:~]; DMARC_NA(0.00)[shrew.net]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; DKIM_TRACE(0.00)[shrew.net:+] This is a multi-part message in MIME format. --------------aSmCDs7yhrXrJYYlTx0uyb8U Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 2/17/24 15:53, Matthew Grooms wrote: > On 2/16/24 12:00, Matthew Grooms wrote: >> On 2/16/24 11:42, Chuck Tuffli wrote: >>> On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote: >>>> >>>> Hi All, >>>> >>>> >>>> I'm in the middle of a project that involves building out a handful >>>> of servers to host virtual Linux instances. Part of that includes >>>> testing bhyve to see how it performs. The intent is to compare host >>>> storage options such as raw vs zvol block devices and ufs vs zfs >>>> disk images using hardware raid vs zfs managed disks. It would also >>>> involve >>>> >>>> >>> … >>>> >>>> Here is a list of a few other things I'd like to try: >>>> >>>> >>>> 1) Wiring guest memory ( unlikely as it's 32G of 256G ) >>>> 2) Downgrading the host to 13.2-RELEASE >>> >>> FWIW we recently did a similar exercise and saw significant >>> performance differences on ZFS backed disk images when comparing >>> 14.0 and 13.2. We didn’t have time to root cause the difference, so >>> it could simply be some tuning difference needed for 14. >> >> Hi Chuck, >> >> That's very helpful feedback. I'll start by downgrading the host to >> 13.2 and report back here. >> > > Unfortunately same story with 13.2. I'm going to try and downgrade to > 12.4 and see if it gets any better. > > ================================================================================ > Begin @ Sat Feb 17 11:00:01 CST 2024 > > Version  2.00       ------Sequential Output------ --Sequential Input- > --Random- >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  > /sec %CP > localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99 1.3g  68 > +++++ +++ > Latency             11759us   29114us    8098us    8649us 23413us    > 4540us > Version  2.00       ------Sequential Create------ --------Random > Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  > /sec %CP >                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ > +++++ +++ > Latency              7791us     131us    1671us     464us 15us    1811us > > End @ Sat Feb 17 11:03:13 CST 2024 > ================================================================================ > Begin @ Sat Feb 17 11:10:01 CST 2024 > > Version  2.00       ------Sequential Output------ --Sequential Input- > --Random- >                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- > --Seeks-- > Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  > /sec %CP > localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99 398m  99 > 16204 563 > Latency             12147us    1079us   24470us    8795us 17853us    > 4384us > Version  2.00       ------Sequential Create------ --------Random > Create-------- > localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- > -Delete-- >               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  > /sec %CP >                  16     0  93 +++++ +++ +++++ +++     0  96 +++++ +++ > +++++ +++ > Latency               118us     159us     269us     164us 54us     844us > > End @ Sat Feb 17 11:18:43 CST 2024 > I wasn't able to get a working 12.4 system in place due to lack of packages. However, I did fire up a FreeBSD 14 VM and let it run overnight on the same SSD array. It consistently ran at a much higher speed for 50+ runs @ 10m intervals ... ================================================================================ Begin @ Sun Feb 18 15:00:00 UTC 2024 Version  1.98       ------Sequential Output------ --Sequential Input- --Random-                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP freebsd.shrew.l 64G  628k  99  1.6g  98  831m  60 1278k  99  1.1g 42 +++++ +++ Latency             13447us   68490us     207ms    7187us 195ms   17665us Version  1.98       ------Sequential Create------ --------Random Create-------- freebsd.shrew.lab   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ Latency             18225us      18us      28us   18812us 18us      25us End @ Sun Feb 18 15:03:11 UTC 2024 I used identical options to run both that VM and the Linux VM I've been testing. The backing store for each VM has a 1TB partition and the guest interfaces are NVME. Now I'm really scratching my head. Chuck, were you testing disk performance in Linux VMs or only FreeBSD? Anyone have ideas on why Linux disk performance would drop off a cliff over time? Thanks, Thanks, -Matthew --------------aSmCDs7yhrXrJYYlTx0uyb8U Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 2/17/24 15:53, Matthew Grooms wrote:
On 2/16/24 12:00, Matthew Grooms wrote:
On 2/16/24 11:42, Chuck Tuffli wrote:
On Fri, Feb 16, 2024, at 9:19 AM, Matthew Grooms wrote:

Hi All,


I'm in the middle of a project that involves building out a handful of servers to host virtual Linux instances. Part of that includes testing bhyve to see how it performs. The intent is to compare host storage options such as raw vs zvol block devices and ufs vs zfs disk images using hardware raid vs zfs managed disks. It would also involve


…

Here is a list of a few other things I'd like to try:


1) Wiring guest memory ( unlikely as it's 32G of 256G )
2) Downgrading the host to 13.2-RELEASE

FWIW we recently did a similar exercise and saw significant performance differences on ZFS backed disk images when comparing 14.0 and 13.2. We didn’t have time to root cause the difference, so it could simply be some tuning difference needed for 14.

Hi Chuck,

That's very helpful feedback. I'll start by downgrading the host to 13.2 and report back here.


Unfortunately same story with 13.2. I'm going to try and downgrade to 12.4 and see if it gets any better.

================================================================================
Begin @ Sat Feb 17 11:00:01 CST 2024

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99  1.3g  68 +++++ +++
Latency             11759us   29114us    8098us    8649us   23413us    4540us
Version  2.00       ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency              7791us     131us    1671us     464us      15us    1811us

End @ Sat Feb 17 11:03:13 CST 2024
================================================================================
Begin @ Sat Feb 17 11:10:01 CST 2024

Version  2.00       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99  398m  99 16204 563
Latency             12147us    1079us   24470us    8795us   17853us    4384us
Version  2.00       ------Sequential Create------ --------Random Create--------
localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16     0  93 +++++ +++ +++++ +++     0  96 +++++ +++ +++++ +++
Latency               118us     159us     269us     164us      54us     844us

End @ Sat Feb 17 11:18:43 CST 2024

I wasn't able to get a working 12.4 system in place due to lack of packages. However, I did fire up a FreeBSD 14 VM and let it run overnight on the same SSD array. It consistently ran at a much higher speed for 50+ runs @ 10m intervals ...

================================================================================
Begin @ Sun Feb 18 15:00:00 UTC 2024

Version  1.98       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
freebsd.shrew.l 64G  628k  99  1.6g  98  831m  60 1278k  99  1.1g  42 +++++ +++
Latency             13447us   68490us     207ms    7187us     195ms   17665us
Version  1.98       ------Sequential Create------ --------Random Create--------
freebsd.shrew.lab   -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
Latency             18225us      18us      28us   18812us      18us      25us

End @ Sun Feb 18 15:03:11 UTC 2024

I used identical options to run both that VM and the Linux VM I've been testing. The backing store for each VM has a 1TB partition and the guest interfaces are NVME. Now I'm really scratching my head.

Chuck, were you testing disk performance in Linux VMs or only FreeBSD?

Anyone have ideas on why Linux disk performance would drop off a cliff over time?

Thanks,

Thanks,

-Matthew

--------------aSmCDs7yhrXrJYYlTx0uyb8U--