Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 22 Feb 2024 23:51:06 -0600
From:      Matthew Grooms <mgrooms@shrew.net>
To:        virtualization@freebsd.org
Subject:   Re: bhyve disk performance issue
Message-ID:  <6a4e7e1d-cca5-45d4-a268-1805a15d9819@shrew.net>
In-Reply-To: <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net>
References:  <6a128904-a4c1-41ec-a83d-56da56871ceb@shrew.net> <28ea168c-1211-4104-b8b4-daed0e60950d@app.fastmail.com> <0ff6f30a-b53a-4d0f-ac21-eaf701d35d00@shrew.net> <6f6b71ac-2349-4045-9eaf-5c50d42b89be@shrew.net> <50614ea4-f0f9-44a2-b5e6-ebb33cfffbc4@shrew.net>

next in thread | previous in thread | raw e-mail | index | archive | help
On 2/18/24 09:21, Matthew Grooms wrote:
> On 2/17/24 15:53, Matthew Grooms wrote:
>>
>> Unfortunately same story with 13.2. I'm going to try and downgrade to 
>> 12.4 and see if it gets any better.
>>
>> ================================================================================
>> Begin @ Sat Feb 17 11:00:01 CST 2024
>>
>> Version  2.00       ------Sequential Output------ --Sequential Input- 
>> --Random-
>>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
>> --Seeks--
>> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  
>> /sec %CP
>> localhost.lo 63640M  690k  99  1.5g  97  727m  78  950k  99 1.3g  68 
>> +++++ +++
>> Latency             11759us   29114us    8098us    8649us 23413us    
>> 4540us
>> Version  2.00       ------Sequential Create------ --------Random 
>> Create--------
>> localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- 
>> -Delete--
>>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  
>> /sec %CP
>>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ 
>> +++++ +++
>> Latency              7791us     131us    1671us     464us 15us    1811us
>>
>> End @ Sat Feb 17 11:03:13 CST 2024
>> ================================================================================
>> Begin @ Sat Feb 17 11:10:01 CST 2024
>>
>> Version  2.00       ------Sequential Output------ --Sequential Input- 
>> --Random-
>>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
>> --Seeks--
>> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  
>> /sec %CP
>> localhost.lo 63640M  667k  99  449m  99  313m  94  940k  99 398m  99 
>> 16204 563
>> Latency             12147us    1079us   24470us    8795us 17853us    
>> 4384us
>> Version  2.00       ------Sequential Create------ --------Random 
>> Create--------
>> localhost.localdoma -Create-- --Read--- -Delete-- -Create-- --Read--- 
>> -Delete--
>>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  
>> /sec %CP
>>                  16     0  93 +++++ +++ +++++ +++     0  96 +++++ +++ 
>> +++++ +++
>> Latency               118us     159us     269us     164us 54us     844us
>>
>> End @ Sat Feb 17 11:18:43 CST 2024
>>
> I wasn't able to get a working 12.4 system in place due to lack of 
> packages. However, I did fire up a FreeBSD 14 VM and let it run 
> overnight on the same SSD array. It consistently ran at a much higher 
> speed for 50+ runs @ 10m intervals ...
>
> ================================================================================
> Begin @ Sun Feb 18 15:00:00 UTC 2024
>
> Version  1.98       ------Sequential Output------ --Sequential Input- 
> --Random-
>                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
> --Seeks--
> Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  
> /sec %CP
> freebsd.shrew.l 64G  628k  99  1.6g  98  831m  60 1278k  99 1.1g  42 
> +++++ +++
> Latency             13447us   68490us     207ms    7187us 195ms   17665us
> Version  1.98       ------Sequential Create------ --------Random 
> Create--------
> freebsd.shrew.lab   -Create-- --Read--- -Delete-- -Create-- --Read--- 
> -Delete--
>               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP  
> /sec %CP
>                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ 
> +++++ +++
> Latency             18225us      18us      28us   18812us 18us      25us
>
> End @ Sun Feb 18 15:03:11 UTC 2024
>
> I used identical options to run both that VM and the Linux VM I've 
> been testing. The backing store for each VM has a 1TB partition and 
> the guest interfaces are NVME. Now I'm really scratching my head.
>
> Chuck, were you testing disk performance in Linux VMs or only FreeBSD?
>
> Anyone have ideas on why Linux disk performance would drop off a cliff 
> over time?
>
I took a detour trying out Xen but apparently that has it's own set of 
performance issues related to the FreeBSD port missing newer features. I 
did install KVM on the same hardware for comparison. I then tested a 
guest provisioned identically to the bhyve VM with a virtio blk device 
which ran for 2.5 hours with consistently impressive output ...

================================================================================
Begin @ Thu Feb 22 20:55:01 CST 2024

Version  2.00       ------Sequential Output------ --Sequential Input- 
--Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
linux2.shrew.pr 31G 2191k  99  1.4g  55  1.1g  59 3484k  99  2.5g 83  
7817 127
Latency              4480us    2528us   17656us    2650us 102us    3568us
Version  2.00       ------Sequential Create------ --------Random 
Create--------
linux2.shrew.prv    -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ 
+++++ +++
Latency              9722us      90us     123us      61us 20us     998us

End @ Thu Feb 22 20:56:11 CST 2024
================================================================================


For comparison, here is the output from a recent run of a Linux VM on 
bhyve using the virtio blk device. Note the performance drop off between 
the first and second run ...


================================================================================
Begin @ Thu Feb 22 22:00:02 CST 2024

Version  2.00       ------Sequential Output------ --Sequential Input- 
--Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
linux.shrew. 63640M  694k  99  1.5g  97  772m  70  985k  99  1.4g 75  
2302 115
Latency             11557us   28959us   27540us    8308us 25227us    8605us
Version  2.00       ------Sequential Create------ --------Random 
Create--------
linux.shrew.lab     -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                  16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ 
+++++ +++
Latency              4058us     125us    1651us      96us 23us     627us

End @ Thu Feb 22 22:03:07 CST 2024
================================================================================
Begin @ Thu Feb 22 22:10:02 CST 2024

Version  2.00       ------Sequential Output------ --Sequential Input- 
--Random-
                     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- 
--Seeks--
Name:Size etc        /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
linux.shrew. 63640M  676k  99  406m  99  300m  92  952k  99  373m 99  
2145 158
Latency             11871us     154us   20673us    9926us 22765us   14034us
Version  2.00       ------Sequential Create------ --------Random 
Create--------
linux.shrew.lab     -Create-- --Read--- -Delete-- -Create-- --Read--- 
-Delete--
               files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  
/sec %CP
                  16     0  95 +++++ +++     0  79     0  96 +++++ 
+++     0  76
Latency              4426us     125us    1687us     576us 72us     654us

End @ Thu Feb 22 22:19:19 CST 2024
================================================================================

It certainly feels like a deficiency in bhyve that isn't tied to any 
particular storage device model. I did notice a pattern in top that I 
thought was peculiar. When watching the bhyve threads while the 
benchmark test is running, I see several CPU threads running at the top 
of the list followed by what I assume to be block device threads. When 
the disk performance is high, it looks like this ...

   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
  5628 root         68    0    32G    31G CPU24   24   0:53  88.90% 
bhyve{vcpu 5}
  5628 root         36    0    32G    31G vmidle   7   0:18  17.86% 
bhyve{vcpu 15}
  5628 root         34    0    32G    31G vmidle  26   1:06  16.76% 
bhyve{vcpu 8}
  5628 root         21    0    32G    31G uwait   37   0:05   2.69% 
bhyve{blk-4:0-2}
  5628 root         21    0    32G    31G uwait   60   0:04   2.64% 
bhyve{blk-4:0-0}
  5628 root         21    0    32G    31G uwait   52   0:06   2.62% 
bhyve{blk-4:0-1}
  5628 root         21    0    32G    31G uwait   14   0:05   2.58% 
bhyve{blk-4:0-6}
  5628 root         21    0    32G    31G RUN     50   0:05   2.51% 
bhyve{blk-4:0-4}
  5628 root         20    0    32G    31G uwait   38   0:05   2.51% 
bhyve{blk-4:0-5}
  5628 root         21    0    32G    31G uwait   56   0:05   2.46% 
bhyve{blk-4:0-3}
  5628 root         20    0    32G    31G uwait   22   0:06   2.45% 
bhyve{blk-4:0-7}

When disk performance drops off, I see that one of the bhyve CPU threads 
shows it's PRI value climb quickly until it hits around 135. At that 
point, the one CPU thread basically is pegged at 100% WCPU until the 
test is done. Other bhyve threads are much less busy ...

   PID USERNAME    PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
  5518 root        135    0    32G    31G CPU59   59   6:49  99.97% 
bhyve{vcpu 2}
  5518 root         26    0    32G    31G vmidle  36   5:40   8.80% 
bhyve{vcpu 15}
  5518 root         23    0    32G    31G vmidle  18   0:41   4.74% 
bhyve{vcpu 13}
  5518 root         20    0    32G    31G vmidle  37   0:09   0.85% 
bhyve{vcpu 10}
  5518 root         20    0    32G    31G uwait    6   0:20   0.72% 
bhyve{blk-4:0-2}
  5518 root         20    0    32G    31G uwait    8   0:20   0.71% 
bhyve{blk-4:0-7}
  5518 root         20    0    32G    31G uwait   43   0:22   0.70% 
bhyve{blk-4:0-0}
  5518 root         20    0    32G    31G uwait   12   0:20   0.70% 
bhyve{blk-4:0-5}
  5518 root         20    0    32G    31G uwait   43   0:19   0.68% 
bhyve{blk-4:0-3}
  5518 root         20    0    32G    31G uwait   36   0:21   0.68% 
bhyve{blk-4:0-6}
  5518 root         20    0    32G    31G uwait   46   0:20   0.68% 
bhyve{blk-4:0-4}
  5518 root         20    0    32G    31G uwait   32   0:20   0.64% 
bhyve{blk-4:0-1}

This pattern in top repeats each time the benchmark is run unless the 
guest is rebooted.

Does bhyve call pthread_attr_setschedparam at run time under certain 
circumstances or is that the scheduler doing it's thing? Anyone have any 
ideas? I'm pretty much out of them :/

Thanks in advance,

-Matthew




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?6a4e7e1d-cca5-45d4-a268-1805a15d9819>