Date: Mon, 29 Oct 2018 20:09:59 -0500 From: Dustin Marquess <dmarquess@gmail.com> To: freebsd@omnilan.de Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: bhyve win-guest benchmark comparing Message-ID: <CAJpsHY5gi85QTjOajWEgxrecBLiw1xzJ66r0REiFtPJH3pb=DQ@mail.gmail.com> In-Reply-To: <f942bab7-947a-d172-26bd-e0948205a643@omnilan.de> References: <9e7f4c01-6cd1-4045-1a5b-69c804b3881b@omnilan.de> <f942bab7-947a-d172-26bd-e0948205a643@omnilan.de>
next in thread | previous in thread | raw e-mail | index | archive | help
It would be interesting to test running it under Xen with FreeBSD as the dom0. -Dustin On Sat, Oct 27, 2018 at 1:04 PM Harry Schmalzbauer <freebsd@omnilan.de> wrote: > Am 22.10.2018 um 13:26 schrieb Harry Schmalzbauer: > =E2=80=A6 > > Test-Runs: > > Each hypervisor had only the one bench-guest running, no other > > tasks/guests were running besides system's native standard processes. > > Since the time between powering up the guest and finishing logon > > differed notably (~5s vs. ~20s) from one host to the other, I did a > > quick synthetic IO-Test beforehand. > > I'm using IOmeter since heise.de published a great test pattern called > > IOmix =E2=80=93 about 18 years ago I guess. This access pattern has al= ways > > perfectly reflected the system performance for human computer usage > > with non-caculation-centric applications, and still is my favourite, > > despite throughput and latency changed by some orders of manitudes > > during the last decade (and I had defined something for "fio" which > > mimics IOmix and shows reasonable relational results; but I'm still > > prefering IOmeter for homogenous IO benchmarking). > > > > The results is about factor 7 :-( > > ~3800iops&69MB/s (CPU-guest-usage 42%IOmeter+12%irq) > > vs. > > ~29000iops&530MB/s (CPU-guest-usage 11%IOmeter+19%irq) > > > > > > [with debug kernel and debug-malloc, numbers are 3000iops&56MB/s, > > virtio-blk instead of ahci,hd: results in 5660iops&104MB/s with > > non-debug kernel > > =E2=80=93 much better, but even higher CPU load and still factor 4= slower] > > > > What I don't understand is, why the IOmeter process differs that much > > in CPU utilization!?! It's the same binary on the same OS (guest) > > with the same OS-driver and the same underlying hardware =E2=80=93 "jus= t" the > > AHCI emulation and the vmm differ... > > > > Unfortunately, the picture for virtio-net vs. vmxnet3 is similar sad. > > Copying a single 5GB file from CIFS share to DB-ssd results in 100% > > guest-CPU usage, where 40% are irqs and the throughput max out at > > ~40MB/s. > > When copying the same file from the same source with the same guest on > > the same host but host booted ESXi, there's 20% guest-CPU usage while > > transfering 111MB/s =E2=80=93 the uplink GbE limit. > > > > These synthetic benchmark very well explain the "feelable" difference > > when using a guest between the two hypervisors, but > =E2=80=A6 > > To add an additional and rather surprinsing result, at least for me: > > Virtualbox provides > 'VBoxManage internalcommands createrawvmdk -filename > "testbench_da0.vmdk" -rawdisk /dev/da0' > > So I could use the exactly same test setup as for ESXi and bhyve. > FreeBSD-Virtualbox (running on the same host installation like bhyve) > performed quiet well, although it doesn't survive IOmix benchmark run > when the "testbench_da0.vmdk" (the "raw" SSD-R0-array) is hooked up to > the SATA controller. > But connected to the emulated SAS controller(LSI1068), it runs without > problems and results in 9600iops@185MB/s with 1%IOmeter+7%irq CPU > utilization (yes, 1% vs. 42% for IOmeter load). > Still far away from what ESXi provides, but almost double performance of > virtio-blk with bhyve, and most important, much less load (host and > guest show exactly the same low values as opposed to the very high loads > which are shown on host and guest with bhyve:virtio-blk). > The HDtune random access benchmark also shows the factor 2, linear over > all block sizes. > > Virtualbox's virtio-net setup gives ~100MB/s with peaks at 111 and ~40% > CPU load. > Guest uses the same driver like with bhyve:virtio-blk, while backend of > virtualbox:virtio-net is vboxnetflt utilizing netgraph and vboxnetadp.ko > vs. tap(4). > So not only the IO efficiency (lower throughput but also much lower CPU > utilization) is remarbably better, but also the network performance. > Even low-bandwidth RDP sessions via GbE-LAN suffer from micro hangs > under bhyve and virtio-net. And 40MB/s transfers cause 100% CPU load on > bhyve =E2=80=93 both runs had exactly the same WIndows virtio-net driver = in use > (RedHat 141). > > Conclusion: Virtualbox vs. ESXi shows a 0.5% efficiency factor, while > bhyve vs. ESXi shows 0.25% overall efficiency factor. > I tried to provide a test environment with shortest hardware paths > possible. At least the benchmarks were run 100% reproducable with the > same binaries. > > So I'm really interested if > =E2=80=A6 > > Are these (emulation(only?) related, I guess) performace issues well > > known? I mean, does somebody know what needs to be done in what area, > > in order to catch up with the other results? So it's just a matter of > > time/resources? > > Or are these results surprising and extensive analysis must be done > > before anybody can tell how to fix the IO limitations? > > > > Is the root cause for the problematic low virtio-net throughput > > probably the same as for the disk IO limits? Both really hurt in my > > use case and the host is not idling in relation, but even showing > > higher load with lower results. So even if the lower > > user-experience-performance would be considered as toleratable, the > > guests/host ratio was only half dense. > > Thanks, > > -harry > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJpsHY5gi85QTjOajWEgxrecBLiw1xzJ66r0REiFtPJH3pb=DQ>