From owner-freebsd-virtualization@freebsd.org Sun Apr 19 15:49:39 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0E46A279644 for ; Sun, 19 Apr 2020 15:49:39 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 494vQn53NXz4V8J for ; Sun, 19 Apr 2020 15:49:37 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id 03JFnTYY010178 for ; Sun, 19 Apr 2020 17:49:29 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 68DA04FE for ; Sun, 19 Apr 2020 17:49:29 +0200 (CEST) Subject: Re: bhyve win-guest benchmark comparing From: Harry Schmalzbauer To: freebsd-virtualization@freebsd.org References: <9e7f4c01-6cd1-4045-1a5b-69c804b3881b@omnilan.de> Organization: OmniLAN Message-ID: Date: Sun, 19 Apr 2020 17:49:28 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <9e7f4c01-6cd1-4045-1a5b-69c804b3881b@omnilan.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Greylist: ACL 136 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Sun, 19 Apr 2020 17:49:29 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) X-Rspamd-Queue-Id: 494vQn53NXz4V8J X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of freebsd@omnilan.de designates 2a00:e10:2800::a130 as permitted sender) smtp.mailfrom=freebsd@omnilan.de X-Spamd-Result: default: False [-4.70 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-virtualization@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; DMARC_NA(0.00)[omnilan.de]; IP_SCORE(-2.40)[ip: (-9.22), ipnet: 2a00:e10:2800::/38(-4.58), asn: 61157(1.83), country: DE(-0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:61157, ipnet:2a00:e10:2800::/38, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Apr 2020 15:49:39 -0000 Am 22.10.2018 um 13:26 schrieb Harry Schmalzbauer: … > > Test-Runs: > Each hypervisor had only the one bench-guest running, no other > tasks/guests were running besides system's native standard processes. > Since the time between powering up the guest and finishing logon > differed notably (~5s vs. ~20s) from one host to the other, I did a > quick synthetic IO-Test beforehand. > I'm using IOmeter since heise.de published a great test pattern called > IOmix – about 18 years ago I guess.  This access pattern has always > perfectly reflected the system performance for human computer usage > with non-caculation-centric applications, and still is my favourite, > despite throughput and latency changed by some orders of manitudes > during the last decade (and I had defined something for "fio" which > mimics IOmix and shows reasonable relational results; but I'm still > prefering IOmeter for homogenous IO benchmarking). > > The results is about factor 7 :-( > ~3800iops&69MB/s (CPU-guest-usage 42%IOmeter+12%irq) >                 vs. > ~29000iops&530MB/s (CPU-guest-usage 11%IOmeter+19%irq) > > >     [with debug kernel and debug-malloc, numbers are 3000iops&56MB/s, >      virtio-blk instead of ahci,hd: results in 5660iops&104MB/s with > non-debug kernel >      – much better, but even higher CPU load and still factor 4 slower] > > What I don't understand is, why the IOmeter process differs that much > in CPU utilization!?!  It's the same binary on the same OS (guest) > with the same OS-driver and the same underlying hardware – "just" the > AHCI emulation and the vmm differ... I repeated this test with a slightly different device backend (Samsung 850pro SSD on mps(4) instead of mfid(4)). After applying r358848 to stable/12, the numbers changed dramatically.0 on the same haswell based Xeon E3 platform. With the single SSD, the IOmeter numbers for ESXi as host drop from ~29000iops&530MB/s to ~11000/205MB/s.     But the numbers for bhyve as host raise from ~3800iops&69MB/s to ~8800/160MB/s at the same time!!! So there's still a penalty of ~20% for ahci-bhyve vs. ahci-esx, but this is a enourmous improvement. Please don't skip the MFC for r358848! Thanks a lot for all the work! -harry