From owner-freebsd-virtualization@freebsd.org Tue Aug 28 14:50:41 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B30CA108EB9D for ; Tue, 28 Aug 2018 14:50:41 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from mail.physik.tu-berlin.de (mail.physik-pool.tu-berlin.de [130.149.50.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 536CC875E8; Tue, 28 Aug 2018 14:50:40 +0000 (UTC) (envelope-from fabian.freyer@physik.tu-berlin.de) Received: from [192.168.0.114] (unknown [130.149.50.197]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.physik.tu-berlin.de (Postfix) with ESMTPSA id 9A5EA61FA3; Tue, 28 Aug 2018 14:50:32 +0000 (UTC) Subject: Re: on bhyve statistics To: "Rodney W. Grimes" , Anish Cc: "freebsd-virtualization@freebsd.org" , Marcelo Araujo References: <201808281337.w7SDbAEe015573@pdx.rh.CN85.dnsmgr.net> From: Fabian Freyer Message-ID: <79435a58-9591-b55a-adca-81e037e904ff@physik.tu-berlin.de> Date: Tue, 28 Aug 2018 16:50:17 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.0 MIME-Version: 1.0 In-Reply-To: <201808281337.w7SDbAEe015573@pdx.rh.CN85.dnsmgr.net> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Aug 2018 14:50:41 -0000 On 8/28/18 3:37 PM, Rodney W. Grimes wrote: >>>> Currently, bhyve does not expose any of these statistics. All the stats >>> available through bhyvectl --get-stats seem to be coming from the VMM, >>> not from the userspace emulation. >> >>> That is correct, byhvectl is a diagnostics tool for getting >> information from the kernel/vmm module. >> >> bhyvectl provide stats related to processor vmx/svm from vmm.ko and is the >> first thing you want to run for performance regression. It will be nice to >> include it as part of bhyve perf tool/dashboard that you are intended to >> build. > > From conversations with Peter Grehan he expressed that bhyvectl is > purely a diagnostics tool that should not be depended on by any > other tools. > > If you want to do similiar things you should program to the libvmmapi > interface, not bhyvectl. For context, this is *not* what I'm aiming to do. While bhyvectl just prints statistics exposed by the VMM through libvmmapi, I'm specifically asking about instrumenting the userland part, bhyve. Also, libvmmapi does have downstream consumers that are not bhyve{,ctl,load}, e.g. grub2-bhyve, bhyve-multiboot[1], maybe even xhyve[2] - for performance reasons, I'd prefer scraping metrics through libvmmapi over shelling out to bhyvectl. [1] https://github.com/fubarnetes/bhyve-multiboot [2] https://github.com/mist64/xhyve