From owner-freebsd-virtualization@freebsd.org Sat Jul 28 19:07:09 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39135105AC01 for ; Sat, 28 Jul 2018 19:07:09 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CD6EC7B181 for ; Sat, 28 Jul 2018 19:07:08 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: by mailman.ysv.freebsd.org (Postfix) id 91C26105ABFF; Sat, 28 Jul 2018 19:07:08 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8045A105ABFE; Sat, 28 Jul 2018 19:07:08 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: from hel.cgi.cz (hel.cgi.cz [178.238.36.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 15A6B7B17F; Sat, 28 Jul 2018 19:07:07 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: from hel.cgi.cz (localhost [127.0.0.1]) by hel.cgi.cz (Postfix) with ESMTP id D15CB1292C8; Sat, 28 Jul 2018 21:06:55 +0200 (CEST) X-Virus-Scanned: amavisd-new at cgi.cz Received: from hel.cgi.cz ([127.0.0.1]) by hel.cgi.cz (hel.cgi.cz [127.0.0.1]) (amavisd-new, port 10024) with LMTP id GVTuqpIfRA8U; Sat, 28 Jul 2018 21:06:54 +0200 (CEST) Received: from mail2.cgi.cz (hermes [172.17.174.1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by hel.cgi.cz (Postfix) with ESMTPS id C3B211292B2; Sat, 28 Jul 2018 21:06:53 +0200 (CEST) Received: from webmail2.cgi.cz (mysql-postfix [172.17.174.1]) by mail2.cgi.cz (Postfix) with ESMTPA id A3C7E54E4A; Sat, 28 Jul 2018 21:06:53 +0200 (CEST) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 28 Jul 2018 21:06:53 +0200 From: Jakub Chromy To: "freebsd-virtualization@freebsd.org" Cc: owner-freebsd-virtualization@freebsd.org Subject: Re: Overcommitting CPUs with BHyve? In-Reply-To: References: Message-ID: <5fa93e55edf1a2f657f92e9bd05fbfc6@cgi.cz> X-Sender: hicks@cgi.cz User-Agent: Roundcube Webmail/1.3.5 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Jul 2018 19:07:09 -0000 > Yes it's a problem, and it's not just BHyve. The problem comes from > stuff > like spinlocks. Unlike normal userland locks, when two CPUs contend on > a > spinlock both are running at the same time. When two vCPUs are > contending > on a spinlock, the host has no idea how to prioritize them. Normally > that's not a problem, because physical CPUs are always supposed to be > able > to run. But when you overcommit vCPUs, some of them must get swapped > out > at all times. If a spinlock is being contended by both a running vCPU > and > a swapped out vCPU, then it might be contended for a long time. The > host's > scheduler simply isn't able to fix that problem. The problem is even > worse > when you're using hyperthreading (which I am) because those eight > logical > cores are really only four physical cores, and spinning on a spinlock > doesn't generate enough pipeline stalls to cause a hyperthread switch. > So > it's probably best to stick with the n - 1 rule. Overcommitting is ok > if > all guests are single-cored because then they won't use spinlocks. But > my > guests aren't all single-cored. I've just checked the handbooks for KVM and although they do recommend to keep the vCPUs single-cored whenever possible, it says that up to 3:1 vCPU vs CPU ratio should not mean significant peformance penalty. We've just run out of the (physical) cores on one of our BHYVE hypervisors... 32 cores (including HT), 16 VMs... and now I need to get another machine. My question is -- does KVM (on Linux) handle the CPU overprovisioning with multiple-vCPU guest better than the current BHYVE implamentation? If yes, how can it be improved? Can we (co)sponsor the development somehow? :) Jakub -- regards www.cgi.cz