From owner-freebsd-virtualization@freebsd.org Wed Jul 25 03:31:04 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7953C103C8A0 for ; Wed, 25 Jul 2018 03:31:04 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 197218CEF4 for ; Wed, 25 Jul 2018 03:31:04 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: by mailman.ysv.freebsd.org (Postfix) id D25C9103C89F; Wed, 25 Jul 2018 03:31:03 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C15E7103C89E for ; Wed, 25 Jul 2018 03:31:03 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3097C8CEF1; Wed, 25 Jul 2018 03:31:03 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id w6P3V1uw037363 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 24 Jul 2018 20:31:01 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id w6P3V10c037362; Tue, 24 Jul 2018 20:31:01 -0700 (PDT) (envelope-from jmg) Date: Tue, 24 Jul 2018 20:31:01 -0700 From: John-Mark Gurney To: Alan Somers Cc: "freebsd-virtualization@freebsd.org" Subject: Re: Overcommitting CPUs with BHyve? Message-ID: <20180725033101.GU2884@funkthat.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.0-RELEASE-p7 amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Tue, 24 Jul 2018 20:31:01 -0700 (PDT) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2018 03:31:04 -0000 Alan Somers wrote this message on Tue, Jul 24, 2018 at 15:30 -0600: > What are people's experiences with overcommitting CPUs in BHyve? I have an > 8-core machine that often runs VMs totalling up to 5 allocated CPUs without > problems. But today I got greedy. I assigned 8 cores to one VM for a big > build job. Obviously, some of those were shared with the host. I also > assigned it 8GB of RAM (out of 16 total). Build performance fell through > the floor, even though the host was idle. Eventually I killed the build > and restarted it with a more modest 2 make jobs (but the VM still had 8 > cores). Performance improved. But eventually the system seemed to be > mostly hung, while I had a build job running on the host as well as in the > VM. I killed both build jobs, which resolved the hung processes. Then I > restarted the host's build alone, and my system completely hung, with > top(1) indicating that many processes were in the pfault state. > > So my questions are: > 1) Is it a known problem to overcommit CPUs with BHyve? Likely as someone else mentioned the spin lock problem... It's best if you can schedule ALL vCPUs at the same time, but obviously the more vCPUs the harder this becomes, and I don't believe that FreeBSD has a scheduler that allows you to do this. The late Benjamin Perrault (iirc) said that his limit was 7 vCPU's per CPU, I don't remember if that was core or threads (likely core).. But I also don't know his work load, or vCPUs per VM... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not."