From owner-freebsd-virtualization@freebsd.org Sun Jul 22 10:15:01 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26B011046D3B for ; Sun, 22 Jul 2018 10:15:01 +0000 (UTC) (envelope-from kalle.carlbark+freebsd@kcbark.net) Received: from smtp.kcbark.net (static-212-247-41-80.cust.tele2.se [212.247.41.80]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B92B78FB89 for ; Sun, 22 Jul 2018 10:15:00 +0000 (UTC) (envelope-from kalle.carlbark+freebsd@kcbark.net) Received: from cloud.kcbark.net (imap.kcbark.net [172.28.10.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.kcbark.net (Postfix) with ESMTPSA id F228757E1E for ; Sun, 22 Jul 2018 12:14:50 +0200 (CEST) Date: Sun, 22 Jul 2018 12:14:50 +0200 From: Kalle Carlbark To: freebsd-virtualization@freebsd.org Subject: Status of HD Audio Emulation For bhyve? Message-ID: <20180722101450.lthlkrwopq6nvooq@cloud.kcbark.net> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: NeoMutt/20180716 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Jul 2018 10:15:01 -0000 Hi! I was trying out the https://reviews.freebsd.org/D12419 on a recent FreeBSD CURRENT (>= 336229) but didn’t get it to work using debian 9 as a guest. Mpg123 just hangs when trying to play an audio file. Also tried Ubuntu 18 but that just hangs at boot time at “Target sound card”. I also tried with FreeBSD as a guest and there mpg123 is erroring out with “dead channel”. I’ve compiled bhyve WITHOUT_CAPSICUM but that doesn’t seem to solve anything. Also built with DEBUG_HDA and that doesn’t output any errors what I can see. Sound works just fine on the host system. Any hints? Thanks! -- Kalle Carlbark From owner-freebsd-virtualization@freebsd.org Sun Jul 22 11:20:38 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CDF21048656 for ; Sun, 22 Jul 2018 11:20:38 +0000 (UTC) (envelope-from robbelics@gmail.com) Received: from mail-lf1-x12a.google.com (mail-lf1-x12a.google.com [IPv6:2a00:1450:4864:20::12a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BFD7E716B1 for ; Sun, 22 Jul 2018 11:20:37 +0000 (UTC) (envelope-from robbelics@gmail.com) Received: by mail-lf1-x12a.google.com with SMTP id m12-v6so4529378lfc.10 for ; Sun, 22 Jul 2018 04:20:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=2NYXCOeCGUzaWz5kRjYHnCjg1ZrHI9J/YGjiPe/iAf0=; b=mxlJecbch5jiCET2cPaLjWp0v9TGinj6NsDexKbtEzwAgmkD14pceb58ZhKuLXAqbo sZLp6AX2/v6E8EwUP/njRr48h/5voTeFTWI291oBUkfplgZnyYJUw1aGZgv3TbBtmN15 dwyw48jry37Ns8scex3QbMF6ncA7+DWyu2dtPuKF4rZ5GhJN+DC5oxbpWZ5HrTVnYsnw l4QiwSittzA+WJg3AxkxbP+iZDpE2p+3yHd3bw/hZ6awnexjhrjGdHv9/9hIQEWLp81G Yf0lVPJwsOUa5MD25DP8778ylRM2PlSfUHbYb51HdnASu2piiIr8MtDPCfBKpYDIPETA MRuw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=2NYXCOeCGUzaWz5kRjYHnCjg1ZrHI9J/YGjiPe/iAf0=; b=f5y3Baiuq6WFNccVGJo0uSxrWgcqw59+11s8bL57Ou4KUbkVgF20KYRw+3Wl1eEg9x yePxWDSTfCS/jx3Tagq5TBto/VA0dQf0ATFB1K+/VVedSGAJoLT1GGjHaqbsiOaQ2BHw 6WJ6sa8m+ac9YbHV8XmS7aTZKVzT/4dI2ihvsUJSb/DDzGNmyLyKQf+HIs+YFW84KkDJ VapapkOcF/wOuG9sfEMQNJJCo/rGqpvCOqlHkUv/k/YaR24itq906Fa6TiXWGAZWulfL PgP/Rvt2vc5ZHjWTG8lVRK1LdkVgjXRuf6SBIiLUY04bmBqZZefKMlzmm3/8rRB7c+91 lpIg== X-Gm-Message-State: AOUpUlGxOe+Z1ECUE/vBpMSKVSwYH5LJTn9bKZtiCpsma29DeTD3b1KE lo/y+6GnQ7ABgF0555Ao7+rAxdg2wgYB+AYQVKeQr5w= X-Google-Smtp-Source: AAOMgpdmOi03eN6DHMbE09EQwVM6165EYO8dgnIpDBbFudDKFkN43bSIaqq4TtDNohdJyuaLD/eu5/DeYy5veNQMFSs= X-Received: by 2002:a19:1749:: with SMTP id n70-v6mr5030458lfi.54.1532258436187; Sun, 22 Jul 2018 04:20:36 -0700 (PDT) MIME-Version: 1.0 From: Rob Belics Date: Sun, 22 Jul 2018 06:20:24 -0500 Message-ID: Subject: Status of HD Audio Emulation For bhyve? To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Jul 2018 11:20:38 -0000 I have had the same issues, too. From owner-freebsd-virtualization@freebsd.org Tue Jul 24 21:30:38 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C424C1056A38 for ; Tue, 24 Jul 2018 21:30:38 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5EF6680F5A for ; Tue, 24 Jul 2018 21:30:38 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 23EFC1056A37; Tue, 24 Jul 2018 21:30:38 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 01FEB1056A36 for ; Tue, 24 Jul 2018 21:30:38 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lf1-x132.google.com (mail-lf1-x132.google.com [IPv6:2a00:1450:4864:20::132]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6EC5B80F51 for ; Tue, 24 Jul 2018 21:30:34 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lf1-x132.google.com with SMTP id v22-v6so3976612lfe.8 for ; Tue, 24 Jul 2018 14:30:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:from:date:message-id:subject:to; bh=WIO6KopsNF7Zk71CvsXyZBdZoR6cECvzFXXfs1a4H1E=; b=AMxZdRwql6gvrV0rOpufDdVjzQhqTvfFz0DeqEitx+f1Mhu6GVy8zVT5Xu2Rw5E22a 5E7+HRpsYul2LgT5Y+Omp653DfKfZWxgatOySYfGiGM8mbUWPDUk/5/xd7fFQAH271sU EZodqQD3X1hbar9rubfUZvhJi8az+3+YJtENRDDY4/SgojDq0FdFPiwQfL8f0jBY5lW2 sXg3rPGurqmP2lrqF5/XI9PgdDv4T8UW0+Ke5EFU4WLUTo2p3Ff06jIwsExJuD+SHwjx MQit/fdZhlblMQYBZYglBRsNJAYw4U1Rqxrb8wIbmkBoM+bSBO4wkvG2LmfHs0dyBasp oHyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=WIO6KopsNF7Zk71CvsXyZBdZoR6cECvzFXXfs1a4H1E=; b=KRy5JwkYen5JQpkJx6LtK9aiVnzAIgr95Rv8T2noZtHLExrU5CIJpbxohQ0UQUU8WR 7PQlwXmrW+H6bZH7obsQy9gfnT3FrTyAeXKJNYj9b1Ba5/kKOw/JUHVMhlcc2WRAEpsv X+MO/JPaw9Pkx+Yglurj+uH1j+KUQ/mTcQjSMz/a9/NcYGlL8UIGHlJQLDIk0Fizne1Y sMByC0C5KyjNl6a6q6Nos8/fPhKYk7WSJ+p1+iZ33Cvkti2MFvVlY4TMjxrl3FguSMYa DLzOAoopenCvHTinh3RYOje2yTzYG/zcfB4yNGHKvTL9aTDIKofCrofQIo4+HZTUx8Nx ZrCw== X-Gm-Message-State: AOUpUlF1H5/qTQmtk1QNXnHgGLLQwXz+sQFM6gHEd2rgEnF8V3NmPIhF OxADBk5u/LB78N2e8nQN9Ft8E940c9fKWcI5EKQL6Q== X-Google-Smtp-Source: AAOMgpdtLcj1YdcCBM0KDznKJ+dBy91JTIz9jU1Vz8W0JHCwWe9gatleJI7nfHFWiB6BJm1I0VyoOQQDU8fpqCuy8Pk= X-Received: by 2002:a19:df43:: with SMTP id q3-v6mr10577491lfj.53.1532467832690; Tue, 24 Jul 2018 14:30:32 -0700 (PDT) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 2002:ab3:7851:0:0:0:0:0 with HTTP; Tue, 24 Jul 2018 14:30:32 -0700 (PDT) From: Alan Somers Date: Tue, 24 Jul 2018 15:30:32 -0600 X-Google-Sender-Auth: Bn3oST8hfNJHjKCT1UZWoRcPqdU Message-ID: Subject: Overcommitting CPUs with BHyve? To: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jul 2018 21:30:39 -0000 What are people's experiences with overcommitting CPUs in BHyve? I have an 8-core machine that often runs VMs totalling up to 5 allocated CPUs without problems. But today I got greedy. I assigned 8 cores to one VM for a big build job. Obviously, some of those were shared with the host. I also assigned it 8GB of RAM (out of 16 total). Build performance fell through the floor, even though the host was idle. Eventually I killed the build and restarted it with a more modest 2 make jobs (but the VM still had 8 cores). Performance improved. But eventually the system seemed to be mostly hung, while I had a build job running on the host as well as in the VM. I killed both build jobs, which resolved the hung processes. Then I restarted the host's build alone, and my system completely hung, with top(1) indicating that many processes were in the pfault state. So my questions are: 1) Is it a known problem to overcommit CPUs with BHyve? 2) Could this be related to the pfault hang, even though the guest was idle at the time? -Alan From owner-freebsd-virtualization@freebsd.org Tue Jul 24 22:13:16 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 693D51057A61 for ; Tue, 24 Jul 2018 22:13:16 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id F2BB5829DC for ; Tue, 24 Jul 2018 22:13:15 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id B75D91057A60; Tue, 24 Jul 2018 22:13:15 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 952DE1057A5F for ; Tue, 24 Jul 2018 22:13:15 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: from mail-ed1-x52b.google.com (mail-ed1-x52b.google.com [IPv6:2a00:1450:4864:20::52b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E90ED829D8 for ; Tue, 24 Jul 2018 22:13:14 +0000 (UTC) (envelope-from shawn.webb@hardenedbsd.org) Received: by mail-ed1-x52b.google.com with SMTP id s16-v6so5394143edq.12 for ; Tue, 24 Jul 2018 15:13:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd.org; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=LQjiMCU4zZr7SN8UZ3c30iqC8ER0LBZYn9UViCxFqMo=; b=aJjGbuOZJbzqD/RK8I7GOKZiiuXsAklfpGapWHeSi8rhzu0+45mDhcwKOrRmcAoVbl y8UbQc1yPGFmo72U0/FIA48JDZDQ5Sluk/GGQcTo9n/y1t8F8g0n2MELITplW1zdhbyc pXcqMdl4EigVoghDU30mBYEZGi6pUvi3s1a777CXnBPI6IeO157uGp3L0VOcfBbHXX3Y Jhs2XHCusZTAfXtww37mCdaPQoCcoeSbHnEn9HgX5mngZcyv+Gcsufvhi0PI6UplpI8C BNjAcwHxWE69gCa1D06H0gRbE6pu3Pq9B2UwBBldF2qQna+jLi2Odlin81PtoWiCZKio 4/hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=LQjiMCU4zZr7SN8UZ3c30iqC8ER0LBZYn9UViCxFqMo=; b=f9kySQu4LEBw8GDnbDbN6weoZRe45SKNmokycMtFHx7SDdjtWQQu6SVUvxLhuHSghO gIsut5w/BPdGEftWKE1XAZub+rC6PRzo24wdJuKBNsJC/HyxzHNQeVCL8minDSJtOKj9 4r5Yl2rN8RLxB6qW5SXYzVpfdb1e9QNvmfEACcS0rKL5b0pDEhmG63GnzOxf2SQMEheI FVlnw9ZChrAdx6ZRfOqfYSDfGviO3MgN9gybuqFdDCXb6Gmikx1We9cfefR0h5hDRiFo TuPozIvAMHsIIuL/JSuyCGc272YeCuP+6q1CYPiT7i59kRSpK/0fKCO5VpdoGlEwhYPU eNnA== X-Gm-Message-State: AOUpUlEv1LZW2ePD1hrUfz1stHs9QEXN8bhoKxVm9ezxs1KXrMBNOGuh yfrUd5PGKKWO0iVF+QLwGHbH/dFJ090eGw== X-Google-Smtp-Source: AAOMgpd8DXcqLzoDeb1ttFNBsM+Rd8IrrFIqC1CBHLKQoIVecx7upTJifJmJfzvB3gyIXEGSRIlXdA== X-Received: by 2002:a50:b4b0:: with SMTP id w45-v6mr20614702edd.254.1532470392541; Tue, 24 Jul 2018 15:13:12 -0700 (PDT) Received: from mutt-hbsd ([178.19.111.150]) by smtp.gmail.com with ESMTPSA id d31-v6sm5843497edd.73.2018.07.24.15.13.09 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 24 Jul 2018 15:13:10 -0700 (PDT) Date: Tue, 24 Jul 2018 18:12:30 -0400 From: Shawn Webb To: Alan Somers Cc: "freebsd-virtualization@freebsd.org" Subject: Re: Overcommitting CPUs with BHyve? Message-ID: <20180724221230.tcljojufbwdl5ywy@mutt-hbsd> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="fbaxk42dnpdw3af4" Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD mutt-hbsd 12.0-CURRENT FreeBSD 12.0-CURRENT X-PGP-Key: http://pgp.mit.edu/pks/lookup?op=vindex&search=0x6A84658F52456EEE User-Agent: NeoMutt/20180622 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jul 2018 22:13:16 -0000 --fbaxk42dnpdw3af4 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jul 24, 2018 at 03:30:32PM -0600, Alan Somers wrote: > What are people's experiences with overcommitting CPUs in BHyve? I have = an > 8-core machine that often runs VMs totalling up to 5 allocated CPUs witho= ut > problems. But today I got greedy. I assigned 8 cores to one VM for a big > build job. Obviously, some of those were shared with the host. I also > assigned it 8GB of RAM (out of 16 total). Build performance fell through > the floor, even though the host was idle. Eventually I killed the build > and restarted it with a more modest 2 make jobs (but the VM still had 8 > cores). Performance improved. But eventually the system seemed to be > mostly hung, while I had a build job running on the host as well as in the > VM. I killed both build jobs, which resolved the hung processes. Then I > restarted the host's build alone, and my system completely hung, with > top(1) indicating that many processes were in the pfault state. >=20 > So my questions are: > 1) Is it a known problem to overcommit CPUs with BHyve? > 2) Could this be related to the pfault hang, even though the guest was id= le > at the time? VMWare's ESXi uses a special scheduler to do what it does. I wonder if it would be worthwhile to investigate implementing a scheduler in FreeBSD that provides decent performance for virtualized workloads. Thanks, --=20 Shawn Webb Cofounder and Security Engineer HardenedBSD Tor-ified Signal: +1 443-546-8752 Tor+XMPP+OTR: lattera@is.a.hacker.sx GPG Key ID: 0x6A84658F52456EEE GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89 3D9E 6A84 658F 5245 6EEE --fbaxk42dnpdw3af4 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEKrq2ve9q9Ia+iT2eaoRlj1JFbu4FAltXpEoACgkQaoRlj1JF bu7rLw/8DnZEhK1tcH+AlRwd3WlXtAeoSnTM8OLW0fR6jOOIYf6cS/+cxL0ALqca Pod0YLDY+rVMYKCZVxS7F8T5eTbiOxYBuaSM1lWKFr0wXaIxJ/EHm3S/Rodoag9Y 1BMtmo65FAN9jBoBjCXT6db3f/KfLKvHKDWtZrS6GdpQRtZPmIIPsmSAbp0wcKlv 0Dw71lyY23krOKx7OHq2ICrxSr6ewrYdnIHx6hQHWxKcpBqYsMbsH5VrXpi4IfWr G9uWGb44jDGxbFcGsLUW1mAAIdlTzgpyItBzcAEX9vZpF/1OKI6RK1BSD6AI7cEp ZKuRggDWjGKwaVafH0wXhm85QEk/okhRnU+KXqfJ5snaw/RiKuIs4mdjE1Kvv+hA /xDctlA5xr06X80rRSySf26XPmg4rbDuAyoHmgJsPCKU1Fx3UunT+4c21lHI0drm 5V4Buf3R2u2lLG/7Dc6Guqh4fDDOg65cfI7ZgSM6/FDgd/KjTa6AM9YgA18E+XdQ VOIDSyyZO5rKwNR5MD6InLheCGi+wIHWaPpj+jb7qYJOJZ7NPMKK2oJgpKQIMms4 VyiJQ4ZXb8e255PiiWH0EctuPVcNpvgR2Hj+Pgu57a1G/OB4TnLCDbp3hWh/Gfu9 tNv33XtE2SV6FTQIjxJi/YfCLT5UiCCMCEE8145PijxYHSlqSdg= =vfCa -----END PGP SIGNATURE----- --fbaxk42dnpdw3af4-- From owner-freebsd-virtualization@freebsd.org Tue Jul 24 23:15:14 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8BD791058831 for ; Tue, 24 Jul 2018 23:15:14 +0000 (UTC) (envelope-from mail@osfux.nl) Received: from vm1982.osfux.nl (vm1982.vellance.net [79.99.187.212]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 059C284310; Tue, 24 Jul 2018 23:15:13 +0000 (UTC) (envelope-from mail@osfux.nl) Received: from vm1982.osfux.nl (localhost [127.0.0.1]) by vm1982.osfux.nl (Postfix) with ESMTP id 1C11D20256; Wed, 25 Jul 2018 01:15:03 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=osfux.nl; s=default; t=1532474103; bh=R10F8iMqxnpaMjlM2BsaTOeBZKSw0YSR+4HEnvyp5RQ=; h=Subject:To:References:Cc:From:Date:In-Reply-To; b=qXpiViEgCafsZRQXLxN5eBetPPIYin8HWl4lPbK36x78LjGzRgvyk9rl4bVE9+0Kt afKndHIQD0toJ3pNt7fxDMeSIxull2/bApMnZuIb04fCL0MITiMHyWpADO2gMCIMU1 ZAAkbAq/E2u3z/Rcc3E5zG/6dnLSjXtPn3t6d0gmvjdpFgAH/oiNB8wdn6syTBBe0E 2NomXCDs/Lz6CMBFqqCAYS+qy4MP/tUBPAJdhicJ1iS4e6/evzpd+N0r5IFLKCcCG7 QH/lsj7QwSDIbcAvGrLCZ36oooCpjpioaVq2avKzi7DKAA2IExc44vtfCLq+z81iZ6 SXuuVjHSWy7YNc7ZplZ03TYFLj8zTZjWP7EzidI6rMgRWpGEqTMeqdxADG+lYiGMXQ Qc7fonit9Gr2IBBnIhlxHsYqSDTPjq0EokA0zXO3IjOp05cZTpfOjyYgsP9PZLwZVy PLODZV6nEktFAqi4i6gpyVInM4HJ/TsxUOtq4x/xLomH8rIuv59VtUwUJpx97xQtGY ggoqO4+a38v3nWtyByVKrcBn0oOHSROOQUGxn/VgbRn6sY08wgNLu/6MvMVcnc1A6I D/BZJPcGtNAFc7CYJSKJv/O2JSZm5PiNa5M0V5ED+2SosADDOgG+VNK+FQfL8RerYC LMVOd2n/eEpkIBo+IIXoFM24= Received: from vm1982.osfux.nl (localhost [127.0.0.1]) by vm1982.osfux.nl (Postfix) with ESMTP id 339052014F; Wed, 25 Jul 2018 01:14:58 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=osfux.nl; s=default; t=1532474102; bh=R10F8iMqxnpaMjlM2BsaTOeBZKSw0YSR+4HEnvyp5RQ=; h=Subject:To:References:Cc:From:Date:In-Reply-To; b=PrO81GCLox2yPTbK2dV3Kpap1T+ISIsr2/grdf3pYhDckwZBoexo2URCsXQJuKzAj cMjYa9vbFjllCtT0d3z2KH/8nPJ4Q5z88xRaipibb4mvYcIRD16hR23shqwdnGocP0 F6t6541Hp2R9sxj54MixqlfmLHyuYZ+I5/hmZPE/XKjykihWaO5xfnmXODbBJROA5+ UkCIjNiJrtXIWh+K11OPgqTPs5C1S2zIRcnHphZADvMb57BTtUec/j/PXXQ7KMoOyl tSfN7LcFpR15dA58hqL/UmUJYwLIH7AuCWRLW8EGD3j73Vr4lJ6q2LsSgSQwjzC/sK AJs1HiGQiH7AR36wpGq0t73TtXjXyhoociufeqcBhPf7oGaxvO+KtQ4rZjbkC3B5Ga N6NOiGHTtcMckma3osK70AGU7rs6hVVwltOojDU6dx9s2UsFqbYPFMfXWyyOdDCiv3 j4Pi6tpKE9Gh3Y6ZiCzEvv8Exzf2j9KJZGgMIWT3y9pqD9Y64chZw9fl4hP0Owmykv Sy/9V87DHH6EIwdQMEMb8sbayOUHwfmE/CArF//0s2sCE6bSCBPQINf0f78SiERts+ OxHem3IvgYuVj/+18FBzwUmi7cJm6IGSjV4FjcOKuBkBufozw23GS/lFT2ZkZju29/ aj2GduKY4EnJjP8zgLNJ/GW0= X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on vm1982.vellance.net X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED shortcircuit=no autolearn=ham autolearn_force=no version=3.4.1 Received: from [192.168.9.78] (ip51ccb320.speed.planet.nl [81.204.179.32]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by vm1982.osfux.nl (Postfix) with ESMTPSA; Wed, 25 Jul 2018 01:14:58 +0200 (CEST) Subject: Re: Overcommitting CPUs with BHyve? To: freebsd-virtualization@freebsd.org References: Cc: asomers@freebsd.org From: Ruben Message-ID: Date: Wed, 25 Jul 2018 01:14:58 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Virus-Scanned: ClamAV using ClamSMTP X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jul 2018 23:15:14 -0000 Using bhyve in several setups. Only one that is overcommittet is a quadcore AMD cpu (A4-5000) with 16 gb ram that runs 14 vms (2 of them dualcore i think). Rock solid. On 07/24/2018 11:30 PM, Alan Somers wrote: > What are people's experiences with overcommitting CPUs in BHyve? From owner-freebsd-virtualization@freebsd.org Tue Jul 24 23:47:54 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED13F105922A for ; Tue, 24 Jul 2018 23:47:53 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 64250853BF for ; Tue, 24 Jul 2018 23:47:53 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 137851D021 for ; Tue, 24 Jul 2018 23:47:52 +0000 (UTC) Subject: Re: Overcommitting CPUs with BHyve? To: freebsd-virtualization@freebsd.org References: From: Allan Jude Openpgp: preference=signencrypt Autocrypt: addr=allanjude@freebsd.org; prefer-encrypt=mutual; keydata= xsFNBFVwZcYBEADwrZDH0xe0ZVjc9ORCc6PcBLwS/RTXA6NkvpD6ea02pZ8lPOVgteuuugFc D34LdDbiWr+479vfrKBh+Y38GL0oZ0/13j10tIlDMHSa5BU0y6ACtnhupFvVlQ57+XaJAb/q 7qkfSiuxVwQ3FY3PL3cl1RrIP5eGHLA9hu4eVbu+FOX/q/XVKz49HaeIaxzo2Q54572VzIo6 C28McX9m65UL5fXMUGJDDLCItLmehZlHsQQ+uBxvODLFpVV2lUgDR/0rDa0B9zHZX8jY8qQ7 ZdCSy7CwClXI054CkXZCaBzgxYh/CotdI8ezmaw7NLs5vWNTxaDEFXaFMQtMVhvqQBpHkfOD 7rjjOmFw00nJL4FuPE5Yut0CPyx8vLjVmNJSt/Y8WxxmhutsqJYFgYfWl/vaWkrFLur/Zcmz IklwLw35HLsCZytCN5A3rGKdRbQjD6QPXOTJu0JPrJF6t2xFkWAT7oxnSV0ELhl2g+JfMMz2 Z1PDmS3NRnyEdqEm7NoRGXJJ7bgxDbN+9SXTyOletqGNXj/bSrBvhvZ0RQrzdHAPwQUfVSU2 qBhQEi2apSZstgVNMan0GUPqCdbE2zpysg+zT7Yhvf9EUQbzPL4LpdK1llT9fZbrdMzEXvEF oSvwJFdV3sqKmZc7b+E3PuxK6GTsKqaukd/3Cj8aLHG1T1im1QARAQABzSJBbGxhbiBKdWRl IDxhbGxhbmp1ZGVAZnJlZWJzZC5vcmc+wsF/BBMBAgApBQJVcGXGAhsjBQkSzAMABwsJCAcD AgEGFQgCCQoLBBYCAwECHgECF4AACgkQGZU1PhKYC34Muw/+JOKpSfhhysWFYiRXynGRDe07 Z6pVsn7DzrPUMRNZfHu8Uujmmy3p2nx9FelIY9yjd2UKHhug+whM54MiIFs90eCRVa4XEsPR 4FFAm0DAWrrb7qhZFcE/GhHdRWpZ341WAElWf6Puj2devtRjfYbikvj5+1V1QmDbju7cEw5D mEET44pTuD2VMRJpu2yZZzkM0i+wKFuPxlhqreufA1VNkZXI/rIfkYWK+nkXd9Efw3YdCyCQ zUgTUCb88ttSqcyhik/li1CDbXBpkzDCKI6I/8fAb7jjOC9LAtrZJrdgONywcVFoyK9ZN7EN AVA+xvYCmuYhR/3zHWH1g4hAm1v1+gIsufhajhfo8/wY1SetlzPaYkSkVQLqD8T6zZyhf+AN bC7ci44UsiKGAplB3phAXrtSPUEqM86kbnHg3fSx37kWKUiYNOnx4AC2VXvEiKsOBlpyt3dw WQbOtOYM+vkfbBwDtoGOOPYAKxc4LOIt9r+J8aD+gTooi9Eo5tvphATf9WkCpl9+aaGbSixB tUpvQMRnSMqTqq4Z7DeiG6VMRQIjsXDSLJEUqcfhnLFo0Ko/RiaHd5xyAQ4DhQ9QpkyQjjNf /3f/dYG7JAtoD30txaQ5V8uHrz210/77DRRX+HJjEj6xCxWUGvQgvEZf5XXyxeePvqZ+zQyT DX61bYw6w6bOwU0EVXBlxgEQAMy7YVnCCLN4oAOBVLZ5nUbVPvpUhsdA94/0/P+uqCIh28Cz ar56OCX0X19N/nAWecxL4H32zFbIRyDB2V/MEh4p9Qvyu/j4i1r3Ex5GhOT2hnit43Ng46z5 29Es4TijrHJP4/l/rB2VOqMKBS7Cq8zk1cWqaI9XZ59imxDNjtLLPPM+zQ1yE3OAMb475QwN UgWxTMw8rkA7CEaqeIn4sqpTSD5C7kT1Bh26+rbgJDZ77D6Uv1LaCZZOaW52okW3bFbdozV8 yM2u+xz2Qs8bHz67p+s+BlygryiOyYytpkiK6Iy4N7FTolyj5EIwCuqzfk0SaRHeOKX2ZRjC qatkgoD/t13PNT38V9tw3qZVOJDS0W6WM8VSg+F+bkM9LgJ8CmKV+Hj0k3pfGfYPOZJ/v18i +SmZmL/Uw2RghnwDWGAsPCKu4uZR777iw7n9Io6Vfxndw2dcS0e9klvFYoaGS6H2F13Asygr WBzFNGFQscN4mUW+ZYBzpTOcHkdT7w8WS55BmXYLna+dYer9/HaAuUrONjujukN4SPS1fMJ2 /CS/idAUKyyVVX5vozoNK2JVC1h1zUAVsdnmhEzNPsvBoqcVNfyqBFROEVLIPwq+lQMGNVjH ekLTKRWf59MEhUC2ztjSKkGmwdg73d6xSXMuq45EgIJV2wPvOgWQonoHH/kxABEBAAHCwWUE GAECAA8FAlVwZcYCGwwFCRLMAwAACgkQGZU1PhKYC34w5A//YViBtZyDV5O+SJT9FFO3lb9x Zdxf0trA3ooCt7gdBkdnBM6T5EmjgVZ3KYYyFfwXZVkteuCCycMF/zVw5eE9FL1+zz9gg663 nY9q2F77TZTKXVWOLlOV2bY+xaK94U4ytogOGhh9b4UnQ/Ct3+6aviCF78Go608BXbmF/GVT 7uhddemk7ItxM1gE5Hscx3saxGKlayaOsdPKeGTVJCDEtHDuOc7/+jGh5Zxpk/Hpi+DUt1ot 8e6hPYLIQa4uVx4f1xxxV858PQ7QysSLr9pTV7FAQ18JclCaMc7JWIa3homZQL/MNKOfST0S 2e+msuRwQo7AnnfFKBUtb02KwpA4GhWryhkjUh/kbVc1wmGxaU3DgXYQ5GV5+Zf4kk/wqr/7 KG0dkTz6NLCVLyDlmAzuFhf66DJ3zzz4yIo3pbDYi3HB/BwJXVSKB3Ko0oUo+6/qMrOIS02L s++QE/z7K12CCcs7WwOjfCYHK7VtE0Sr/PfybBdTbuDncOuAyAIeIKxdI2nmQHzl035hhvQX s4CSghsP319jAOQiIolCeSbTMD4QWMK8RL/Pe1FI1jC3Nw9s+jq8Dudtbcj2UwAP/STUEbJ9 5rznzuuhPjE0e++EU/RpWmcaIMK/z1zZDMN+ce2v1qzgV936ZhJ3iaVzyqbEE81gDxg3P+IM kiYh4ZtPB4Q= Message-ID: Date: Tue, 24 Jul 2018 19:47:47 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="ufdwLLnHSBp9FFXIXB1kcmN0LElR0gzIA" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Jul 2018 23:47:54 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --ufdwLLnHSBp9FFXIXB1kcmN0LElR0gzIA Content-Type: multipart/mixed; boundary="LnBBkp24fKFRDj36YuniA37gh3QUFhj40"; protected-headers="v1" From: Allan Jude To: freebsd-virtualization@freebsd.org Message-ID: Subject: Re: Overcommitting CPUs with BHyve? References: In-Reply-To: --LnBBkp24fKFRDj36YuniA37gh3QUFhj40 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 2018-07-24 17:30, Alan Somers wrote: > What are people's experiences with overcommitting CPUs in BHyve? I hav= e an > 8-core machine that often runs VMs totalling up to 5 allocated CPUs wit= hout > problems. But today I got greedy. I assigned 8 cores to one VM for a = big > build job. Obviously, some of those were shared with the host. I also= > assigned it 8GB of RAM (out of 16 total). Build performance fell throu= gh > the floor, even though the host was idle. Eventually I killed the buil= d > and restarted it with a more modest 2 make jobs (but the VM still had 8= > cores). Performance improved. But eventually the system seemed to be > mostly hung, while I had a build job running on the host as well as in = the > VM. I killed both build jobs, which resolved the hung processes. Then= I > restarted the host's build alone, and my system completely hung, with > top(1) indicating that many processes were in the pfault state. >=20 > So my questions are: > 1) Is it a known problem to overcommit CPUs with BHyve? > 2) Could this be related to the pfault hang, even though the guest was = idle > at the time? >=20 > -Alan > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fr= eebsd.org" >=20 Bhyve has a command line flag, -p, to let you pin a vCPU to a physical CPU. This might avoid some of the issues with the threads hopping around all the time. If you were anyone else, I'd also ask if you ensured your vfs.zfs.arc_max was low enough to actually leave some ram for the VM to u= se. --=20 Allan Jude --LnBBkp24fKFRDj36YuniA37gh3QUFhj40-- --ufdwLLnHSBp9FFXIXB1kcmN0LElR0gzIA Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJbV7qnAAoJEBmVNT4SmAt+8fMQAJgOCZ4RdlDo0S6/La12v+Qm /TswUy8DFGtbNc0h6nDXUVPv4vjSFXUUS1mGV2dceRuqs567289EDYjK6DT0/jkd pPnhHDDWP5+kokLuLVbt34qp0vxia+5/CBcnxntFWSglH0bEzqN5wmqcpJYhUPuK 76k+soOdhvajUlBEE+UbE2cNZkFpYdgUo7WLGzh8116PJT2a+N0uIxHFbppwsSQb Vjh9M4rut/rI4gh0zxQyljbMFD/VW93AfwQ6KgE155GL+gVul2BNwX982AHlMxk5 vX1U5jUuLuatN9olEyhFhareVGM5mY8oAb8U6yUkLvn7+nN1FJcUCtsh/zsBIVAN boci1iH1IwsFCswlXk3NZbJqoON9M1z7woEXVqXQBtSsyPQ3hU1quymmTHM3Xb2d g1zBy3njmAZYUQ1DLxg0GV9j/X/1JVMOhZIR1NRyV0rFMdL3RGjH8QSXRlZrfdSR a8agfWLypqRK5dsxX2o7wx8GJa1iPRbXH6KSs5hQZpJ8izYizswRhv2inHWiWWvM B91md7GY1Olrk9B1LcyQ3ipOFvXTDKT6xrVYy2HNIPLtknSMXnab2mLnRIbqxiWo MVz0NMOIHPHVk58aaPnZikSlXViezSChJUX/HaYcuVE/0IORAoxDqIFuYdoo/+hm tWNPMUZOYzxkNnTLaIIz =9yAw -----END PGP SIGNATURE----- --ufdwLLnHSBp9FFXIXB1kcmN0LElR0gzIA-- From owner-freebsd-virtualization@freebsd.org Wed Jul 25 00:02:50 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E18C8105992F for ; Wed, 25 Jul 2018 00:02:49 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 67E3885C59 for ; Wed, 25 Jul 2018 00:02:49 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 251B1105992E; Wed, 25 Jul 2018 00:02:49 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DCF7A105992D for ; Wed, 25 Jul 2018 00:02:48 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-lj1-x234.google.com (mail-lj1-x234.google.com [IPv6:2a00:1450:4864:20::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3F0FF85C57; Wed, 25 Jul 2018 00:02:48 +0000 (UTC) (envelope-from asomers@gmail.com) Received: by mail-lj1-x234.google.com with SMTP id 203-v6so5081100ljj.13; Tue, 24 Jul 2018 17:02:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=nu8DJv13MxsHp5xvSMN5P3AgZIjDuL6a6x1LGD0oHrY=; b=OzYylUw/UfZdxCcsl1MWt2PDSWrW4SqkWmX7wVBs/N7hPUT0C62RGQAoNcKrfDdubM wJjsdoqB5hdJHMpf898hb2g9wdPb9xn+tBf/dUY/kSRMIShQ0KhG77TSFXgn1jk1pDjk IIJa4U3yzGJHUERHMRo75W6btiFhkmN+kxi27raCR/l2Qu3zqwh77sxlUNHbHZ/241Fn psHthLzxGyNBfSUb51fD57Tv8nTOK+h4hphI4meuoJ3QXFvQ/1u0I5zNe8RHGoamalMg 4UKWefutb3QZO/Aor1KUPje8+HRFAfz5F44RhwSrnABCpSc4hX81NFng5cyK7SoojqKj HUxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=nu8DJv13MxsHp5xvSMN5P3AgZIjDuL6a6x1LGD0oHrY=; b=W/6SXozl1foMe9GeJ2Smi4XyE+hHum+s1oeKWME8bL5Qqyz/jHMVCVTW2jnlVyrvD2 Q/na5jrmuIHC5X3BFwvHbM3DlWIML89aXUOB0CbHfygZSF4Ad6VgcJJhfa7UkDOLszCx EBAUsTFs63WxbEcs+Lfi5sp55eCSjuLazVe6xWRSz0haWINPlkhOU+fv8xwevdHNc0wb iKdB2idvwZm/v4vskBgWcksieG9ujudIsjv/0htnvKZ5hsfQhU2uz+t8fTlv75lbNyvs xmK4kETOnvTKE4eCcielO75Mo+yC7K2MN1rAaOSwprl6LhMctXLhs0JceaXJBPQq9jae UkJg== X-Gm-Message-State: AOUpUlFZymwluTUQ2wD4FTLkpR+smKQ0KYJ+Sc7cAB+i9MAmpOeDYh1p upEVC5d9RNwqww2tDPuM0XL6zfMLfsaFGRznLC/g98YA X-Google-Smtp-Source: AAOMgpde2OuRd6h7GaV+zVFHHyvsHY6W2TsZfBzCb2sgWombu2i8SzvUcCDts61q76EZn3UTK+WqZKnsjBXSbplB4rQ= X-Received: by 2002:a2e:1517:: with SMTP id s23-v6mr14104070ljd.73.1532476966562; Tue, 24 Jul 2018 17:02:46 -0700 (PDT) MIME-Version: 1.0 Sender: asomers@gmail.com Received: by 2002:ab3:7851:0:0:0:0:0 with HTTP; Tue, 24 Jul 2018 17:02:45 -0700 (PDT) In-Reply-To: References: From: Alan Somers Date: Tue, 24 Jul 2018 18:02:45 -0600 X-Google-Sender-Auth: WiTtqnsFS5kcoO2d0PDdiEKxfVE Message-ID: Subject: Re: Overcommitting CPUs with BHyve? To: Alan Somers Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2018 00:02:50 -0000 An anonymous BHyve expert has explained things to me off-list. Details below. On Tue, Jul 24, 2018 at 3:30 PM, Alan Somers wrote: > What are people's experiences with overcommitting CPUs in BHyve? I have > an 8-core machine that often runs VMs totalling up to 5 allocated CPUs > without problems. But today I got greedy. I assigned 8 cores to one VM > for a big build job. Obviously, some of those were shared with the host. > I also assigned it 8GB of RAM (out of 16 total). Build performance fell > through the floor, even though the host was idle. Eventually I killed the > build and restarted it with a more modest 2 make jobs (but the VM still had > 8 cores). Performance improved. But eventually the system seemed to be > mostly hung, while I had a build job running on the host as well as in the > VM. I killed both build jobs, which resolved the hung processes. Then I > restarted the host's build alone, and my system completely hung, with > top(1) indicating that many processes were in the pfault state. > > So my questions are: > 1) Is it a known problem to overcommit CPUs with BHyve? > Yes it's a problem, and it's not just BHyve. The problem comes from stuff like spinlocks. Unlike normal userland locks, when two CPUs contend on a spinlock both are running at the same time. When two vCPUs are contending on a spinlock, the host has no idea how to prioritize them. Normally that's not a problem, because physical CPUs are always supposed to be able to run. But when you overcommit vCPUs, some of them must get swapped out at all times. If a spinlock is being contended by both a running vCPU and a swapped out vCPU, then it might be contended for a long time. The host's scheduler simply isn't able to fix that problem. The problem is even worse when you're using hyperthreading (which I am) because those eight logical cores are really only four physical cores, and spinning on a spinlock doesn't generate enough pipeline stalls to cause a hyperthread switch. So it's probably best to stick with the n - 1 rule. Overcommitting is ok if all guests are single-cored because then they won't use spinlocks. But my guests aren't all single-cored. 2) Could this be related to the pfault hang, even though the guest was idle > at the time? > The expert suspects the ZFS ARC was competing with the guest for RAM. IIUC, ZFS will sometimes greedily grow its ARC by swapping out idle parts of the guest's RAM. But the guest isn't aware of this behavior, and will happily allocate memory from the swapped-out portion. The result is a battle between the ARC and the guest for physical RAM. The best solution is to limit the maximum amount of RAM used by the ARC with the vfs.zfs.arc_max sysctl. More info: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=222916 Thanks to everyone who commented, especially the Anonymous Coward. -Alan From owner-freebsd-virtualization@freebsd.org Wed Jul 25 00:14:34 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 490081059D07 for ; Wed, 25 Jul 2018 00:14:34 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DB629863BD for ; Wed, 25 Jul 2018 00:14:33 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id A00D21059D06; Wed, 25 Jul 2018 00:14:33 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7DFDE1059D05 for ; Wed, 25 Jul 2018 00:14:33 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: from mail-oi0-f65.google.com (mail-oi0-f65.google.com [209.85.218.65]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 21A70863BB; Wed, 25 Jul 2018 00:14:33 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: by mail-oi0-f65.google.com with SMTP id l10-v6so10804756oii.0; Tue, 24 Jul 2018 17:14:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZDed5HDT2HZ4AHtztTS7En1DcZrwiqU3zW/4o9MeGW8=; b=PBt2Q9j6hWxD0jqnUaXOeKanI2PpeouCytxY/6cZeuVdE9QmCitvHVf9SjwIpYrh2W Z/KN2BMfPKkrUcJD/Hb8/nDwXLtd/CQ+kshZUaYafJrVlVTw2tZAq+xSCg/H2sQgIHuI dRXoIybrb9Fb1fS2cJde5MzfblAKTMAf8+9YMZfP1wzQ0R76n9gulP9REIvW1doNTLcn wOLWvb1Eyam7pI0Wykmf7xDkrAyn3LKIKUG5N6eSbhM6ym4IvVTpUI00fNhHNB6eyiRY 8IV6xr3f7gmXkPhyv4tUX1XqoqLJxDc0oN6JA/Efn9Md3zxmi8DoUnua4i0++gZhRxw+ Yltg== X-Gm-Message-State: AOUpUlHpcmc2Ix8AVo21yu30VHqAbvihg1I8BcaNmJ4pBMZYYiwP0zt2 7pouj/PrKhynq1ivY755X3QoqwbVW6ttmh0LNxaO2A== X-Google-Smtp-Source: AAOMgpeqxd5ampMSAOeRqniy//lHF7ZgYSjm6jofwQlO/9fOLNhgMsAy/gd5duEb/M/zabSkFVy7oB5O497JO10Ivis= X-Received: by 2002:aca:6142:: with SMTP id v63-v6mr1006842oib.137.1532474363720; Tue, 24 Jul 2018 16:19:23 -0700 (PDT) MIME-Version: 1.0 References: <20180724221230.tcljojufbwdl5ywy@mutt-hbsd> In-Reply-To: <20180724221230.tcljojufbwdl5ywy@mutt-hbsd> From: Jason Tubnor Date: Wed, 25 Jul 2018 09:19:12 +1000 Message-ID: Subject: Re: Overcommitting CPUs with BHyve? To: Shawn Webb Cc: Alan Somers , "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2018 00:14:34 -0000 On Wed, 25 Jul 2018 at 08:12, Shawn Webb wrote: > On Tue, Jul 24, 2018 at 03:30:32PM -0600, Alan Somers wrote: > > What are people's experiences with overcommitting CPUs in BHyve? I have > an > > 8-core machine that often runs VMs totalling up to 5 allocated CPUs > without > > problems. But today I got greedy. I assigned 8 cores to one VM for a > big > > build job. Obviously, some of those were shared with the host. I also > > assigned it 8GB of RAM (out of 16 total). Build performance fell through > > the floor, even though the host was idle. Eventually I killed the build > > and restarted it with a more modest 2 make jobs (but the VM still had 8 > > cores). Performance improved. But eventually the system seemed to be > > mostly hung, while I had a build job running on the host as well as in > the > > VM. I killed both build jobs, which resolved the hung processes. Then I > > restarted the host's build alone, and my system completely hung, with > > top(1) indicating that many processes were in the pfault state. > > > > So my questions are: > > 1) Is it a known problem to overcommit CPUs with BHyve? > > 2) Could this be related to the pfault hang, even though the guest was > idle > > at the time? > 1) Not that I have experienced. 2) More likely RAM pressure. Are you running ZFS? What is you ARC capped at? (Total guest + System + ARC < System Total Ram) > VMWare's ESXi uses a special scheduler to do what it does. I wonder if > it would be worthwhile to investigate implementing a scheduler in > FreeBSD that provides decent performance for virtualized workloads. > > > From owner-freebsd-virtualization@freebsd.org Wed Jul 25 01:09:37 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 78F7D1035242 for ; Wed, 25 Jul 2018 01:09:37 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 0184E88E4B for ; Wed, 25 Jul 2018 01:09:37 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: by mailman.ysv.freebsd.org (Postfix) id B709B1035241; Wed, 25 Jul 2018 01:09:36 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3DA91035240 for ; Wed, 25 Jul 2018 01:09:36 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 00C2688E49; Wed, 25 Jul 2018 01:09:35 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w6P19QC9038768; Tue, 24 Jul 2018 18:09:26 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w6P19QVf038767; Tue, 24 Jul 2018 18:09:26 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201807250109.w6P19QVf038767@pdx.rh.CN85.dnsmgr.net> Subject: Re: Overcommitting CPUs with BHyve? In-Reply-To: To: Alan Somers Date: Tue, 24 Jul 2018 18:09:26 -0700 (PDT) CC: "freebsd-virtualization@freebsd.org" X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2018 01:09:37 -0000 > What are people's experiences with overcommitting CPUs in BHyve? I have an > 8-core machine that often runs VMs totalling up to 5 allocated CPUs without > problems. But today I got greedy. I assigned 8 cores to one VM for a big > build job. Obviously, some of those were shared with the host. I also > assigned it 8GB of RAM (out of 16 total). Build performance fell through > the floor, even though the host was idle. Eventually I killed the build > and restarted it with a more modest 2 make jobs (but the VM still had 8 > cores). Performance improved. But eventually the system seemed to be > mostly hung, while I had a build job running on the host as well as in the > VM. I killed both build jobs, which resolved the hung processes. Then I > restarted the host's build alone, and my system completely hung, with > top(1) indicating that many processes were in the pfault state. > > So my questions are: > 1) Is it a known problem to overcommit CPUs with BHyve? > 2) Could this be related to the pfault hang, even though the guest was idle > at the time? I on occasion do over commit vCPU's in bhyve, but I do so with a few specific conditions: 1) I count CPU's as real Cores, not Hyperthread cores, I do not expect hyperthreading to work well in over commit. 2) I always wire my VM's memory, I NEVER overcommit memory, that just leads to bad and ugly. (-S option to bhyveload and bhyve) This totally takes ARC issues out of the picture, but you may not be able to start your VM's if you dont decrease the ARC. 3) Watch out for host side disk drive IOP saturation, you can easily stahly your guests if your trying to do to much I/O, they usually recover from this on there own, though it can make things go pretty slow for a time. Firing off 16 VM's doing "nightly" on a single spindle host is a sure way to have some very long runs. My work load runs from an always running 6 vCPU light load, to an occasion guest running make -j4 buildworlds (total of 10 vCPU load). My host has 4 cores, 8 threads. The 10vCPU load usually drives the host to a load average of 5. The 6 vCPU always present light load very rarely drives the load above 1. I think the secret sauce is wired memory :-) -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Wed Jul 25 03:31:04 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7953C103C8A0 for ; Wed, 25 Jul 2018 03:31:04 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 197218CEF4 for ; Wed, 25 Jul 2018 03:31:04 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: by mailman.ysv.freebsd.org (Postfix) id D25C9103C89F; Wed, 25 Jul 2018 03:31:03 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C15E7103C89E for ; Wed, 25 Jul 2018 03:31:03 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3097C8CEF1; Wed, 25 Jul 2018 03:31:03 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id w6P3V1uw037363 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 24 Jul 2018 20:31:01 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id w6P3V10c037362; Tue, 24 Jul 2018 20:31:01 -0700 (PDT) (envelope-from jmg) Date: Tue, 24 Jul 2018 20:31:01 -0700 From: John-Mark Gurney To: Alan Somers Cc: "freebsd-virtualization@freebsd.org" Subject: Re: Overcommitting CPUs with BHyve? Message-ID: <20180725033101.GU2884@funkthat.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Operating-System: FreeBSD 11.0-RELEASE-p7 amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Tue, 24 Jul 2018 20:31:01 -0700 (PDT) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2018 03:31:04 -0000 Alan Somers wrote this message on Tue, Jul 24, 2018 at 15:30 -0600: > What are people's experiences with overcommitting CPUs in BHyve? I have an > 8-core machine that often runs VMs totalling up to 5 allocated CPUs without > problems. But today I got greedy. I assigned 8 cores to one VM for a big > build job. Obviously, some of those were shared with the host. I also > assigned it 8GB of RAM (out of 16 total). Build performance fell through > the floor, even though the host was idle. Eventually I killed the build > and restarted it with a more modest 2 make jobs (but the VM still had 8 > cores). Performance improved. But eventually the system seemed to be > mostly hung, while I had a build job running on the host as well as in the > VM. I killed both build jobs, which resolved the hung processes. Then I > restarted the host's build alone, and my system completely hung, with > top(1) indicating that many processes were in the pfault state. > > So my questions are: > 1) Is it a known problem to overcommit CPUs with BHyve? Likely as someone else mentioned the spin lock problem... It's best if you can schedule ALL vCPUs at the same time, but obviously the more vCPUs the harder this becomes, and I don't believe that FreeBSD has a scheduler that allows you to do this. The late Benjamin Perrault (iirc) said that his limit was 7 vCPU's per CPU, I don't remember if that was core or threads (likely core).. But I also don't know his work load, or vCPUs per VM... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-virtualization@freebsd.org Wed Jul 25 19:37:03 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A33D1052AE5 for ; Wed, 25 Jul 2018 19:37:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 16BEA8B6C4 for ; Wed, 25 Jul 2018 19:37:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id CC5C41052AE4; Wed, 25 Jul 2018 19:37:02 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BB19C1052AE3 for ; Wed, 25 Jul 2018 19:37:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B9B28B6C0 for ; Wed, 25 Jul 2018 19:37:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id A32DD26A8D for ; Wed, 25 Jul 2018 19:37:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w6PJb1YZ059313 for ; Wed, 25 Jul 2018 19:37:01 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w6PJb1jw059312 for virtualization@FreeBSD.org; Wed, 25 Jul 2018 19:37:01 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 218731] FREEBSD 11 SCSI drive issue with all 11.xxxx versions running under HYPER-V as a VM Date: Wed, 25 Jul 2018 19:37:01 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mx5arbc@hotmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2018 19:37:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D218731 --- Comment #18 from ds --- This bug seems to be fixed in version in XIGMANAS 11.2 that uses the 11.2-RELEASE Freebsd kernel. I can now see hard drives attached via SCCI virtual definition under HYPER-V in Windows 10 Professional. Thanks. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Jul 25 19:47:56 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3106A1052ED3 for ; Wed, 25 Jul 2018 19:47:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C263F8BDDE for ; Wed, 25 Jul 2018 19:47:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 832081052ED2; Wed, 25 Jul 2018 19:47:55 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71E671052ED1 for ; Wed, 25 Jul 2018 19:47:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 13B438BDDC for ; Wed, 25 Jul 2018 19:47:55 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5ACD426BEF for ; Wed, 25 Jul 2018 19:47:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w6PJlsX9078603 for ; Wed, 25 Jul 2018 19:47:54 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w6PJlskQ078602 for virtualization@FreeBSD.org; Wed, 25 Jul 2018 19:47:54 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 218731] FREEBSD 11 SCSI drive issue with all 11.xxxx versions running under HYPER-V as a VM Date: Wed, 25 Jul 2018 19:47:53 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: decui@microsoft.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 25 Jul 2018 19:47:56 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D218731 --- Comment #19 from Dexuan Cui --- (In reply to ds from comment #18) Since it works for you, can you please close the bug? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Jul 26 13:35:28 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1B0CA104F664 for ; Thu, 26 Jul 2018 13:35:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A89028F331 for ; Thu, 26 Jul 2018 13:35:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 6D191104F663; Thu, 26 Jul 2018 13:35:27 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59DAC104F662 for ; Thu, 26 Jul 2018 13:35:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EA0578F330 for ; Thu, 26 Jul 2018 13:35:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 2378F10294 for ; Thu, 26 Jul 2018 13:35:26 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w6QDZQbi058764 for ; Thu, 26 Jul 2018 13:35:26 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w6QDZPOE058763 for virtualization@FreeBSD.org; Thu, 26 Jul 2018 13:35:25 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 218731] FREEBSD 11 SCSI drive issue with all 11.xxxx versions running under HYPER-V as a VM Date: Thu, 26 Jul 2018 13:35:25 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.1-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mx5arbc@hotmail.com X-Bugzilla-Status: Closed X-Bugzilla-Resolution: FIXED X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: resolution bug_status Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Jul 2018 13:35:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D218731 ds changed: What |Removed |Added ---------------------------------------------------------------------------- Resolution|--- |FIXED Status|New |Closed --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Jul 26 14:34:03 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8903110510B5 for ; Thu, 26 Jul 2018 14:34:03 +0000 (UTC) (envelope-from diane@getgps.site) Received: from m2000.SaveBusinessEnergyQuotes.site (m2181.SaveBusinessEnergyQuotes.site [91.200.149.181]) by mx1.freebsd.org (Postfix) with ESMTP id 0D9A9921FD for ; Thu, 26 Jul 2018 14:34:02 +0000 (UTC) (envelope-from diane@getgps.site) Received: from m7000.au-webdesignquotes.site (m7233.au-webdesignquotes.site [185.225.126.233]) by m2000.SaveBusinessEnergyQuotes.site (Postfix) with ESMTP id BA3F21D934F for ; Thu, 26 Jul 2018 15:45:28 +0300 (IDT) Received: from m4100.au-fleetgpsquotes.website (m4110.au-fleetgpsquotes.website [185.237.187.46]) by m7000.au-webdesignquotes.site (Postfix) with ESMTP id B6D224256D for ; Thu, 26 Jul 2018 15:57:33 +0300 (IDT) Received: from m1.bizquotes.click (unknown [10.20.15.1]) by m4100.au-fleetgpsquotes.website (Postfix) with ESMTP id B3CAF41A78 for ; Thu, 26 Jul 2018 15:57:11 +0300 (IDT) Received: from list.bizquotes.click (unknown [192.168.100.137]) by m1.bizquotes.click (Postfix) with ESMTP id AD5154BC1 for ; Thu, 26 Jul 2018 15:53:07 +0300 (IDT) To: freebsd-virtualization@freebsd.org Subject: GPS Vehicle Tracking for Charlie's Choice Message-ID: <8e2ee358f72b1a202c7c6922718eaf39@list.bizquotes.click> Date: Thu, 26 Jul 2018 12:36:59 +0000 From: "Diane Anderson " Reply-To: diane@getgps.site MIME-Version: 1.0 X-Mailer-LID: 42,41,41 X-Mailer-RecptId: 34790047 X-Mailer-SID: 21383 X-Mailer-Sent-By: 1 Content-Type: text/plain; format=flowed; charset="UTF-8" Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Jul 2018 14:34:03 -0000 Hello, My name is Diane and I am with GetGPS, a company that provides businesses with the ability to monitor their employees on the road while increasing productivity. I want to help you better understand the GPS fleet tracking industry and get you the lowest service rates possible. Even if you already have a source for GPS fleet tracking, it is good to know what your options are- as there are constantly newer and more advanced alternatives. We will show you the most current options with the best discounts provided by highly reputable nationwide GPS software providers. Please provide the following information in order to compose a Quote: * Full Name * Company Name * Phone number * Zip * Fleet size I encourage you to check out free quote service by replying with your needs. Best regards, Diane Anderson Diane@GetGPS.site Unsubscribe by reply back "REMOVE". From owner-freebsd-virtualization@freebsd.org Fri Jul 27 15:03:37 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD37A104DCF5 for ; Fri, 27 Jul 2018 15:03:37 +0000 (UTC) (envelope-from josiaslg@bsd.com.br) Received: from mail-pg1-x52d.google.com (mail-pg1-x52d.google.com [IPv6:2607:f8b0:4864:20::52d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 589CD7920B for ; Fri, 27 Jul 2018 15:03:36 +0000 (UTC) (envelope-from josiaslg@bsd.com.br) Received: by mail-pg1-x52d.google.com with SMTP id z8-v6so3394379pgu.8 for ; Fri, 27 Jul 2018 08:03:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsd.com.br; s=capeta; h=mime-version:in-reply-to:references:from:date:message-id:subject:cc; bh=Hh+Fd2/2hs3g1qG1jNtzB1KjIejq9bkSTxvwKqVbCe8=; b=Nh+wsFHOMBVjvtN2zdh/Lu246xSKxvCl8S6KuvBTx84yrKW+M68UhqowwID4zvrzfH GHoWaELLiTNQcJMNWNwsZUb5YWyPuc2lWP1HfSZo1QIIw4AywaRm42x1KTW2ZBOaFwGu qPR3Hzz9ezRcU03GkfVrMg/Rng2OAHs7IGcNk= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:cc; bh=Hh+Fd2/2hs3g1qG1jNtzB1KjIejq9bkSTxvwKqVbCe8=; b=M4NQN438NyTOF653yRUzFbwfFnlmWfw6PG8z1opcL9jZM3XRYNFwlgUtMO2jr5eybw xnTLW/oVdeuJh02Gxai+tvaZqbBpVHbjiNhyNujgo40dS9Z5l4qGEoUmEegNvCMx/KRZ 3bLuidcvnu+Ivll6akgpaFxF0JD4uFks/PDAIH/0KaKw7F1McWdtIcJNHA1U34qiVYWn aXQJMeY2ztiNGKAYSrSpOXojIs0twp7yIW2BogLHM3jzRy0N/rCuc0yAErQlP8jv/OvG L8ENw5/UNDCK0z6yTzARblg54BTp2tzRDMOI3qrnOhgWFIbXc0d+a2yxyFBU5egaMOJk fvMg== X-Gm-Message-State: AOUpUlGhWjuB4AkmC+vFwjdOCqH5F+idHqWBYVdcSy58gKjCjdmnZbOH oBSJT1rBgTVzo341ACQ8ihzwhSqGL2vIjbLpwlEEk6jJ38Y= X-Google-Smtp-Source: AAOMgpevBBy++cf+t33FZs9G1SNw4Qtaa0tXCN6f/0G6Lr/7pd7BqKoSrt0f/i4GOGKZl/0/kQhnYR05hsmcD165Mgw= X-Received: by 2002:a63:f18:: with SMTP id e24-v6mr6544828pgl.320.1532703815682; Fri, 27 Jul 2018 08:03:35 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a17:90a:a586:0:0:0:0 with HTTP; Fri, 27 Jul 2018 08:03:35 -0700 (PDT) In-Reply-To: References: <1595599bd62b43b2842b9a652c6d1773@SERVER.ad.usd-group.com> <1530874279.512578.1431934576.27AF56F1@webmail.messagingengine.com> <1dbba8ac-eca4-bb09-9d3b-ec26ed1452e5@cgi.cz> From: =?UTF-8?Q?Josias_L=2E_Gon=C3=A7alves?= Date: Fri, 27 Jul 2018 12:03:35 -0300 Message-ID: Subject: Re: bhyve server 2016 lockup Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Jul 2018 15:03:38 -0000 2018-07-06 9:05 GMT-03:00 Matt Churchyard : > Thanks for the replies guys. I've increased memory and switched to > virtio-net. Been running for an hour or two so far. Windows just complain= s > about the boot device if I try to move to virtio-blk. > I might set up another test using different settings during installation > and see how that goes. > > Only really played about with bhyve so far but starting to consider using > for production machines, so any hard lockup is a bit of a worry. > > Regards, > Matt > > > Dave, > > > you got working Windows guest with virtio-blk? > > > I thought it was still unstable. I've tested virtio storage (on bhyve) = + > virtio storage drivers on several windows guest ~ 2 months ago and they > were crashing all the time. > > > So I use ahci-hd with every windows guest we run (from 7 to Win Server > 2012). > > > -- > > > regards > > > Jakub Chromy > > > CGI Systems div. > ---------------- > CGI CZ s.r.o. > sales@cgi.cz > 775 144 257 > 234 697 102 > www.cgi.cz > > On 6.7.2018 12:51, Dave Cottlehuber wrote: > > On Fri, 6 Jul 2018, at 10:22, Matt Churchyard wrote: > >> Hello, > >> > >> I have a server 2016 virtual machine which I only created for testing > >> and don't use much. Booting it up recently I've found that it locks > >> up consistently within a few minutes of booting. What can I do to try > >> and figure out what's causing the problem? > >> > >> Host was recently updated to 11.2 > >> # uname -a > >> FreeBSD dev.--- 11.2-RELEASE FreeBSD 11.2-RELEASE #0 r335510: Fri Jun = 22 > >> 04:32:14 UTC 2018 root@releng2.nyi.freebsd.org:/ > usr/obj/usr/src/sys/ > >> GENERIC amd64 > >> > >> Bhyve command is as follows > >> # bhyve -c 2 -m 2G -Huwl bootrom,/usr/local/share/uefi-firmware/ > >> BHYVE_UEFI.fd > >> -U 841e8764-75f4-11e8-b2e3-50e549369bc6 -l com1,stdio > >> -s 0,hostbridge -s 31,lpc -s 3:0,ahci-cd,/vm/.config/null.iso -s > >> 4:0,ahci-hd,/vm/w2016/disk0.img -s 5:0,e1000,tap1,mac=3D58:9c:fc:08:8e= :70 > >> -s 6:0,fbuf,tcp=3D0.0.0.0:5900 -s 7:0,xhci,tablet w2016 > > Hi Matt, > > > > try using virtio for nic & storage to start with. I've used specificall= y > these drivers https://fedorapeople.org/groups/virt/virtio-win/direct- > downloads/archive-virtio/virtio-win-0.1.96/virtio-win-0.1.96.iso and had > trouble with both later and earlier drivers, YMMV, and I sadly forget whe= re > I got this advice. > > > > Also, it=E2=80=99s possible to install virtio drivers directly from the= CD > without pointy-clicky stupidness. Just add the CD to your bhyve command, = in > a later "PCI slot" than the existing =E2=80=9Chard disk=E2=80=9D, as the = lowest numbered > slot is what=E2=80=99s used to boot from AFAICT. > > > > - open an elevated command prompt > > - run for /d %i in (d:\*) do @pnputil -a %i\w7\amd64\*.inf (you may > > need a different driver than w7 here) > > - some of those will fail but just carry on > > - reboot FTW > > > > more notes here on my bhyve setup https://hackmd.io/s/rJvJuE-CW# > > > > A+ > > Dave > > _______________________________________________ > > freebsd-virtualization@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org= / > mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" > If BLOCKIF_IOV_MAX is turned a arg in bhyve, we could possible use specific values to linux machines and windows machines. I have change this in block_if.h and have some machines to just run windows and others to run just linux. Is a kind of fragmentation i don't like and if have some option to pass in bhyve specify the BLOCKIF_IOV_MAX, would be good. If not possible, create some option where if specify windows like guest, other value of BLOCKIF_IOV_MAX is used by bhyve in execution. From owner-freebsd-virtualization@freebsd.org Sat Jul 28 19:07:09 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39135105AC01 for ; Sat, 28 Jul 2018 19:07:09 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id CD6EC7B181 for ; Sat, 28 Jul 2018 19:07:08 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: by mailman.ysv.freebsd.org (Postfix) id 91C26105ABFF; Sat, 28 Jul 2018 19:07:08 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8045A105ABFE; Sat, 28 Jul 2018 19:07:08 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: from hel.cgi.cz (hel.cgi.cz [178.238.36.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 15A6B7B17F; Sat, 28 Jul 2018 19:07:07 +0000 (UTC) (envelope-from hicks@cgi.cz) Received: from hel.cgi.cz (localhost [127.0.0.1]) by hel.cgi.cz (Postfix) with ESMTP id D15CB1292C8; Sat, 28 Jul 2018 21:06:55 +0200 (CEST) X-Virus-Scanned: amavisd-new at cgi.cz Received: from hel.cgi.cz ([127.0.0.1]) by hel.cgi.cz (hel.cgi.cz [127.0.0.1]) (amavisd-new, port 10024) with LMTP id GVTuqpIfRA8U; Sat, 28 Jul 2018 21:06:54 +0200 (CEST) Received: from mail2.cgi.cz (hermes [172.17.174.1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by hel.cgi.cz (Postfix) with ESMTPS id C3B211292B2; Sat, 28 Jul 2018 21:06:53 +0200 (CEST) Received: from webmail2.cgi.cz (mysql-postfix [172.17.174.1]) by mail2.cgi.cz (Postfix) with ESMTPA id A3C7E54E4A; Sat, 28 Jul 2018 21:06:53 +0200 (CEST) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 28 Jul 2018 21:06:53 +0200 From: Jakub Chromy To: "freebsd-virtualization@freebsd.org" Cc: owner-freebsd-virtualization@freebsd.org Subject: Re: Overcommitting CPUs with BHyve? In-Reply-To: References: Message-ID: <5fa93e55edf1a2f657f92e9bd05fbfc6@cgi.cz> X-Sender: hicks@cgi.cz User-Agent: Roundcube Webmail/1.3.5 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 28 Jul 2018 19:07:09 -0000 > Yes it's a problem, and it's not just BHyve. The problem comes from > stuff > like spinlocks. Unlike normal userland locks, when two CPUs contend on > a > spinlock both are running at the same time. When two vCPUs are > contending > on a spinlock, the host has no idea how to prioritize them. Normally > that's not a problem, because physical CPUs are always supposed to be > able > to run. But when you overcommit vCPUs, some of them must get swapped > out > at all times. If a spinlock is being contended by both a running vCPU > and > a swapped out vCPU, then it might be contended for a long time. The > host's > scheduler simply isn't able to fix that problem. The problem is even > worse > when you're using hyperthreading (which I am) because those eight > logical > cores are really only four physical cores, and spinning on a spinlock > doesn't generate enough pipeline stalls to cause a hyperthread switch. > So > it's probably best to stick with the n - 1 rule. Overcommitting is ok > if > all guests are single-cored because then they won't use spinlocks. But > my > guests aren't all single-cored. I've just checked the handbooks for KVM and although they do recommend to keep the vCPUs single-cored whenever possible, it says that up to 3:1 vCPU vs CPU ratio should not mean significant peformance penalty. We've just run out of the (physical) cores on one of our BHYVE hypervisors... 32 cores (including HT), 16 VMs... and now I need to get another machine. My question is -- does KVM (on Linux) handle the CPU overprovisioning with multiple-vCPU guest better than the current BHYVE implamentation? If yes, how can it be improved? Can we (co)sponsor the development somehow? :) Jakub -- regards www.cgi.cz