From owner-freebsd-virtualization@FreeBSD.ORG Wed Nov 12 22:31:18 2014 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D80C4A62; Wed, 12 Nov 2014 22:31:17 +0000 (UTC) Received: from mail-wg0-x235.google.com (mail-wg0-x235.google.com [IPv6:2a00:1450:400c:c00::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 67D742ED; Wed, 12 Nov 2014 22:31:17 +0000 (UTC) Received: by mail-wg0-f53.google.com with SMTP id b13so15427726wgh.26 for ; Wed, 12 Nov 2014 14:31:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=LebA+T87HannqAvVtzNUYjGtPmqgGQLNYMf4WJnxokA=; b=cAnGvvKpO+w/m3NvVk5XW2/e1qnaqseWslEeZg4hU3l59B9LwnoSLyCNs9h33jh1vh hA9ECIw2d0EJlXPmOuMPFCbPcNB/EH5LJhM1ohT3bOwuq417fY7b30JI/BGzc8sQDHvo P+suQfkJLUSDBOB3DnqJ040ZJJb1O1pmWyvDCU0G1YdLlFBfsZOIBQqaWQ4YKQJSFOAu aCngDoy3xheshCIbekIxw9rxzjsuOqkOrIA7IOovznrhqcMbDnPClKiHdIs/KMZfwcpy uJ1Mws7Md45eVV7SM/CeVt3vlU6A0soa2KGrxbmIgeihkpShj4Ko34iMyHMHSi6kVYgg SFXQ== MIME-Version: 1.0 X-Received: by 10.180.9.106 with SMTP id y10mr52823522wia.30.1415831475826; Wed, 12 Nov 2014 14:31:15 -0800 (PST) Received: by 10.27.11.42 with HTTP; Wed, 12 Nov 2014 14:31:15 -0800 (PST) In-Reply-To: <49cc2695179830c899aef87b7a8b72d8@openmailbox.org> References: <8e39523570b667d223091adc3791c9e2@openmailbox.org> <5463D746.4030803@freebsd.org> <49cc2695179830c899aef87b7a8b72d8@openmailbox.org> Date: Wed, 12 Nov 2014 14:31:15 -0800 Message-ID: Subject: Re: How hard is BHyVe's 16 vCPU limit, is it configurable under any circumstance? From: Neel Natu To: Tinker Content-Type: text/plain; charset=UTF-8 Cc: owner-freebsd-virtualization@freebsd.org, "freebsd-virtualization@freebsd.org" , Allan Jude X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Nov 2014 22:31:18 -0000 Hi Tinker, On Wed, Nov 12, 2014 at 2:08 PM, Tinker wrote: > On 2014-11-12 23:55, Allan Jude wrote: >> >> On 2014-11-12 16:39, tinkr@openmailbox.org wrote: >>> >>> Hi! >>> >>> In order justify giving energy to BHyVe, I need to know if it's >>> "future-proof" in that the 16 vCPU limit can be increased? >>> >>> Please let the world know if BHyVe's 16 vCPU limit can be lifted in some >>> way, by configuration, patch, etc. (and if you want to, why this limit >>> is in place by default today). >>> >>> Thanks!! >>> Tinker >>> >>> _______________________________________________ >>> freebsd-virtualization@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >>> To unsubscribe, send any mail to >>> "freebsd-virtualization-unsubscribe@freebsd.org" >> >> >> You can increase the limit by editing sys/amd64/include/vmm.h >> >> #define VM_MAXCPU 16 >> >> From what I've been told, things scale badly above 24 CPUs. They plan to >> solve this issue, but have not yet. If you system has enough cores to >> support using more than 16 per VM, you can modify the file and recompile >> the kernel and use as many CPUs as you want, but not much testing has >> happened with bigger numbers. > > > Dear Allan, > > Thank you very much for responding. > > Aha, great! > > > Only for completeness, do you have any particular idea about > * When the scaling above 24 vCPU:s will be optimized, like approx how many > years away is it (like 1 or more than 1)?, and bhyve allocates memory for the maximum number of vcpus when the virtual machine is created. This is simple to implement and also fits in nicely with bhyve's loader model where the guest loader (bhyveload/grub-bhyve) is separate from bhyve. The downside is that if VM_MAXCPU is a large number then there is a large memory penalty for virtual machines that only use 1 or 2 vcpus. Hence the desire to cap it at a smallish number. There are a few ways to deal with this: - patch the code to change VM_MAXCPU (this is what you need to do today) - allow the maximum vcpus to be changed via a tunable (this can be done in short order) - the limit can go away when bhyve moves to a single process model because we'll know the number of vcpus at VM creation time. > * What the technological reason for the scaling is, is it that somehow the > BHyVe instances on the different cores need to inter-communicate, for > instance that all disk and network IO is done via one single core currently? > Actually, the performance depends more on the workload and the guest OS so you should definitely try how it goes for your application. We'll be happy to help fix any issues that come up. best Neel > > In all cases, your response is great news, as your baseline answer that it's > doable and only a question of optimization and tweaking of present > functionality - > > Thanks again! > > Tinker > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org"