From owner-freebsd-virtualization@FreeBSD.ORG Mon Mar 25 11:06:52 2013 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E49D2EB for ; Mon, 25 Mar 2013 11:06:52 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id CF286DB for ; Mon, 25 Mar 2013 11:06:52 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r2PB6qeL007346 for ; Mon, 25 Mar 2013 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r2PB6qaC007344 for freebsd-virtualization@FreeBSD.org; Mon, 25 Mar 2013 11:06:52 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 25 Mar 2013 11:06:52 GMT Message-Id: <201303251106.r2PB6qaC007344@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-virtualization@FreeBSD.org Subject: Current problem reports assigned to freebsd-virtualization@FreeBSD.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Mar 2013 11:06:53 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/170096 virtualization[vimage] Dynamically-attached network interface will c o kern/169991 virtualization[run] [vimage] panic after device plugged in o kern/165252 virtualization[vimage] [pf] [panic] kernel panics with VIMAGE and PF o kern/161094 virtualization[vimage] [pf] [panic] kernel panic with pf + VIMAGE wh o kern/160541 virtualization[vimage][pf][patch] panic: userret: Returning on td 0x o kern/160496 virtualization[vimage] [pf] [patch] kernel panic with pf + VIMAGE o kern/148155 virtualization[vimage] [pf] Kernel panic with PF/IPFilter + VIMAGE k a kern/147950 virtualization[vimage] [carp] VIMAGE + CARP = kernel crash s kern/143808 virtualization[pf] pf does not work inside jail a kern/141696 virtualization[rum] [vimage] [panic] rum(4)+ vimage = kernel panic 10 problems total. From owner-freebsd-virtualization@FreeBSD.ORG Mon Mar 25 18:54:14 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 070B5666 for ; Mon, 25 Mar 2013 18:54:14 +0000 (UTC) (envelope-from fivo.11235813@gmail.com) Received: from mail-wi0-x22a.google.com (mail-wi0-x22a.google.com [IPv6:2a00:1450:400c:c05::22a]) by mx1.freebsd.org (Postfix) with ESMTP id A0E45A24 for ; Mon, 25 Mar 2013 18:54:13 +0000 (UTC) Received: by mail-wi0-f170.google.com with SMTP id hm11so11426311wib.1 for ; Mon, 25 Mar 2013 11:54:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:from:date:message-id:subject:to :content-type; bh=mWJ9PThACJEk2foCs+mrczpSZXiJUo1sV1gfjvQi+GQ=; b=Zpn8dMTR1WcLhN+UW0Gn9GAq0XP/FVYdaUPcSacG+27XHW3g2c/dRNAaV/UD5YxE2k OfUeSfWD9GE380czUhj8Zu8LeKadqLoFn7TsDlw3oTDCKlrFqHvwsyULwSQpTd4rv6UP fo70FNFpucmwRXp6IONGe2SOvnQYvxv5PQUb+cE8ZAk31yLEltOIp7U6TJR/lr5eiQhx p5IpAwM4zZa3Zy2JpXyIvE24DvUunHR4uhHrqyvVFeAJnMmXoReXZ6t2bxgCNTP9Aln3 YL4f6CTZjZCX0WTWJ08ZNadZw/uR4NcOzPB3BIVxAMX2+0VnZLgnyATsVbmExMJU1SWB cxNw== X-Received: by 10.180.74.67 with SMTP id r3mr26404063wiv.14.1364237652911; Mon, 25 Mar 2013 11:54:12 -0700 (PDT) MIME-Version: 1.0 Received: by 10.194.169.161 with HTTP; Mon, 25 Mar 2013 11:53:51 -0700 (PDT) From: =?UTF-8?Q?Iori=E3=80=80YONEJI?= Date: Tue, 26 Mar 2013 03:53:51 +0900 Message-ID: Subject: suspend/resume on BHyVe To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 25 Mar 2013 18:54:14 -0000 Hello. I'm thinking of adding suspend/resume features on BHyVe. For this, I think those below must be implemented. - virtual machine state command interface - saving registers per CPU - dumping physical memory - saving virt-io and other device emulation state To save registers, the sysctl used in bhyvectl (vmmctl command previously) is helpful, however, it's interface is no good because getting register value cause a sysctl call so to context-switch per one register, and for getting all registers, it's not efficient. I think it's more preferable to make a struct to set or unset boolean fields per register, to tell which registers kernel should return, and kernel returns those state with struct vmxctx. struct vmxctx is such. struct vmxctx { register_t tmpstk[32]; /* vmx_return() stack */ register_t tmpstktop; register_t guest_rdi; /* Guest state */ register_t guest_rsi; : : And, considering memory dump, /dev/vmm/vmname is a file that is a map of guest memory, so memory dump doesn't seem hard, just stop vm, write back all guest cache, and copy memory file to a regular file. Finally, I don't know much about device state, but I think there must some state to be saved, like network stack. I'm not sure I wrote former, so I appreciate your ideas and suggestions. Thanks, Iori. From owner-freebsd-virtualization@FreeBSD.ORG Tue Mar 26 05:04:58 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BB6FC2D8 for ; Tue, 26 Mar 2013 05:04:58 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-ia0-x22d.google.com (mail-ia0-x22d.google.com [IPv6:2607:f8b0:4001:c02::22d]) by mx1.freebsd.org (Postfix) with ESMTP id 9130B9D2 for ; Tue, 26 Mar 2013 05:04:58 +0000 (UTC) Received: by mail-ia0-f173.google.com with SMTP id h37so6159616iak.32 for ; Mon, 25 Mar 2013 22:04:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type:content-transfer-encoding; bh=0UbwFeVvJM+LztMJpsX18UCYHt5WH0sKIvW84ONMik4=; b=aF1Rg8AWIlFhPBip5FMYYWJK2lsSSesC3X8YiFTxZQrqMbNPxl5M41eH8bs2+cWTFL Toi8WmfGv4XM/5TMdVh3wlwHF85PNuHdDXcWdmUmNEIiew8NOvpTpjH3yrMYqTeEDz8y BTcs9YdPv8UUYleKNySr0dd4NLKDCEueraMqxkE49JxfmIvEgG+Gw0/znLg7RosIWwB7 AQ6h9OaW41xZSyBg6kik2YYTPOXzSFYhswDxTCNCiHOSRKF3SlYPAvkc84jLyXimj1W5 MUCX6ieAY+FgkE7TO7k/UROrkHS27zwkdUrkbkSXocOrWGUTXTuRunm6Dj9osYVWGfWt M03w== MIME-Version: 1.0 X-Received: by 10.43.62.12 with SMTP id wy12mr8429981icb.19.1364274298348; Mon, 25 Mar 2013 22:04:58 -0700 (PDT) Received: by 10.42.23.132 with HTTP; Mon, 25 Mar 2013 22:04:58 -0700 (PDT) In-Reply-To: References: Date: Mon, 25 Mar 2013 22:04:58 -0700 Message-ID: Subject: Re: suspend/resume on BHyVe From: Neel Natu To: Iori YONEJI Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Mar 2013 05:04:58 -0000 Hi Lori, On Mon, Mar 25, 2013 at 11:53 AM, Iori=E3=80=80YONEJI wrote: > Hello. > > I'm thinking of adding suspend/resume features on BHyVe. > Fantastic! > For this, I think those below must be implemented. > > - virtual machine state command interface > - saving registers per CPU > - dumping physical memory > - saving virt-io and other device emulation state > > > To save registers, the sysctl used in bhyvectl (vmmctl command > previously) is helpful, > however, it's interface is no good because getting register value > cause a sysctl call > so to context-switch per one register, and for getting all registers, > it's not efficient. > I think it's more preferable to make a struct to set or unset boolean > fields per register, > to tell which registers kernel should return, and kernel returns those > state with struct vmxctx. > > struct vmxctx is such. > struct vmxctx { > register_t tmpstk[32]; /* vmx_return() stack */ > register_t tmpstktop; > > register_t guest_rdi; /* Guest state */ > register_t guest_rsi; > : > : > > > And, considering memory dump, /dev/vmm/vmname is a file that is a map > of guest memory, > so memory dump doesn't seem hard, just stop vm, write back all guest > cache, and copy > memory file to a regular file. > > Finally, I don't know much about device state, but I think there must > some state to be saved, like > network stack. > > I'm not sure I wrote former, so I appreciate your ideas and suggestions. > I think that you are on the right track. A brute force way of figuring out all the state must be saved is to look at all the initialization functions that are called when a vm and a vcpu are created. So, this would be vm_create() and vcpu_init() in the kernel module. There is also the hardware assist state that is maintained by the processor (VT-x or SVM) and this includes things like guest interruptibility, guest run state etc. I am assuming that it would be sufficient to save the VMCS page after telling the processor to flush any state it may be caching on chip. There is also emulated pci bus, virtio devices and legacy isa device state that would need to be saved by the userspace 'bhyve' process. And finally there is the matter of how to communicate with 'bhyve' process that it needs to suspend the virtual machine and write its state to disk - perhaps a signal would be good enough place to start. This certainly sounds like an interesting and challenging project and we would be happy to help in any way we can. best Neel > Thanks, Iori. > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@free= bsd.org" From owner-freebsd-virtualization@FreeBSD.ORG Tue Mar 26 15:45:45 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A124CE64 for ; Tue, 26 Mar 2013 15:45:45 +0000 (UTC) (envelope-from larrymelia@gmail.com) Received: from mail-ia0-x230.google.com (mail-ia0-x230.google.com [IPv6:2607:f8b0:4001:c02::230]) by mx1.freebsd.org (Postfix) with ESMTP id 77EFD6CB for ; Tue, 26 Mar 2013 15:45:45 +0000 (UTC) Received: by mail-ia0-f176.google.com with SMTP id i1so6524189iaa.35 for ; Tue, 26 Mar 2013 08:45:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:mime-version:sender:from:date:x-google-sender-auth :message-id:subject:to:content-type; bh=Wx78Ilbp6+yHzK3qrDeBDBI3+NSik+Q/4vb4KnqKWqU=; b=FROl4uJpEW0Bu5YTJ0tSm42wwNgrXvTANni3n+JLgNWUM2qCSPBej7RrsVDSt+Vy5i 0nW6vP8uKB2GqYUHFy210KnfpBsQoU5pSHxHQ2VfqMKDvb9Q8psIzSXTOqlc6L2L1KYm tjvnTQ+X3o47hGDo+a7jYUolmM7rptnI78/8X0HfJMeLoZG5OY0x7cJtQyp+PdAQhB2J C2a4l6GItNTQ6FDk9UlT8d2YRHbdTz+P6jFm5p5+EfrjNsxm/RItM4pH8ZUWtgIH7kKu YTN/zB3JPcK7fPTJIi3jGAtwxNe39KModjVsVZmQXZ5WJtm0eIcbvZ+rlfWfhAWTPT+f HlWg== X-Received: by 10.50.135.105 with SMTP id pr9mr1732390igb.6.1364312745067; Tue, 26 Mar 2013 08:45:45 -0700 (PDT) MIME-Version: 1.0 Sender: larrymelia@gmail.com Received: by 10.64.110.162 with HTTP; Tue, 26 Mar 2013 08:45:04 -0700 (PDT) From: Larry Melia Date: Tue, 26 Mar 2013 08:45:04 -0700 X-Google-Sender-Auth: 0RgQ7oneKqXwsrH7JeB5cXssc7A Message-ID: Subject: New vendor branch for FreeBSD on Hyper-V. Please review. To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Mar 2013 15:45:45 -0000 We are currently preparing our =93enlightened=94 device drivers for inclusi= on with the standard distribution of FreeBSD. They provide significant performance enhancements to FreeBSD on Hyper-V. As part of this process we are making our source code available for review by the FreeBSD community. We would truly appreciate your comments and concerns. There are two options for enabling the drivers. You can include them as a monolithic build ( https://github.com/FreeBSDonHyper-V/freebsd/wiki/Build-the-kernel-with-the-= HyperV-drivers) or as loadable drivers by manually configuring them to load at boot time. So, while we plan to include them in the GENERIC build for AMD64, they are only loaded by including them in the boot configuration. Since we will be actively maintaining our drivers, we are placing them in a vendor branch. However, there=92s a small patch to the ATA driver that when it detects the presence of Hyper-V it disables it. This allows our =93enlightened=94 storage driver to replace it and provide Fast IDE for the file system. Furthermore, we wish that the current maintainer of the ATA driver also maintain this patch. This will allow our focus to be on our drivers and prevent us being a hindrance to any future changes to the ATA driver. Our source code can be reviewed on Github at https://github.com/FreeBSDonHyper-V/VendorBranchForFreeBSDonHyper-V. You may leave comments or questions on this issue board https://github.com/FreeBSDonHyper-V/VendorBranchForFreeBSDonHyper-V/issues. Or if you prefer, you may post them on this mailing list ( http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization). From owner-freebsd-virtualization@FreeBSD.ORG Wed Mar 27 07:17:09 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 90591E8F for ; Wed, 27 Mar 2013 07:17:09 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-ea0-x22e.google.com (mail-ea0-x22e.google.com [IPv6:2a00:1450:4013:c01::22e]) by mx1.freebsd.org (Postfix) with ESMTP id 30596941 for ; Wed, 27 Mar 2013 07:17:09 +0000 (UTC) Received: by mail-ea0-f174.google.com with SMTP id m14so1081990eaj.19 for ; Wed, 27 Mar 2013 00:17:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=x-received:sender:message-id:date:from:user-agent:mime-version:to :subject:content-type:content-transfer-encoding; bh=2swH8qV9u3AKBowjws2G5LJgWK+Y8lhp7DQbZhQIthY=; b=kArz5li4OL64Xh0KHu23Lxo3XI9PFHtcIE0juIy3CVCNoaENPjWEO7soWi5yL//w2h xftNxTi0dCezhI8XH9No+JCv8CERW0r9Ydv80YKVpdyxcHwzt9+X4ObYVcMesK1+vM5B tQKaJEtfDHsQNfyurRzfxeVWfSWW0mPPXRX4wpypgK9BaeKaMlKkOhqt/WCXkt/2jEQ4 M5HRUdkPIfBDgGtFZ9pZkozH1lQCSl0YmHpHk7l2RZUrgmIoniLUS/6BigMnDJPQlnlK 2OBQXFafrUNXzaurvqQ07oNnPG16rrL1msk0DBZrTgFto4b73lGN5bwlNLimWE8nrZm9 tQeQ== X-Received: by 10.15.36.67 with SMTP id h43mr52477238eev.5.1364368628344; Wed, 27 Mar 2013 00:17:08 -0700 (PDT) Received: from mavbook.mavhome.dp.ua (mavhome.mavhome.dp.ua. [213.227.240.37]) by mx.google.com with ESMTPS id t4sm29353059eel.0.2013.03.27.00.17.05 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 27 Mar 2013 00:17:07 -0700 (PDT) Sender: Alexander Motin Message-ID: <51529CF0.5090503@FreeBSD.org> Date: Wed, 27 Mar 2013 09:17:04 +0200 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130326 Thunderbird/17.0.4 MIME-Version: 1.0 To: Larry Melia , freebsd-virtualization@freebsd.org Subject: Re: New vendor branch for FreeBSD on Hyper-V. Please review. Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 07:17:09 -0000 Hi. As ATA driver maintainer I've took a look on that part of the code. I have some comments about what I see. Please keep me CC'd, as I am normally not on this list. 1) You are using machine dependent function do_cpuid() in machine independent ATA code. It will just not build on any architecture except amd64 and i386. You should periodically run `make universe` to be sure that you code does not breaks the build for some rare archs. As minimum that code should be wrapped into respective #ifdef's, but I would prefer it to be moved out somewhere to MD code (like MD code that does general VM detection now), as this doesn't really belong to the ATA driver. 2) You are disabling "ad" ATA disk driver. Here I have two objections: a) "ad" driver is obsoleted and not used since FreeBSD 9.0, when CAM-based ATA stack replaces it completely, wrapping only controller driver parts of old ata(4) into CAM SIM, which is now only one of four ATA/SATA SIMs in the system. Your change is effectively null now. b) I believe you are using wrong approach here. I believe instead of completely disabling disk driver you should disable only specific Hyper-V emulated ATA controller device instance. I've never used Hyper-V, but I guess it should support PCI device pass-through. Just imagine what happen if somebody wish to pass-through ATA/SATA controller? You will disable it also. 3) Looking on what I suppose is "Fast IDE" driver in the file hv_storvsc_drv_freebsd.c, it seems more like cut-down SAS, then IDE. It doesn't support ATA commands, it reports itself to CAM as SAS. The only differences I see are disabling LUNs and limiting queue to 1 slot. Just a smile: it is neither really "Fast" nor at all "IDE". :) 4) In you SAS driver in storvsc_action() you have hardcoded CTS_SCSI_FLAGS_TAG_ENB flag. While I guess it should not cause any problems, it wont allow user to control it if he decide to experiment with disabling it. -- Alexander Motin From owner-freebsd-virtualization@FreeBSD.ORG Wed Mar 27 09:31:28 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 43598490 for ; Wed, 27 Mar 2013 09:31:28 +0000 (UTC) (envelope-from kindule@gmail.com) Received: from sam.nabble.com (sam.nabble.com [216.139.236.26]) by mx1.freebsd.org (Postfix) with ESMTP id 2CAAB75D for ; Wed, 27 Mar 2013 09:31:28 +0000 (UTC) Received: from [192.168.236.26] (helo=sam.nabble.com) by sam.nabble.com with esmtp (Exim 4.72) (envelope-from ) id 1UKmhb-0004ue-St for freebsd-virtualization@freebsd.org; Wed, 27 Mar 2013 02:31:27 -0700 Date: Wed, 27 Mar 2013 02:31:27 -0700 (PDT) From: kindule To: freebsd-virtualization@freebsd.org Message-ID: <1364376687875-5799323.post@n5.nabble.com> Subject: KVM with freeBSD and SR-IOV vlan doesn't working! MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 09:31:28 -0000 Recently, we use KVM and SR-IOV in our project. And we choose freeBSD10 as the guest os.As we use the ip address in the igb0 of our freeBSD10 guest, it working no problem.However when i use vlan in our igb0 of the freeBSD10 guest,we can see the vlan assigned right and we can ping the vlan address without problem.But we add a gateway of the vlan area.we can't connnected to the gateway. there are some os messages: Host: Debian 7.0 and KVM 1.2 Guest: FreeBSD10-current And thanks for your help! -- View this message in context: http://freebsd.1045724.n5.nabble.com/KVM-with-freeBSD-and-SR-IOV-vlan-doesn-t-working-tp5799323.html Sent from the freebsd-virtualization mailing list archive at Nabble.com. From owner-freebsd-virtualization@FreeBSD.ORG Wed Mar 27 09:59:34 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8DF19183 for ; Wed, 27 Mar 2013 09:59:34 +0000 (UTC) (envelope-from syuu@dokukino.com) Received: from mail-pa0-f49.google.com (mail-pa0-f49.google.com [209.85.220.49]) by mx1.freebsd.org (Postfix) with ESMTP id 66993A02 for ; Wed, 27 Mar 2013 09:59:34 +0000 (UTC) Received: by mail-pa0-f49.google.com with SMTP id kp14so1813696pab.22 for ; Wed, 27 Mar 2013 02:59:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dokukino.com; s=google; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=Ifem+uC9GSVZm2yi7oSRjnQQx5cli7xMcBC2AYIW1R0=; b=JcIxQXi8hUjBaUl7UoVsXJzBkaYxfboBGGSeYYHy3SJ30ntkxzdOU8KZycOmKn3qzw tSAWGxRm1NpkbzG25bHSiMaIFCrWFKjJ+nRT79kEgmiyIggAlK8bhF0FBzKCbWMdIFT6 cYHhlKtO4wGMlOjQijajo7xw2JHKpfnu9HRIY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:x-gm-message-state; bh=Ifem+uC9GSVZm2yi7oSRjnQQx5cli7xMcBC2AYIW1R0=; b=DcGJYkNY/hLuBADWas1fxRWk23RDZAFWiayap0aqVIPyMFjpUJ5mQQwFeqNIC6j4eq Nago1AwnFizuau0x7v9hw8J/NexWBwFkO7hyUAASa03CiG7TbQtK5OmecNwcFNyPRHkd VEb74hcT7tLldLU5TtRRahthJOQRmLVUkm5ETpt/jI5d8Acg3grbEIimzs3QWqFOhmVw u9+BasIa44OIsiRSeyka9mTM9xBLMsPY9WlSphru6e5SN9ThGU6ooPdNct72lOBoj6ZP rzN6OzTWUBRa/6drp+oHeFwJiIStNSORVnj/KCnAK9fVSMR588mY4U3b82vJSrZNsBPj xx0Q== X-Received: by 10.66.156.196 with SMTP id wg4mr29017380pab.23.1364378368133; Wed, 27 Mar 2013 02:59:28 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.243.41 with HTTP; Wed, 27 Mar 2013 02:58:48 -0700 (PDT) In-Reply-To: References: From: Takuya ASADA Date: Wed, 27 Mar 2013 18:58:48 +0900 Message-ID: Subject: suspend/resume on BHyVe To: Neel Natu X-Gm-Message-State: ALoCoQmQzdPDjBMizkZWURJ13ROEBHtp+WSWbHRTEBIK/d0zk3zRNmxUl6BZJzjc0zF7JHj63PwT Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 09:59:34 -0000 Hi, I had some discussion with Iori about the project since last year, and now I'm suggesting him to apply Google Summer of Code'13 with the project. (GSoC'13 will start next month) > For this, I think those below must be implemented. > > > > - virtual machine state command interface > > - saving registers per CPU > > - dumping physical memory > > - saving virt-io and other device emulation state > > > > > > To save registers, the sysctl used in bhyvectl (vmmctl command > > previously) is helpful, > Maybe he meant ioctl. > however, it's interface is no good because getting register value > > cause a sysctl call > > so to context-switch per one register, and for getting all registers, > > it's not efficient. > I think it's more preferable to make a struct to set or unset boolean > > fields per register, > > to tell which registers kernel should return, and kernel returns those > > state with struct vmxctx. > > > > struct vmxctx is such. > > struct vmxctx { > > register_t tmpstk[32]; /* vmx_return() stack */ > > register_t tmpstktop; > > > > register_t guest_rdi; /* Guest state */ > > register_t guest_rsi; > > : > > : It looks we don't really have to take care register values on VMCS on here, just registers on vmxctx is enough(described below). Then, how about to add vmxctx dumping ioctl? > > > > > > And, considering memory dump, /dev/vmm/vmname is a file that is a map > > of guest memory, > > so memory dump doesn't seem hard, just stop vm, write back all guest > > cache, and copy > > memory file to a regular file. > > > > Finally, I don't know much about device state, but I think there must > > some state to be saved, like > > network stack. > > > > I'm not sure I wrote former, so I appreciate your ideas and suggestions. > > > > I think that you are on the right track. > > A brute force way of figuring out all the state must be saved is to > look at all the initialization functions that are called when a vm and > a vcpu are created. So, this would be vm_create() and vcpu_init() in > the kernel module. > > There is also the hardware assist state that is maintained by the > processor (VT-x or SVM) and this includes things like guest > interruptibility, guest run state etc. I am assuming that it would be > sufficient to save the VMCS page after telling the processor to flush > any state it may be caching on chip. I think, just dump whole VMCS page after calling VMCLEAR instruction is easiest way to do this. (I also considered to dump only necessary values on VMCS by VMREAD instruction, but maybe it's easy to break guest state mistakenly, and we don't get advantage by doing that way.) Then maybe we need VMCS dumping ioctl here. There is also emulated pci bus, virtio devices and legacy isa device > state that would need to be saved by the userspace 'bhyve' process. What is the necessary operation for virtio devices to suspend/resume? Maybe dump all rings of the devices? It doesn't have registers, right? > And finally there is the matter of how to communicate with 'bhyve' > process that it needs to suspend the virtual machine and write its > state to disk - perhaps a signal would be good enough place to start. How about this idea: bhyvectl sends VM_SUSPEND ioctl. If the guests is in VMX non-root mode, VM_SUSPEND ioctl handler sends IPI to interrupt the guest thread. Then the guest thread breaks vmx_run() loop, exit to userland with exitcode VM_EXITCODE_SUSPEND. Or, if the guests is not in VM_RUN ioctl but performing userland work(such as running virtio host-side driver), maybe you just need to wait bhyve process sends VM_RUN ioctl. When bhyve sends VM_RUN ioctl, vmm.ko should not perform VMEnter. It should just returns VM_EXITCODE_SUSPEND. On both cases, vmm.ko returns VM_EXITCODE_SUSPEND at the end. Then bhyve process can perform suspend action in VM_EXITCODE_SUSPEND handler. I think this is simple. > This certainly sounds like an interesting and challenging project and > we would be happy to help in any way we can. > > best > Neel > > > Thanks, Iori. > > _______________________________________________ > > freebsd-virtualization@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@FreeBSD.ORG Wed Mar 27 10:05:48 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5E04F341 for ; Wed, 27 Mar 2013 10:05:48 +0000 (UTC) (envelope-from syuu@dokukino.com) Received: from mail-pb0-f47.google.com (mail-pb0-f47.google.com [209.85.160.47]) by mx1.freebsd.org (Postfix) with ESMTP id 3A0FAAA4 for ; Wed, 27 Mar 2013 10:05:48 +0000 (UTC) Received: by mail-pb0-f47.google.com with SMTP id rp2so5200072pbb.34 for ; Wed, 27 Mar 2013 03:05:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dokukino.com; s=google; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=ovotHR5wcOBH9rptwSOJJdPHTUMePuuBD2pLt91lwbU=; b=bILSKsX0jx+AzzCFVcyhp3owLr06PzlsnaPF1zaUBG6nAwBEqB62vq02fqu2T44E9L b2LRGO3vNX5iUYkSgY0pDUUdsD1m8SXu7OsP/+wxbkpE0dUY7FCNKipUG5e05/hYJk1E bsGLneuSUIPCZncoKQkBtkp/zMOWEpxDWlovE= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:x-gm-message-state; bh=ovotHR5wcOBH9rptwSOJJdPHTUMePuuBD2pLt91lwbU=; b=fygRPS1FeXGKV/mgenAMC0Pej4PEgPcFUM0D7fXvyGWA+XqcrgGhI1kz6NJ+Dw4rjl QzvBZM1uIBQu1usj+709IeAW0891KA7pqYU3IBLGay2tksOnhDQLfwsnvktvtcZyB1aY vbyJVxe+iJ/mCNTUeEYQUE2Msv023yfqh0xJTMbuQfNIixHK0wtH1x/YZfYhw5dUWq/r gO6A842Kkx2nV90sh1mJrY43WmTSBL5BhOF8BveuPmfych12svs6p2ASl1JIPjbUoWE+ CQK6L7cGqjbT94USmHo1a1XoQumK3exHEaeOuTIar0drsyI/0eKNbt4Rpv1GchbEAKmW xgTg== X-Received: by 10.66.81.2 with SMTP id v2mr28558583pax.204.1364378742188; Wed, 27 Mar 2013 03:05:42 -0700 (PDT) MIME-Version: 1.0 Received: by 10.68.243.41 with HTTP; Wed, 27 Mar 2013 03:05:02 -0700 (PDT) In-Reply-To: References: From: Takuya ASADA Date: Wed, 27 Mar 2013 19:05:02 +0900 Message-ID: Subject: Re: suspend/resume on BHyVe To: Neel Natu Content-Type: text/plain; charset=UTF-8 X-Gm-Message-State: ALoCoQmDtUfw+G8SKjRXKagNTb2CdJFkOBPBDpjS8D8BJJj0umMJHByI0rxPkcJa9brJkMLak/hC Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Mar 2013 10:05:48 -0000 Hi, > A brute force way of figuring out all the state must be saved is to > look at all the initialization functions that are called when a vm and > a vcpu are created. So, this would be vm_create() and vcpu_init() in > the kernel module. I think these things are need to dump by resume action: - guest memory area - struct vmxctx - VMCS page - VLAPIC registers - virtio device state (dumping virtio ring?) - current configuration of bhyve process (with was configured by an arguments passed to bhyve process) Maybe there's more. From owner-freebsd-virtualization@FreeBSD.ORG Thu Mar 28 20:58:30 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id EF94883E for ; Thu, 28 Mar 2013 20:58:30 +0000 (UTC) (envelope-from mack@macktronics.com) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by mx1.freebsd.org (Postfix) with ESMTP id AA9D028B for ; Thu, 28 Mar 2013 20:58:30 +0000 (UTC) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by coco.macktronics.com (Postfix) with ESMTP id DC1454AC40 for ; Thu, 28 Mar 2013 15:58:23 -0500 (CDT) Date: Thu, 28 Mar 2013 15:58:23 -0500 (CDT) From: Dan Mack To: freebsd-virtualization@freebsd.org Subject: bhyve tty / login problems / panic Message-ID: <20130328155008.B5464@coco.macktronics.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Mar 2013 20:58:31 -0000 I haven't spent too much time debugging this yet but I'd thought I'd ask just in case someone else has seen this: - fresh 10.x system (248804) , standard FreeBSD iso image I made with make release - latest vmrun.sh release.iso ... - install went okay, everything seemed normal - boot splash screen comes up fine, select default and do first boot - cannot login ... first few attempts result in 'login timed out messages' (but they get emitted as soon as I hit CR) - eventually I can get in after a few tries but then the shell immediately boots me after emitting the motd :-) - and just leaving it sit results in panic - my kernel is mainly just GENERIC with the debugging yanked out and raid, scsi, and wifi devices. Any ideas? Below is the boot log and what I saw ... Booting... GDB: no debug ports present KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #4 r248804: Wed Mar 27 19:29:38 CDT 2013 root@olive.example.com:/usr/obj/usr/src/sys/MACKGEN amd64 FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.83-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0x8fa3ab7f Features2=0x83bee217 AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant real memory = 536870912 (512 MB) avail memory = 482803712 (460 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 2 package(s) x 1 core(s) cpu0 (BSP): APIC ID: 0 login: /amd64 (cocopuff.example.com) (console)ds.KLOCAL>ic 0 mtu 1500tation)3fff at device 2.0 on pci0i0 FreeBSD/amd64 (cocopuff.example.com) (console) login: root Login timed out after 300 seconds FreeBSD/amd64 (cocopuff.example.com) (console) login: root Login timed out after 300 seconds FreeBSD/amd64 (cocopuff.example.com) (console) login: root Login timed out after 300 seconds FreeBSD/amd64 (cocopuff.example.com) (console) login: root Login timed out after 300 seconds FreeBSD/amd64 (cocopuff.example.com) (console) login: root Password: Jul 12 07:12:58 cocopuff login: ROOT LOGIN (root) ON console FreeBSD 10.0-CURRENT (MACKGEN) #4 r248804: Wed Mar 27 19:29:38 CDT 2013 Welcome to FreeBSD! Before seeking technical support, please use the following resources: o Security advisories and updated errata information for all releases are at http://www.FreeBSD.org/releases/ - always consult the ERRATA section for your release first as it's updated frequently. o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and, along with the mailing lists, can be searched by going to http://www.FreeBSD.org/search/. If the doc package has been installed (or fetched via pkg install lang-freebsd-doc, where lang is the 2-letter language code, e.g. en), they are also available formatted If you still have a question or problem, please take the output of `uname -a', along with any relevant error messages, and email it as a question to the questions@FreeBSD.org mailing list. If you are unfamiliar with FreeBSD's directory layout, please refer to the hier(7) manual page. If you are not familiar with manual pages, type `man man'. Edit /etc/motd to change this login announcement. You have new mail. You have new mail. root@cocopuff:~ # auto-logout FreeBSD/amd64 (cocopuff.macktronics.com) (console) login: panic: deadlkres: possible deadlock detected for 0xfffffe001c919920, blocked for 303081 ticks cpuid = 0 KDB: enter: panic [ thread pid 0 tid 100027 ] Stopped at kdb_enter+0x3e: movq $0,kdb_why db> bt Tracing pid 0 tid 100027 td 0xfffffe000358f490 kdb_enter() at kdb_enter+0x3e/frame 0xffffff80002bcae0 panic() at panic+0x176/frame 0xffffff80002bcb60 deadlkres() at deadlkres+0x488/frame 0xffffff80002bcbb0 fork_exit() at fork_exit+0x9a/frame 0xffffff80002bcbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002bcbf0 --- trap 0, rip = 0, rsp = 0xffffff80002bccb0, rbp = 0 --- From owner-freebsd-virtualization@FreeBSD.ORG Thu Mar 28 21:13:02 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 465CEB10 for ; Thu, 28 Mar 2013 21:13:02 +0000 (UTC) (envelope-from mack@macktronics.com) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by mx1.freebsd.org (Postfix) with ESMTP id 1963230E for ; Thu, 28 Mar 2013 21:13:01 +0000 (UTC) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by coco.macktronics.com (Postfix) with ESMTP id 84A9C4AC41 for ; Thu, 28 Mar 2013 16:13:01 -0500 (CDT) Date: Thu, 28 Mar 2013 16:13:01 -0500 (CDT) From: Dan Mack To: freebsd-virtualization@freebsd.org Subject: Re: bhyve tty / login problems / panic In-Reply-To: <20130328155008.B5464@coco.macktronics.com> Message-ID: <20130328160915.T5464@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Mar 2013 21:13:02 -0000 Update#1: I was using the default of 2 virtual CPUs. When I switched to 1 CPU via vmrun.sh, the system booted clean and let me login. Also, when running with two vCPUs, the bhyve process is spinning at 100% even when the guest is doing nothing. When running with one CPU, bhyve is mostly idle when the guest is idle. Whatever was happening, it was impacting system time in a bad way as all of the log files were already getting rotated even though the system was less than 30 minutes old :-) Let me know if there is anything I can do to help and if you have any ideas on a solution (other than only using uni-processor guests). Thanks, Dan On Thu, 28 Mar 2013, Dan Mack wrote: > > I haven't spent too much time debugging this yet but I'd thought I'd ask just > in case someone else has seen this: > > - fresh 10.x system (248804) , standard FreeBSD iso image I made with > make release > > - latest vmrun.sh release.iso ... > - install went okay, everything seemed normal > > - boot splash screen comes up fine, select default and do first boot > > - cannot login ... first few attempts result in 'login timed out messages' > (but they get emitted as soon as I hit CR) > > - eventually I can get in after a few tries but then the shell immediately > boots me after emitting the motd :-) > > - and just leaving it sit results in panic > > - my kernel is mainly just GENERIC with the debugging yanked out and raid, > scsi, and wifi devices. > > Any ideas? > > > Below is the boot log and what I saw ... > Booting... > GDB: no debug ports present > KDB: debugger backends: ddb > KDB: current backend: ddb > Copyright (c) 1992-2013 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 10.0-CURRENT #4 r248804: Wed Mar 27 19:29:38 CDT 2013 > root@olive.example.com:/usr/obj/usr/src/sys/MACKGEN amd64 > FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 > CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.83-MHz K8-class CPU) > Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping > = 7 > > Features=0x8fa3ab7f > > Features2=0x83bee217 > AMD Features=0x2c100800 > AMD Features2=0x1 > TSC: P-state invariant > real memory = 536870912 (512 MB) > avail memory = 482803712 (460 MB) > Event timer "LAPIC" quality 400 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > FreeBSD/SMP: 2 package(s) x 1 core(s) > cpu0 (BSP): APIC ID: 0 > login: /amd64 (cocopuff.example.com) (console)ds.KLOCAL>ic 0 mtu > 1500tation)3fff at device 2.0 on pci0i0 > > FreeBSD/amd64 (cocopuff.example.com) (console) > > login: root > Login timed out after 300 seconds > > FreeBSD/amd64 (cocopuff.example.com) (console) > > login: root > Login timed out after 300 seconds > > FreeBSD/amd64 (cocopuff.example.com) (console) > > login: root > Login timed out after 300 seconds > > FreeBSD/amd64 (cocopuff.example.com) (console) > > login: root > Login timed out after 300 seconds > > FreeBSD/amd64 (cocopuff.example.com) (console) > > login: root > Password: > Jul 12 07:12:58 cocopuff login: ROOT LOGIN (root) ON console > FreeBSD 10.0-CURRENT (MACKGEN) #4 r248804: Wed Mar 27 19:29:38 CDT 2013 > > Welcome to FreeBSD! > > Before seeking technical support, please use the following resources: > > o Security advisories and updated errata information for all releases are > at http://www.FreeBSD.org/releases/ - always consult the ERRATA section > for your release first as it's updated frequently. > > o The Handbook and FAQ documents are at http://www.FreeBSD.org/ and, > along with the mailing lists, can be searched by going to > http://www.FreeBSD.org/search/. If the doc package has been installed > (or fetched via pkg install lang-freebsd-doc, where lang is the > 2-letter language code, e.g. en), they are also available formatted > > If you still have a question or problem, please take the output of > `uname -a', along with any relevant error messages, and email it > as a question to the questions@FreeBSD.org mailing list. If you are > unfamiliar with FreeBSD's directory layout, please refer to the hier(7) > manual page. If you are not familiar with manual pages, type `man man'. > > Edit /etc/motd to change this login announcement. > > You have new mail. > You have new mail. > root@cocopuff:~ # auto-logout > > FreeBSD/amd64 (cocopuff.macktronics.com) (console) > > login: panic: deadlkres: possible deadlock detected for 0xfffffe001c919920, > blocked for 303081 ticks > > cpuid = 0 > KDB: enter: panic > [ thread pid 0 tid 100027 ] > Stopped at kdb_enter+0x3e: movq $0,kdb_why > db> bt > Tracing pid 0 tid 100027 td 0xfffffe000358f490 > kdb_enter() at kdb_enter+0x3e/frame 0xffffff80002bcae0 > panic() at panic+0x176/frame 0xffffff80002bcb60 > deadlkres() at deadlkres+0x488/frame 0xffffff80002bcbb0 > fork_exit() at fork_exit+0x9a/frame 0xffffff80002bcbf0 > fork_trampoline() at fork_trampoline+0xe/frame 0xffffff80002bcbf0 > --- trap 0, rip = 0, rsp = 0xffffff80002bccb0, rbp = 0 --- > > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org" > From owner-freebsd-virtualization@FreeBSD.ORG Fri Mar 29 04:13:38 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 81006E5A for ; Fri, 29 Mar 2013 04:13:38 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 40BC5A22 for ; Fri, 29 Mar 2013 04:13:38 +0000 (UTC) Received: from dommail.onthenet.com.au (dommail.OntheNet.com.au [203.13.70.57]) by alto.onthenet.com.au (Postfix) with ESMTPS id B2A7011B1B; Fri, 29 Mar 2013 14:13:30 +1000 (EST) Received: from Peter-Grehans-MacBook-Pro.local (mobile-166-137-151-036.mycingular.net [166.137.151.36]) by dommail.onthenet.com.au (MOS 4.2.4-GA) with ESMTP id BLB93373 (AUTH peterg@ptree32.com.au); Fri, 29 Mar 2013 14:13:27 +1000 Message-ID: <515514E2.1080905@freebsd.org> Date: Thu, 28 Mar 2013 22:13:22 -0600 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.28) Gecko/20120306 Thunderbird/3.1.20 MIME-Version: 1.0 To: Dan Mack Subject: Re: bhyve tty / login problems / panic References: <20130328155008.B5464@coco.macktronics.com> In-Reply-To: <20130328155008.B5464@coco.macktronics.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2013 04:13:38 -0000 Hi Dan, > Any ideas? Just to confirm: your host system is an i7 3930k, which looks like a 6 core system. Should be plenty beefy enough for a 2 vCPU guest. The issue you are seeing looks a bit like what happens when systems are heavily oversubscribed and vCPUs can't get enough run time. How much spare RAM is on the host when you started the vm ? Also, is there much running on the host when the guest is started ? later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Fri Mar 29 19:19:20 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A7C8274; Fri, 29 Mar 2013 19:19:20 +0000 (UTC) (envelope-from mack@macktronics.com) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by mx1.freebsd.org (Postfix) with ESMTP id 5205D734; Fri, 29 Mar 2013 19:19:19 +0000 (UTC) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by coco.macktronics.com (Postfix) with ESMTP id 3D5844AC40; Fri, 29 Mar 2013 14:19:19 -0500 (CDT) Date: Fri, 29 Mar 2013 14:19:19 -0500 (CDT) From: Dan Mack To: Peter Grehan Subject: Re: bhyve tty / login problems / panic In-Reply-To: <515514E2.1080905@freebsd.org> Message-ID: <20130329140714.L12104@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2013 19:19:20 -0000 On Thu, 28 Mar 2013, Peter Grehan wrote: > Hi Dan, > >> Any ideas? > > Just to confirm: your host system is an i7 3930k, which looks like a 6 core > system. Should be plenty beefy enough for a 2 vCPU guest. That's what I thought, 6 cores, 12 threads, 64 GB RAM. > The issue you are seeing looks a bit like what happens when systems are > heavily oversubscribed and vCPUs can't get enough run time. I was running in on an idle system; only bhyve was running. > How much spare RAM is on the host when you started the vm ? Also, is there > much running on the host when the guest is started ? 64GB Here's some more information (sorry for the formatting): Case 1: single vCPU via: root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco Top shows it idle (note low load, 4GB memory given to guest, still 53GB free RAM, nothing else clocking CPU cycles): root@olive:~ # top -bH last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 14:11:51 24 processes: 1 running, 23 sleeping Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other Swap: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% bhyve{coco vcpu 0} 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% bhyve{coco mevent} So, case #1 all is normal. I was able to build a kernel with very good performance. The boot sequence looks like the following: root@olive:/vms/coco # ./vmrun.sh coco Launching virtual machine "coco" ... Consoles: userboot FreeBSD/amd64 User boot, Revision 1.1 (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08] / ______ ____ _____ _____ | ____| | _ \ / ____| __ \ | |___ _ __ ___ ___ | |_) | (___ | | | | | ___| '__/ _ \/ _ \| _ < \___ \| | | | | | | | | __/ __/| |_) |____) | |__| | | | | | | | || | | | |_| |_| \___|\___||____/|_____/|_____/ ``` ` s` `.....---.......--.``` -/ +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. | | yo`:. :o `+- | 1. Boot Multi User [Enter] | y/ -/` -o/ | 2. Boot [S]ingle User | .- ::/sy+:. | 3. [Esc]ape to loader prompt | / `-- / | 4. Reboot | `: :` | | `: :` | Options: | / / | 5. Configure Boot [O]ptions... | .- -. | | -- -. | | `:` `:` | | .-- `--. | | .---.....----. +-----------------------------------------+ Booting... Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64 FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0x8fa3ab7f Features2=0x83bee217 AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant real memory = 5368709120 (5120 MB) avail memory = 3957534720 (3774 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: random device not loaded; using insecure entropy ioapic0: Changing APIC ID to 1 ioapic0 irqs 0-16 on motherboard kbd0 at kbdmux0 acpi0: on motherboard atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib0: no PRT entry for 0.31.INTA virtio_pci0: port 0x2000-0x201f mem 0xc0000000-0xc0001fff at device 1.0 on pci0 vtnet0: on virtio_pci0 virtio_pci0: host features: 0x18020 virtio_pci0: negotiated features: 0x18020 vtnet0: Ethernet address: 00:a0:98:ac:0d:df virtio_pci1: port 0x2040-0x207f mem 0xc0002000-0xc0003fff at device 2.0 on pci0 vtblk0: on virtio_pci1 virtio_pci1: host features: 0x10000004 virtio_pci1: negotiated features: 0x10000004 vtblk0: 8192MB (16777216 512 byte sectors) uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 uart2: console (9600,n,8,1) Timecounters tick every 10.000 msec g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 Trying to mount root from ufs:/dev/vtbd0p2 [rw]... Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf. Setting hostid: 0x24bfd025. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% fragmentation) g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT Mounting local file systems:. Writing entropy file:. Setting hostname: cocopuff.example.com. vtnet0: link state changed to UP Starting Network: lo0 vtnet0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 vtnet0: flags=8943 metric 0 mtu 1500 options=80028 ether 00:a0:98:ac:0d:df inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet 1000baseT status: active Starting devd. add net default: gateway 172.22.22.252 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. Clearing /tmp (X related). Updating motd:. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Fri Mar 29 14:09:20 CDT 2013 FreeBSD/amd64 (cocopuff.example.com) (console) login: ====== Next, boot the same image with two (2) vCPU: Command run by vmrun.sh: root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco Top shows it running out of control: root@olive:~ # top -bH last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 14:17:32 25 processes: 3 running, 22 sleeping Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other Swap: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% bhyve{coco vcpu 1} 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% bhyve{coco vcpu 0} 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% bhyve{coco mevent} 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh Even though, it's just sitting at the login prompt: root@olive:/vms/coco # ./vmrun.sh coco Launching virtual machine "coco" ... Consoles: userboot FreeBSD/amd64 User boot, Revision 1.1 (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08] / ______ ____ _____ _____ | ____| | _ \ / ____| __ \ | |___ _ __ ___ ___ | |_) | (___ | | | | | ___| '__/ _ \/ _ \| _ < \___ \| | | | | | | | | __/ __/| |_) |____) | |__| | | | | | | | || | | | |_| |_| \___|\___||____/|_____/|_____/ ``` ` s` `.....---.......--.``` -/ +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. | | yo`:. :o `+- | 1. Boot Multi User [Enter] | y/ -/` -o/ | 2. Boot [S]ingle User | .- ::/sy+:. | 3. [Esc]ape to loader prompt | / `-- / | 4. Reboot | `: :` | | `: :` | Options: | / / | 5. Configure Boot [O]ptions... | .- -. | | -- -. | | `:` `:` | | .-- `--. | | .---.....----. +-----------------------------------------+ Booting... Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64 FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0x8fa3ab7f Features2=0x83bee217 AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant real memory = 5368709120 (5120 MB) avail memory = 3957534720 (3774 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 2 package(s) x 1 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 random device not loaded; using insecure entropy ioapic0: Changing APIC ID to 2 ioapic0 irqs 0-16 on motherboard kbd0 at kbdmux0 acpi0: on motherboard atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib0: no PRT entry for 0.31.INTA virtio_pci0: port 0x2000-0x201f mem 0xc0000000-0xc0001fff at device 1.0 on pci0 vtnet0: on virtio_pci0 virtio_pci0: host features: 0x18020 virtio_pci0: negotiated features: 0x18020 vtnet0: Ethernet address: 00:a0:98:ac:0d:df virtio_pci1: port 0x2040-0x207f mem 0xc0002000-0xc0003fff at device 2.0 on pci0 vtblk0: on virtio_pci1 virtio_pci1: host features: 0x10000004 virtio_pci1: negotiated features: 0x10000004 vtblk0: 8192MB (16777216 512 byte sectors) uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 uart2: console (9600,n,8,1) Timecounters tick every 10.000 msec g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/vtbd0p2 [rw]... Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf. Setting hostid: 0x24bfd025. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% fragmentation) g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT Mounting local file systems:. Writing entropy file:. Setting hostname: cocopuff.example.com. vtnet0: link state changed to UP Starting Network: lo0 vtnet0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 vtnet0: flags=8943 metric 0 mtu 1500 options=80028 ether 00:a0:98:ac:0d:df inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet 1000baseT status: active Starting devd. add net default: gateway 172.22.22.252 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. syslogd: timed out waiting for child /etc/rc: WARNING: failed to start syslogd Clearing /tmp (X related). Updating motd:. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Thu Apr 18 02:06:43 CDT 2013 FreeBSD/amd64 (cocopuff.example.com) (console) login: ========== end of boot sequence So, something is broken with SMP on the guest, I'd be happy to help just let me know. Dan From owner-freebsd-virtualization@FreeBSD.ORG Fri Mar 29 19:38:32 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 182F1597; Fri, 29 Mar 2013 19:38:32 +0000 (UTC) (envelope-from mack@macktronics.com) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by mx1.freebsd.org (Postfix) with ESMTP id B662B7DC; Fri, 29 Mar 2013 19:38:31 +0000 (UTC) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by coco.macktronics.com (Postfix) with ESMTP id 241D64AC40; Fri, 29 Mar 2013 14:38:31 -0500 (CDT) Date: Fri, 29 Mar 2013 14:38:31 -0500 (CDT) From: Dan Mack To: Peter Grehan Subject: Re: bhyve tty / login problems / panic In-Reply-To: <20130329140714.L12104@coco.macktronics.com> Message-ID: <20130329143127.W12104@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2013 19:38:32 -0000 I ran the procsystime dtrace script on bhyve when it was chewing up all the CPU in vCPU=2 mode, and this is what I see for about 10s of runtime: root@olive:/usr/share/dtrace/toolkit # ./procsystime -n bhyve -aT Tracing... Hit Ctrl-C to end... dtrace: 158536 dynamic variable drops with non-empty dirty list dtrace: 207447 dynamic variable drops with non-empty dirty list dtrace: 189205 dynamic variable drops with non-empty dirty list dtrace: 164341 dynamic variable drops with non-empty dirty list dtrace: 246307 dynamic variable drops with non-empty dirty list dtrace: 187640 dynamic variable drops with non-empty dirty list dtrace: 214771 dynamic variable drops with non-empty dirty list dtrace: 221265 dynamic variable drops with non-empty dirty list ^C dtrace: 243468 dynamic variable drops with non-empty dirty list Elapsed Times for processes bhyve, SYSCALL TIME (ns) _umtx_op 18349 writev 135712 preadv 16175267 pwritev 22924378 ioctl 4353897920 TOTAL: 4393151626 CPU Times for processes bhyve, SYSCALL TIME (ns) _umtx_op 8815 writev 103145 pwritev 10647023 preadv 15159976 ioctl 3943399889 TOTAL: 3969318848 Syscall Counts for processes bhyve, SYSCALL COUNT _umtx_op 17 writev 43 pwritev 639 preadv 1183 ioctl 652782 TOTAL: 654664 Not sure if that helps or not. Hotkernel shows the cpu in the kernel`acpi_cpu_c1 call most of the time, normal ? Dan root@olive:/usr/share/dtrace/toolkit # ./hotkernel Sampling... Hit Ctrl-C to end. ^C FUNCTION COUNT PCNT zfs.ko`space_map_sync 1 0.0% kernel`syscall_thread_exit 1 0.0% kernel`should_yield 1 0.0% zfs.ko`lzjb_compress 1 0.0% zfs.ko`dbuf_rele 1 0.0% zfs.ko`dsl_pool_tempreserve_space 1 0.0% kernel`_bus_dmamap_load_buffer 1 0.0% kernel`cpu_idleclock 1 0.0% kernel`_rw_runlock_cookie 1 0.0% kernel`pmap_enter 1 0.0% kernel`__lockmgr_args 1 0.0% zfs.ko`vdev_geom_io_start 1 0.0% kernel`ahci_ch_intr 1 0.0% kernel`__rw_rlock 1 0.0% kernel`VOP_UNLOCK_APV 1 0.0% zfs.ko`arc_buf_thaw 1 0.0% kernel`atomic_set_int 1 0.0% kernel`g_io_schedule_down 1 0.0% kernel`__mtx_lock_flags 1 0.0% kernel`_callout_stop_safe 1 0.0% kernel`sched_idletd 1 0.0% kernel`_sx_slock 1 0.0% vmm.ko`lapic_set_intr 1 0.0% kernel`cpusetobj_ffs 1 0.0% zfs.ko`dbuf_hold 1 0.0% vmm.ko`vlapic_intr_accepted 1 0.0% zfs.ko`sa_attr_op 1 0.0% zfs.ko`dmu_buf_hold_array_by_dnode 1 0.0% zfs.ko`taskq_run_safe 1 0.0% kernel`vm_reserv_free_page 1 0.0% kernel`_sx_sunlock 1 0.0% kernel`vn_rlimit_fsize 1 0.0% zfs.ko`vdev_queue_io_to_issue 1 0.0% kernel`vn_write 1 0.0% kernel`lock_init 1 0.0% vmm.ko`vmx_resume 1 0.0% zfs.ko`zio_gang_tree_free 1 0.0% zfs.ko`dbuf_rele_and_unlock 1 0.0% kernel`mnt_vnode_next_active 1 0.0% kernel`vm_page_remove 1 0.0% zfs.ko`vdev_queue_deadline_compare 1 0.0% kernel`zone_free_item 1 0.0% vmm.ko`vmx_launch 1 0.0% kernel`m_pullup 1 0.0% kernel`pmap_change_wiring 1 0.0% kernel`sched_userret 1 0.0% zfs.ko`dbuf_write_done 1 0.0% zfs.ko`zio_execute 1 0.0% kernel`PHYS_TO_VM_PAGE 2 0.0% zfs.ko`avl_rotation 2 0.0% kernel`ehci_device_isoc_fs_enter 2 0.0% vmm.ko`vm_exitinfo 2 0.0% zfs.ko`zfs_freebsd_write 2 0.0% vmm.ko`vcpu_stats 2 0.0% kernel`atomic_add_long 2 0.0% vmm.ko`0xffffffff81 2 0.0% kernel`vm_reserv_alloc_page 2 0.0% kernel`pagezero 2 0.0% kernel`bzero 2 0.0% zfs.ko`fletcher_2_native 2 0.0% vmm.ko`vmx_getreg 2 0.0% vmm.ko`vm_guest_msrs 3 0.0% vmm.ko`vmx_setreg 3 0.0% vmm.ko`vm_get_register 3 0.0% kernel`sx_try_xlock_ 3 0.0% kernel`vm_radix_lookup_le 3 0.0% vmm.ko`lapic_pending_intr 4 0.0% kernel`syscall_thread_enter 4 0.0% kernel`cpu_set_syscall_retval 4 0.0% kernel`__mnt_vnode_next_all 4 0.0% kernel`vfs_stdsync 4 0.0% vmm.ko`vmm_fetch_instruction 5 0.0% vmm.ko`vlapic_op_mem_read 6 0.0% kernel`_sx_xunlock 7 0.0% vmm.ko`vmm_emulate_instruction 7 0.0% vmm.ko`vlapic_op_mem_write 7 0.0% kernel`__mtx_lock_spin_flags 8 0.0% vmm.ko`lapic_timer_tick 8 0.0% zfs.ko`list_next 8 0.0% zfs.ko`l2arc_feed_thread 8 0.0% kernel`userret 9 0.0% vmm.ko`vm_gpa2hpa 9 0.0% vmm.ko`vmm_decode_instruction 11 0.0% vmm.ko`ept_vmmmap_get 11 0.0% vmm.ko`vm_lapic 11 0.0% kernel`spinlock_enter 14 0.0% kernel`__mtx_lock_sleep 15 0.0% vmm.ko`vcpu_set_state 16 0.0% vmm.ko`vm_nmi_pending 16 0.0% kernel`malloc_type_zone_allocated 16 0.0% kernel`devfs_ioctl_f 16 0.0% kernel`pmap_kextract 17 0.0% kernel`sys_ioctl 18 0.0% kernel`malloc 21 0.0% kernel`cpu_fetch_syscall_args 24 0.0% kernel`uma_zfree_arg 28 0.0% vmm.ko`vlapic_pending_intr 29 0.0% kernel`trap 30 0.0% kernel`hpet_get_timecount 31 0.0% dtrace.ko`dtrace_trap 35 0.0% vmm.ko`vlapic_update_ppr 39 0.0% vmm.ko`vmmdev_ioctl 40 0.0% kernel`copyout 43 0.0% kernel`critical_enter 44 0.0% kernel`copyin 51 0.1% vmm.ko`vmcs_read 53 0.1% kernel`fpusave 61 0.1% kernel`uma_zalloc_arg 62 0.1% kernel`devvn_refthread 69 0.1% kernel`atomic_load_acq_int 70 0.1% kernel`fpurestore 70 0.1% vmm.ko`vlapic_timer_tick 75 0.1% kernel`amd64_syscall 77 0.1% kernel`bcopy 91 0.1% kernel`fpudna 103 0.1% kernel`free 103 0.1% kernel`critical_exit 105 0.1% kernel`dev_relthread 108 0.1% kernel`spinlock_exit 119 0.1% kernel`kern_ioctl 138 0.1% kernel`fpuexit 242 0.3% kernel`0xffffffff80 306 0.3% vmm.ko`restore_host_msrs 398 0.4% vmm.ko`vm_run 428 0.4% vmm.ko`restore_guest_msrs 433 0.5% kernel`cpu_idle 906 1.0% vmm.ko`vmx_run 9266 9.7% kernel`acpi_cpu_c1 81134 85.2% Hope this helps, let me know if you have any ideas on what to try next. Dan On Fri, 29 Mar 2013, Dan Mack wrote: > On Thu, 28 Mar 2013, Peter Grehan wrote: > >> Hi Dan, >> >>> Any ideas? >> >> Just to confirm: your host system is an i7 3930k, which looks like a 6 core >> system. Should be plenty beefy enough for a 2 vCPU guest. > > That's what I thought, 6 cores, 12 threads, 64 GB RAM. > >> The issue you are seeing looks a bit like what happens when systems are >> heavily oversubscribed and vCPUs can't get enough run time. > > I was running in on an idle system; only bhyve was running. > > >> How much spare RAM is on the host when you started the vm ? Also, is there >> much running on the host when the guest is started ? > > 64GB > > Here's some more information (sorry for the formatting): > > Case 1: single vCPU via: > root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 > /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s > 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco > > Top shows it idle (note low load, 4GB memory given to guest, still 53GB free > RAM, nothing else clocking CPU cycles): > > root@olive:~ # top -bH > last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 > 14:11:51 > 24 processes: 1 running, 23 sleeping > > Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free > ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other > Swap: > > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% bhyve{coco > vcpu 0} > 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd > 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd > 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh > 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% bhyve{coco > mevent} > > > So, case #1 all is normal. I was able to build a kernel with very good > performance. > > The boot sequence looks like the following: > > root@olive:/vms/coco # ./vmrun.sh coco > Launching virtual machine "coco" ... > Consoles: userboot > > FreeBSD/amd64 User boot, Revision 1.1 > (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) > Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 > data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08] > / > ______ ____ _____ _____ > | ____| | _ \ / ____| __ \ > | |___ _ __ ___ ___ | |_) | (___ | | | | > | ___| '__/ _ \/ _ \| _ < \___ \| | | | > | | | | | __/ __/| |_) |____) | |__| | > | | | | | | || | | | > |_| |_| \___|\___||____/|_____/|_____/ ``` ` > s` `.....---.......--.``` -/ > +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. > | | yo`:. :o `+- > | 1. Boot Multi User [Enter] | y/ -/` -o/ > | 2. Boot [S]ingle User | .- ::/sy+:. > | 3. [Esc]ape to loader prompt | / `-- / > | 4. Reboot | `: :` > | | `: :` > | Options: | / / > | 5. Configure Boot [O]ptions... | .- -. > | | -- -. > | | `:` `:` > | | .-- `--. > | | .---.....----. > +-----------------------------------------+ > > > Booting... > Copyright (c) 1992-2013 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 > root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64 > FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 > CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU) > Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping > = 7 > Features=0x8fa3ab7f > Features2=0x83bee217 > AMD Features=0x2c100800 > AMD Features2=0x1 > TSC: P-state invariant > real memory = 5368709120 (5120 MB) > avail memory = 3957534720 (3774 MB) > Event timer "LAPIC" quality 400 > ACPI APIC Table: > random device not loaded; using insecure entropy > ioapic0: Changing APIC ID to 1 > ioapic0 irqs 0-16 on motherboard > kbd0 at kbdmux0 > acpi0: on motherboard > atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib0: no PRT entry for 0.31.INTA > virtio_pci0: port 0x2000-0x201f mem > 0xc0000000-0xc0001fff at device 1.0 on pci0 > vtnet0: on virtio_pci0 > virtio_pci0: host features: 0x18020 > virtio_pci0: negotiated features: 0x18020 > vtnet0: Ethernet address: 00:a0:98:ac:0d:df > virtio_pci1: port 0x2040-0x207f mem > 0xc0002000-0xc0003fff at device 2.0 on pci0 > vtblk0: on virtio_pci1 > virtio_pci1: host features: 0x10000004 > virtio_pci1: negotiated features: 0x10000004 > vtblk0: 8192MB (16777216 512 byte sectors) > uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 > uart2: console (9600,n,8,1) > Timecounters tick every 10.000 msec > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 > Trying to mount root from ufs:/dev/vtbd0p2 [rw]... > Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf. > Setting hostid: 0x24bfd025. > Entropy harvesting: interrupts ethernet point_to_point kickstart. > Starting file system checks: > /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS > /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% > fragmentation) > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > Mounting local file systems:. > Writing entropy file:. > Setting hostname: cocopuff.example.com. > vtnet0: link state changed to UP > Starting Network: lo0 vtnet0. > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > vtnet0: flags=8943 metric 0 > mtu 1500 > options=80028 > ether 00:a0:98:ac:0d:df > inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 > inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 > nd6 options=29 > media: Ethernet 1000baseT > status: active > Starting devd. > add net default: gateway 172.22.22.252 > add net ::ffff:0.0.0.0: gateway ::1 > add net ::0.0.0.0: gateway ::1 > add net fe80::: gateway ::1 > add net ff02::: gateway ::1 > ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib > 32-bit compatibility ldconfig path: /usr/lib32 > Creating and/or trimming log files. > Starting syslogd. > Clearing /tmp (X related). > Updating motd:. > Performing sanity check on sshd configuration. > Starting sshd. > Starting cron. > Starting background file system checks in 60 seconds. > > Fri Mar 29 14:09:20 CDT 2013 > > FreeBSD/amd64 (cocopuff.example.com) (console) > > login: > > > ====== Next, boot the same image with two (2) vCPU: > > Command run by vmrun.sh: > > root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 > /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s > 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco > > Top shows it running out of control: > > root@olive:~ # top -bH > last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 > 14:17:32 > 25 processes: 3 running, 22 sleeping > > Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free > ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other > Swap: > > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% bhyve{coco > vcpu 1} > 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% bhyve{coco > vcpu 0} > 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd > 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd > 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd > 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron > 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd > 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh > 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% bhyve{coco > mevent} > 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh > > Even though, it's just sitting at the login prompt: > > root@olive:/vms/coco # ./vmrun.sh coco > Launching virtual machine "coco" ... > Consoles: userboot > > FreeBSD/amd64 User boot, Revision 1.1 > (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) > Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 > data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08] > / > ______ ____ _____ _____ > | ____| | _ \ / ____| __ \ > | |___ _ __ ___ ___ | |_) | (___ | | | | > | ___| '__/ _ \/ _ \| _ < \___ \| | | | > | | | | | __/ __/| |_) |____) | |__| | > | | | | | | || | | | > |_| |_| \___|\___||____/|_____/|_____/ ``` ` > s` `.....---.......--.``` -/ > +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. > | | yo`:. :o `+- > | 1. Boot Multi User [Enter] | y/ -/` -o/ > | 2. Boot [S]ingle User | .- ::/sy+:. > | 3. [Esc]ape to loader prompt | / `-- / > | 4. Reboot | `: :` > | | `: :` > | Options: | / / > | 5. Configure Boot [O]ptions... | .- -. > | | -- -. > | | `:` `:` > | | .-- `--. > | | .---.....----. > +-----------------------------------------+ > > > Booting... > Copyright (c) 1992-2013 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 > root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64 > FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 > CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU) > Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping > = 7 > Features=0x8fa3ab7f > Features2=0x83bee217 > AMD Features=0x2c100800 > AMD Features2=0x1 > TSC: P-state invariant > real memory = 5368709120 (5120 MB) > avail memory = 3957534720 (3774 MB) > Event timer "LAPIC" quality 400 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > FreeBSD/SMP: 2 package(s) x 1 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > random device not loaded; using insecure entropy > ioapic0: Changing APIC ID to 2 > ioapic0 irqs 0-16 on motherboard > kbd0 at kbdmux0 > acpi0: on motherboard > atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib0: no PRT entry for 0.31.INTA > virtio_pci0: port 0x2000-0x201f mem > 0xc0000000-0xc0001fff at device 1.0 on pci0 > vtnet0: on virtio_pci0 > virtio_pci0: host features: 0x18020 > virtio_pci0: negotiated features: 0x18020 > vtnet0: Ethernet address: 00:a0:98:ac:0d:df > virtio_pci1: port 0x2040-0x207f mem > 0xc0002000-0xc0003fff at device 2.0 on pci0 > vtblk0: on virtio_pci1 > virtio_pci1: host features: 0x10000004 > virtio_pci1: negotiated features: 0x10000004 > vtblk0: 8192MB (16777216 512 byte sectors) > uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 > uart2: console (9600,n,8,1) > Timecounters tick every 10.000 msec > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > SMP: AP CPU #1 Launched! > Trying to mount root from ufs:/dev/vtbd0p2 [rw]... > Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf. > Setting hostid: 0x24bfd025. > Entropy harvesting: interrupts ethernet point_to_point kickstart. > Starting file system checks: > /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS > /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% > fragmentation) > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT > Mounting local file systems:. > Writing entropy file:. > Setting hostname: cocopuff.example.com. > vtnet0: link state changed to UP > Starting Network: lo0 vtnet0. > lo0: flags=8049 metric 0 mtu 16384 > options=600003 > inet6 ::1 prefixlen 128 > inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 > inet 127.0.0.1 netmask 0xff000000 > nd6 options=21 > vtnet0: flags=8943 metric 0 > mtu 1500 > options=80028 > ether 00:a0:98:ac:0d:df > inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 > inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 > nd6 options=29 > media: Ethernet 1000baseT > status: active > Starting devd. > add net default: gateway 172.22.22.252 > add net ::ffff:0.0.0.0: gateway ::1 > add net ::0.0.0.0: gateway ::1 > add net fe80::: gateway ::1 > add net ff02::: gateway ::1 > ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib > 32-bit compatibility ldconfig path: /usr/lib32 > Creating and/or trimming log files. > Starting syslogd. > syslogd: timed out waiting for child > /etc/rc: WARNING: failed to start syslogd > Clearing /tmp (X related). > Updating motd:. > Performing sanity check on sshd configuration. > Starting sshd. > Starting cron. > Starting background file system checks in 60 seconds. > > Thu Apr 18 02:06:43 CDT 2013 > > FreeBSD/amd64 (cocopuff.example.com) (console) > > login: > > > ========== end of boot sequence > > So, something is broken with SMP on the guest, I'd be happy to help just let > me know. > > Dan > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org" > From owner-freebsd-virtualization@FreeBSD.ORG Fri Mar 29 19:41:56 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 892C169C; Fri, 29 Mar 2013 19:41:56 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-ie0-x234.google.com (mail-ie0-x234.google.com [IPv6:2607:f8b0:4001:c03::234]) by mx1.freebsd.org (Postfix) with ESMTP id 4DFC27FC; Fri, 29 Mar 2013 19:41:56 +0000 (UTC) Received: by mail-ie0-f180.google.com with SMTP id a11so847030iee.25 for ; Fri, 29 Mar 2013 12:41:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=ExFFq9aiip5+4BOduRJIm7buU8DYdsyD+ZAdaaIcX0Y=; b=isRK58Ny8Y6Z5NCmtgldBN7HLk2BAjkAAzQbz4jkKEafFldMPsuDBp5qsarmV0Hw12 qTm7G6QDOm8jFu4H6bs6vWB77LPsCZZExOA6qzoioaoLs4nQ/bQWbyAgUmI40dNHzZ8c +dIz/uaXwSZnZ9zeUvgeIh7/RBt7koPuzFzcBhfZnhwRVe2ZgiITz7fwhvF1Fq2l4ork i0QyAEQirpeyaft0z/9ahEU9NA43U6KtJs3JedHlHwm8F2QFxPzK0B/3f6D/0j+UNEGV di+fieYSxdArVDP+Ac0GSwr7rHfMqWav4Y7iyCj3/fxVPs+79TuznUYSIMlcQpIdvzA5 3lYg== MIME-Version: 1.0 X-Received: by 10.50.36.199 with SMTP id s7mr444393igj.56.1364586115898; Fri, 29 Mar 2013 12:41:55 -0700 (PDT) Received: by 10.42.23.132 with HTTP; Fri, 29 Mar 2013 12:41:55 -0700 (PDT) In-Reply-To: <20130329143127.W12104@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> Date: Fri, 29 Mar 2013 12:41:55 -0700 Message-ID: Subject: Re: bhyve tty / login problems / panic From: Neel Natu To: Dan Mack Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2013 19:41:56 -0000 Hi Dan, On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack wrote: > > I ran the procsystime dtrace script on bhyve when it was chewing up all > the CPU in vCPU=2 mode, and this is what I see for about 10s of runtime: > > root@olive:/usr/share/dtrace/**toolkit # ./procsystime -n bhyve -aT > Tracing... Hit Ctrl-C to end... > dtrace: 158536 dynamic variable drops with non-empty dirty list > dtrace: 207447 dynamic variable drops with non-empty dirty list > dtrace: 189205 dynamic variable drops with non-empty dirty list > dtrace: 164341 dynamic variable drops with non-empty dirty list > dtrace: 246307 dynamic variable drops with non-empty dirty list > dtrace: 187640 dynamic variable drops with non-empty dirty list > dtrace: 214771 dynamic variable drops with non-empty dirty list > dtrace: 221265 dynamic variable drops with non-empty dirty list > ^C > dtrace: 243468 dynamic variable drops with non-empty dirty list > > Elapsed Times for processes bhyve, > > SYSCALL TIME (ns) > _umtx_op 18349 > writev 135712 > preadv 16175267 > pwritev 22924378 > ioctl 4353897920 > TOTAL: 4393151626 > > CPU Times for processes bhyve, > > SYSCALL TIME (ns) > _umtx_op 8815 > writev 103145 > pwritev 10647023 > preadv 15159976 > ioctl 3943399889 > TOTAL: 3969318848 > > Syscall Counts for processes bhyve, > > SYSCALL COUNT > _umtx_op 17 > writev 43 > pwritev 639 > preadv 1183 > ioctl 652782 > TOTAL: 654664 > > Not sure if that helps or not. Hotkernel shows the cpu in the > kernel`acpi_cpu_c1 call most of the time, normal ? > > Dan > > root@olive:/usr/share/dtrace/**toolkit # ./hotkernel > Sampling... Hit Ctrl-C to end. > ^C > FUNCTION COUNT PCNT > zfs.ko`space_map_sync 1 0.0% > kernel`syscall_thread_exit 1 0.0% > kernel`should_yield 1 0.0% > zfs.ko`lzjb_compress 1 0.0% > zfs.ko`dbuf_rele 1 0.0% > zfs.ko`dsl_pool_tempreserve_**space 1 0.0% > kernel`_bus_dmamap_load_buffer 1 0.0% > kernel`cpu_idleclock 1 0.0% > kernel`_rw_runlock_cookie 1 0.0% > kernel`pmap_enter 1 0.0% > kernel`__lockmgr_args 1 0.0% > zfs.ko`vdev_geom_io_start 1 0.0% > kernel`ahci_ch_intr 1 0.0% > kernel`__rw_rlock 1 0.0% > kernel`VOP_UNLOCK_APV 1 0.0% > zfs.ko`arc_buf_thaw 1 0.0% > kernel`atomic_set_int 1 0.0% > kernel`g_io_schedule_down 1 0.0% > kernel`__mtx_lock_flags 1 0.0% > kernel`_callout_stop_safe 1 0.0% > kernel`sched_idletd 1 0.0% > kernel`_sx_slock 1 0.0% > vmm.ko`lapic_set_intr 1 0.0% > kernel`cpusetobj_ffs 1 0.0% > zfs.ko`dbuf_hold 1 0.0% > vmm.ko`vlapic_intr_accepted 1 0.0% > zfs.ko`sa_attr_op 1 0.0% > zfs.ko`dmu_buf_hold_array_by_**dnode 1 0.0% > zfs.ko`taskq_run_safe 1 0.0% > kernel`vm_reserv_free_page 1 0.0% > kernel`_sx_sunlock 1 0.0% > kernel`vn_rlimit_fsize 1 0.0% > zfs.ko`vdev_queue_io_to_issue 1 0.0% > kernel`vn_write 1 0.0% > kernel`lock_init 1 0.0% > vmm.ko`vmx_resume 1 0.0% > zfs.ko`zio_gang_tree_free 1 0.0% > zfs.ko`dbuf_rele_and_unlock 1 0.0% > kernel`mnt_vnode_next_active 1 0.0% > kernel`vm_page_remove 1 0.0% > zfs.ko`vdev_queue_deadline_**compare 1 0.0% > kernel`zone_free_item 1 0.0% > vmm.ko`vmx_launch 1 0.0% > kernel`m_pullup 1 0.0% > kernel`pmap_change_wiring 1 0.0% > kernel`sched_userret 1 0.0% > zfs.ko`dbuf_write_done 1 0.0% > zfs.ko`zio_execute 1 0.0% > kernel`PHYS_TO_VM_PAGE 2 0.0% > zfs.ko`avl_rotation 2 0.0% > kernel`ehci_device_isoc_fs_**enter 2 0.0% > vmm.ko`vm_exitinfo 2 0.0% > zfs.ko`zfs_freebsd_write 2 0.0% > vmm.ko`vcpu_stats 2 0.0% > kernel`atomic_add_long 2 0.0% > vmm.ko`0xffffffff81 2 0.0% > kernel`vm_reserv_alloc_page 2 0.0% > kernel`pagezero 2 0.0% > kernel`bzero 2 0.0% > zfs.ko`fletcher_2_native 2 0.0% > vmm.ko`vmx_getreg 2 0.0% > vmm.ko`vm_guest_msrs 3 0.0% > vmm.ko`vmx_setreg 3 0.0% > vmm.ko`vm_get_register 3 0.0% > kernel`sx_try_xlock_ 3 0.0% > kernel`vm_radix_lookup_le 3 0.0% > vmm.ko`lapic_pending_intr 4 0.0% > kernel`syscall_thread_enter 4 0.0% > kernel`cpu_set_syscall_retval 4 0.0% > kernel`__mnt_vnode_next_all 4 0.0% > kernel`vfs_stdsync 4 0.0% > vmm.ko`vmm_fetch_instruction 5 0.0% > vmm.ko`vlapic_op_mem_read 6 0.0% > kernel`_sx_xunlock 7 0.0% > vmm.ko`vmm_emulate_instruction 7 0.0% > vmm.ko`vlapic_op_mem_write 7 0.0% > kernel`__mtx_lock_spin_flags 8 0.0% > vmm.ko`lapic_timer_tick 8 0.0% > zfs.ko`list_next 8 0.0% > zfs.ko`l2arc_feed_thread 8 0.0% > kernel`userret 9 0.0% > vmm.ko`vm_gpa2hpa 9 0.0% > vmm.ko`vmm_decode_instruction 11 0.0% > vmm.ko`ept_vmmmap_get 11 0.0% > vmm.ko`vm_lapic 11 0.0% > kernel`spinlock_enter 14 0.0% > kernel`__mtx_lock_sleep 15 0.0% > vmm.ko`vcpu_set_state 16 0.0% > vmm.ko`vm_nmi_pending 16 0.0% > kernel`malloc_type_zone_**allocated 16 0.0% > kernel`devfs_ioctl_f 16 0.0% > kernel`pmap_kextract 17 0.0% > kernel`sys_ioctl 18 0.0% > kernel`malloc 21 0.0% > kernel`cpu_fetch_syscall_args 24 0.0% > kernel`uma_zfree_arg 28 0.0% > vmm.ko`vlapic_pending_intr 29 0.0% > kernel`trap 30 0.0% > kernel`hpet_get_timecount 31 0.0% > dtrace.ko`dtrace_trap 35 0.0% > vmm.ko`vlapic_update_ppr 39 0.0% > vmm.ko`vmmdev_ioctl 40 0.0% > kernel`copyout 43 0.0% > kernel`critical_enter 44 0.0% > kernel`copyin 51 0.1% > vmm.ko`vmcs_read 53 0.1% > kernel`fpusave 61 0.1% > kernel`uma_zalloc_arg 62 0.1% > kernel`devvn_refthread 69 0.1% > kernel`atomic_load_acq_int 70 0.1% > kernel`fpurestore 70 0.1% > vmm.ko`vlapic_timer_tick 75 0.1% > kernel`amd64_syscall 77 0.1% > kernel`bcopy 91 0.1% > kernel`fpudna 103 0.1% > kernel`free 103 0.1% > kernel`critical_exit 105 0.1% > kernel`dev_relthread 108 0.1% > kernel`spinlock_exit 119 0.1% > kernel`kern_ioctl 138 0.1% > kernel`fpuexit 242 0.3% > kernel`0xffffffff80 306 0.3% > vmm.ko`restore_host_msrs 398 0.4% > vmm.ko`vm_run 428 0.4% > vmm.ko`restore_guest_msrs 433 0.5% > kernel`cpu_idle 906 1.0% > vmm.ko`vmx_run 9266 9.7% > kernel`acpi_cpu_c1 81134 85.2% > > Hope this helps, let me know if you have any ideas on what to try next. > > Could you try the following command to get vcpu stats? sudo bhyvectl --vm=vm1 --get-stats --cpu=0 sudo bhyvectl --vm=vm1 --get-stats --cpu=1 best Neel > Dan > > > > > On Fri, 29 Mar 2013, Dan Mack wrote: > > On Thu, 28 Mar 2013, Peter Grehan wrote: >> >> Hi Dan, >>> >>> Any ideas? >>>> >>> >>> Just to confirm: your host system is an i7 3930k, which looks like a 6 >>> core system. Should be plenty beefy enough for a 2 vCPU guest. >>> >> >> That's what I thought, 6 cores, 12 threads, 64 GB RAM. >> >> The issue you are seeing looks a bit like what happens when systems are >>> heavily oversubscribed and vCPUs can't get enough run time. >>> >> >> I was running in on an idle system; only bhyve was running. >> >> >> How much spare RAM is on the host when you started the vm ? Also, is >>> there much running on the host when the guest is started ? >>> >> >> 64GB >> >> Here's some more information (sorry for the formatting): >> >> Case 1: single vCPU via: >> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 >> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >> >> Top shows it idle (note low load, 4GB memory given to guest, still 53GB >> free RAM, nothing else clocking CPU cycles): >> >> root@olive:~ # top -bH >> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 >> 14:11:51 >> 24 processes: 1 running, 23 sleeping >> >> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free >> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other >> Swap: >> >> >> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% >> bhyve{coco vcpu 0} >> 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd >> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% >> bhyve{coco mevent} >> >> >> So, case #1 all is normal. I was able to build a kernel with very good >> performance. >> >> The boot sequence looks like the following: >> >> root@olive:/vms/coco # ./vmrun.sh coco >> Launching virtual machine "coco" ... >> Consoles: userboot >> >> FreeBSD/amd64 User boot, Revision 1.1 >> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+**0x12bb08] >> / >> ______ ____ _____ _____ >> | ____| | _ \ / ____| __ \ >> | |___ _ __ ___ ___ | |_) | (___ | | | | >> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >> | | | | | __/ __/| |_) |____) | |__| | >> | | | | | | || | | | >> |_| |_| \___|\___||____/|_____/|_____/ ``` ` >> s` `.....---.......--.``` -/ >> +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. >> | | yo`:. :o `+- >> | 1. Boot Multi User [Enter] | y/ -/` -o/ >> | 2. Boot [S]ingle User | .- ::/sy+:. >> | 3. [Esc]ape to loader prompt | / `-- / >> | 4. Reboot | `: :` >> | | `: :` >> | Options: | / / >> | 5. Configure Boot [O]ptions... | .- -. >> | | -- -. >> | | `:` `:` >> | | .-- `--. >> | | .---.....----. >> +-----------------------------**------------+ >> >> >> Booting... >> Copyright (c) 1992-2013 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >> root@cocopuff.example.com:/**sys/amd64/compile/BHYVE amd64 >> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU) >> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >> Stepping = 7 >> Features=0x8fa3ab7f> SEP,PGE,CMOV,PAT,PSE36,DTS,**MMX,FXSR,SSE,SSE2,SS,PBE> >> Features2=0x83bee217> CX16,xTPR,PDCM,PCID,DCA,SSE4.**1,SSE4.2,x2APIC,POPCNT,TSCDLT,**AESNI,HV> >> AMD Features=0x2c100800 >> AMD Features2=0x1 >> TSC: P-state invariant >> real memory = 5368709120 (5120 MB) >> avail memory = 3957534720 (3774 MB) >> Event timer "LAPIC" quality 400 >> ACPI APIC Table: >> random device not loaded; using insecure entropy >> ioapic0: Changing APIC ID to 1 >> ioapic0 irqs 0-16 on motherboard >> kbd0 at kbdmux0 >> acpi0: on motherboard >> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >> Event timer "RTC" frequency 32768 Hz quality 0 >> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >> pcib0: port 0xcf8-0xcff on acpi0 >> pci0: on pcib0 >> pcib0: no PRT entry for 0.31.INTA >> virtio_pci0: port 0x2000-0x201f mem >> 0xc0000000-0xc0001fff at device 1.0 on pci0 >> vtnet0: on virtio_pci0 >> virtio_pci0: host features: 0x18020 >> virtio_pci0: negotiated features: 0x18020 >> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >> virtio_pci1: port 0x2040-0x207f mem >> 0xc0002000-0xc0003fff at device 2.0 on pci0 >> vtblk0: on virtio_pci1 >> virtio_pci1: host features: 0x10000004 >> virtio_pci1: negotiated features: 0x10000004 >> vtblk0: 8192MB (16777216 512 byte sectors) >> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 >> uart2: console (9600,n,8,1) >> Timecounters tick every 10.000 msec >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 >> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-**00a098ac0ddf. >> Setting hostid: 0x24bfd025. >> Entropy harvesting: interrupts ethernet point_to_point kickstart. >> Starting file system checks: >> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% >> fragmentation) >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> Mounting local file systems:. >> Writing entropy file:. >> Setting hostname: cocopuff.example.com. >> vtnet0: link state changed to UP >> Starting Network: lo0 vtnet0. >> lo0: flags=8049 metric 0 mtu 16384 >> options=600003 >> inet6 ::1 prefixlen 128 >> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >> inet 127.0.0.1 netmask 0xff000000 >> nd6 options=21 >> vtnet0: flags=8943 >> metric 0 mtu 1500 >> options=80028 >> ether 00:a0:98:ac:0d:df >> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >> nd6 options=29 >> media: Ethernet 1000baseT >> status: active >> Starting devd. >> add net default: gateway 172.22.22.252 >> add net ::ffff:0.0.0.0: gateway ::1 >> add net ::0.0.0.0: gateway ::1 >> add net fe80::: gateway ::1 >> add net ff02::: gateway ::1 >> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >> 32-bit compatibility ldconfig path: /usr/lib32 >> Creating and/or trimming log files. >> Starting syslogd. >> Clearing /tmp (X related). >> Updating motd:. >> Performing sanity check on sshd configuration. >> Starting sshd. >> Starting cron. >> Starting background file system checks in 60 seconds. >> >> Fri Mar 29 14:09:20 CDT 2013 >> >> FreeBSD/amd64 (cocopuff.example.com) (console) >> >> login: >> >> >> ====== Next, boot the same image with two (2) vCPU: >> >> Command run by vmrun.sh: >> >> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 >> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >> >> Top shows it running out of control: >> >> root@olive:~ # top -bH >> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 >> 14:17:32 >> 25 processes: 3 running, 22 sleeping >> >> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free >> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other >> Swap: >> >> >> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% >> bhyve{coco vcpu 1} >> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% >> bhyve{coco vcpu 0} >> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd >> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd >> 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd >> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron >> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% >> bhyve{coco mevent} >> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh >> >> Even though, it's just sitting at the login prompt: >> >> root@olive:/vms/coco # ./vmrun.sh coco >> Launching virtual machine "coco" ... >> Consoles: userboot >> >> FreeBSD/amd64 User boot, Revision 1.1 >> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+**0x12bb08] >> / >> ______ ____ _____ _____ >> | ____| | _ \ / ____| __ \ >> | |___ _ __ ___ ___ | |_) | (___ | | | | >> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >> | | | | | __/ __/| |_) |____) | |__| | >> | | | | | | || | | | >> |_| |_| \___|\___||____/|_____/|_____/ ``` ` >> s` `.....---.......--.``` -/ >> +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. >> | | yo`:. :o `+- >> | 1. Boot Multi User [Enter] | y/ -/` -o/ >> | 2. Boot [S]ingle User | .- ::/sy+:. >> | 3. [Esc]ape to loader prompt | / `-- / >> | 4. Reboot | `: :` >> | | `: :` >> | Options: | / / >> | 5. Configure Boot [O]ptions... | .- -. >> | | -- -. >> | | `:` `:` >> | | .-- `--. >> | | .---.....----. >> +-----------------------------**------------+ >> >> >> Booting... >> Copyright (c) 1992-2013 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >> root@cocopuff.example.com:/**sys/amd64/compile/BHYVE amd64 >> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU) >> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >> Stepping = 7 >> Features=0x8fa3ab7f> SEP,PGE,CMOV,PAT,PSE36,DTS,**MMX,FXSR,SSE,SSE2,SS,PBE> >> Features2=0x83bee217> CX16,xTPR,PDCM,PCID,DCA,SSE4.**1,SSE4.2,x2APIC,POPCNT,TSCDLT,**AESNI,HV> >> AMD Features=0x2c100800 >> AMD Features2=0x1 >> TSC: P-state invariant >> real memory = 5368709120 (5120 MB) >> avail memory = 3957534720 (3774 MB) >> Event timer "LAPIC" quality 400 >> ACPI APIC Table: >> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >> FreeBSD/SMP: 2 package(s) x 1 core(s) >> cpu0 (BSP): APIC ID: 0 >> cpu1 (AP): APIC ID: 1 >> random device not loaded; using insecure entropy >> ioapic0: Changing APIC ID to 2 >> ioapic0 irqs 0-16 on motherboard >> kbd0 at kbdmux0 >> acpi0: on motherboard >> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >> Event timer "RTC" frequency 32768 Hz quality 0 >> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >> pcib0: port 0xcf8-0xcff on acpi0 >> pci0: on pcib0 >> pcib0: no PRT entry for 0.31.INTA >> virtio_pci0: port 0x2000-0x201f mem >> 0xc0000000-0xc0001fff at device 1.0 on pci0 >> vtnet0: on virtio_pci0 >> virtio_pci0: host features: 0x18020 >> virtio_pci0: negotiated features: 0x18020 >> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >> virtio_pci1: port 0x2040-0x207f mem >> 0xc0002000-0xc0003fff at device 2.0 on pci0 >> vtblk0: on virtio_pci1 >> virtio_pci1: host features: 0x10000004 >> virtio_pci1: negotiated features: 0x10000004 >> vtblk0: 8192MB (16777216 512 byte sectors) >> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 >> uart2: console (9600,n,8,1) >> Timecounters tick every 10.000 msec >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> SMP: AP CPU #1 Launched! >> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-**00a098ac0ddf. >> Setting hostid: 0x24bfd025. >> Entropy harvesting: interrupts ethernet point_to_point kickstart. >> Starting file system checks: >> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% >> fragmentation) >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >> Mounting local file systems:. >> Writing entropy file:. >> Setting hostname: cocopuff.example.com. >> vtnet0: link state changed to UP >> Starting Network: lo0 vtnet0. >> lo0: flags=8049 metric 0 mtu 16384 >> options=600003 >> inet6 ::1 prefixlen 128 >> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >> inet 127.0.0.1 netmask 0xff000000 >> nd6 options=21 >> vtnet0: flags=8943 >> metric 0 mtu 1500 >> options=80028 >> ether 00:a0:98:ac:0d:df >> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >> nd6 options=29 >> media: Ethernet 1000baseT >> status: active >> Starting devd. >> add net default: gateway 172.22.22.252 >> add net ::ffff:0.0.0.0: gateway ::1 >> add net ::0.0.0.0: gateway ::1 >> add net fe80::: gateway ::1 >> add net ff02::: gateway ::1 >> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >> 32-bit compatibility ldconfig path: /usr/lib32 >> Creating and/or trimming log files. >> Starting syslogd. >> syslogd: timed out waiting for child >> /etc/rc: WARNING: failed to start syslogd >> Clearing /tmp (X related). >> Updating motd:. >> Performing sanity check on sshd configuration. >> Starting sshd. >> Starting cron. >> Starting background file system checks in 60 seconds. >> >> Thu Apr 18 02:06:43 CDT 2013 >> >> FreeBSD/amd64 (cocopuff.example.com) (console) >> >> login: >> >> >> ========== end of boot sequence >> >> So, something is broken with SMP on the guest, I'd be happy to help just >> let me know. >> >> Dan >> >> ______________________________**_________________ >> freebsd-virtualization@**freebsd.org mailing list >> http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-** >> unsubscribe@freebsd.org " >> >> ______________________________**_________________ > freebsd-virtualization@**freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization > To unsubscribe, send any mail to "freebsd-virtualization-** > unsubscribe@freebsd.org " > From owner-freebsd-virtualization@FreeBSD.ORG Fri Mar 29 22:20:59 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E0FE35F7; Fri, 29 Mar 2013 22:20:59 +0000 (UTC) (envelope-from mack@macktronics.com) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by mx1.freebsd.org (Postfix) with ESMTP id 5B45EDAE; Fri, 29 Mar 2013 22:20:59 +0000 (UTC) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by coco.macktronics.com (Postfix) with ESMTP id 4F85E4AC40; Fri, 29 Mar 2013 17:20:53 -0500 (CDT) Date: Fri, 29 Mar 2013 17:20:53 -0500 (CDT) From: Dan Mack To: Neel Natu Subject: Re: bhyve tty / login problems / panic In-Reply-To: Message-ID: <20130329172023.L15559@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2013 22:20:59 -0000 Here you go Neel: root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 vcpu0 vm exits due to external interrupt 45051 number of times hlt was intercepted 2277 number of NMIs delivered to vcpu 0 vcpu total runtime 105001652482 number of ticks vcpu was idle 919 vcpu migration across host cpus 13 number of times hlt was ignored 0 total number of vm exits 9240925 root@olive:~ # root@olive:~ # root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 vcpu1 vm exits due to external interrupt 149431 number of times hlt was intercepted 4222 number of NMIs delivered to vcpu 0 vcpu total runtime 106876622528 number of ticks vcpu was idle 869 vcpu migration across host cpus 20 number of times hlt was ignored 0 total number of vm exits 8065909 Dan On Fri, 29 Mar 2013, Neel Natu wrote: > Hi Dan, > > On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack wrote: > >> >> I ran the procsystime dtrace script on bhyve when it was chewing up all >> the CPU in vCPU=2 mode, and this is what I see for about 10s of runtime: >> >> root@olive:/usr/share/dtrace/**toolkit # ./procsystime -n bhyve -aT >> Tracing... Hit Ctrl-C to end... >> dtrace: 158536 dynamic variable drops with non-empty dirty list >> dtrace: 207447 dynamic variable drops with non-empty dirty list >> dtrace: 189205 dynamic variable drops with non-empty dirty list >> dtrace: 164341 dynamic variable drops with non-empty dirty list >> dtrace: 246307 dynamic variable drops with non-empty dirty list >> dtrace: 187640 dynamic variable drops with non-empty dirty list >> dtrace: 214771 dynamic variable drops with non-empty dirty list >> dtrace: 221265 dynamic variable drops with non-empty dirty list >> ^C >> dtrace: 243468 dynamic variable drops with non-empty dirty list >> >> Elapsed Times for processes bhyve, >> >> SYSCALL TIME (ns) >> _umtx_op 18349 >> writev 135712 >> preadv 16175267 >> pwritev 22924378 >> ioctl 4353897920 >> TOTAL: 4393151626 >> >> CPU Times for processes bhyve, >> >> SYSCALL TIME (ns) >> _umtx_op 8815 >> writev 103145 >> pwritev 10647023 >> preadv 15159976 >> ioctl 3943399889 >> TOTAL: 3969318848 >> >> Syscall Counts for processes bhyve, >> >> SYSCALL COUNT >> _umtx_op 17 >> writev 43 >> pwritev 639 >> preadv 1183 >> ioctl 652782 >> TOTAL: 654664 >> >> Not sure if that helps or not. Hotkernel shows the cpu in the >> kernel`acpi_cpu_c1 call most of the time, normal ? >> >> Dan >> >> root@olive:/usr/share/dtrace/**toolkit # ./hotkernel >> Sampling... Hit Ctrl-C to end. >> ^C >> FUNCTION COUNT PCNT >> zfs.ko`space_map_sync 1 0.0% >> kernel`syscall_thread_exit 1 0.0% >> kernel`should_yield 1 0.0% >> zfs.ko`lzjb_compress 1 0.0% >> zfs.ko`dbuf_rele 1 0.0% >> zfs.ko`dsl_pool_tempreserve_**space 1 0.0% >> kernel`_bus_dmamap_load_buffer 1 0.0% >> kernel`cpu_idleclock 1 0.0% >> kernel`_rw_runlock_cookie 1 0.0% >> kernel`pmap_enter 1 0.0% >> kernel`__lockmgr_args 1 0.0% >> zfs.ko`vdev_geom_io_start 1 0.0% >> kernel`ahci_ch_intr 1 0.0% >> kernel`__rw_rlock 1 0.0% >> kernel`VOP_UNLOCK_APV 1 0.0% >> zfs.ko`arc_buf_thaw 1 0.0% >> kernel`atomic_set_int 1 0.0% >> kernel`g_io_schedule_down 1 0.0% >> kernel`__mtx_lock_flags 1 0.0% >> kernel`_callout_stop_safe 1 0.0% >> kernel`sched_idletd 1 0.0% >> kernel`_sx_slock 1 0.0% >> vmm.ko`lapic_set_intr 1 0.0% >> kernel`cpusetobj_ffs 1 0.0% >> zfs.ko`dbuf_hold 1 0.0% >> vmm.ko`vlapic_intr_accepted 1 0.0% >> zfs.ko`sa_attr_op 1 0.0% >> zfs.ko`dmu_buf_hold_array_by_**dnode 1 0.0% >> zfs.ko`taskq_run_safe 1 0.0% >> kernel`vm_reserv_free_page 1 0.0% >> kernel`_sx_sunlock 1 0.0% >> kernel`vn_rlimit_fsize 1 0.0% >> zfs.ko`vdev_queue_io_to_issue 1 0.0% >> kernel`vn_write 1 0.0% >> kernel`lock_init 1 0.0% >> vmm.ko`vmx_resume 1 0.0% >> zfs.ko`zio_gang_tree_free 1 0.0% >> zfs.ko`dbuf_rele_and_unlock 1 0.0% >> kernel`mnt_vnode_next_active 1 0.0% >> kernel`vm_page_remove 1 0.0% >> zfs.ko`vdev_queue_deadline_**compare 1 0.0% >> kernel`zone_free_item 1 0.0% >> vmm.ko`vmx_launch 1 0.0% >> kernel`m_pullup 1 0.0% >> kernel`pmap_change_wiring 1 0.0% >> kernel`sched_userret 1 0.0% >> zfs.ko`dbuf_write_done 1 0.0% >> zfs.ko`zio_execute 1 0.0% >> kernel`PHYS_TO_VM_PAGE 2 0.0% >> zfs.ko`avl_rotation 2 0.0% >> kernel`ehci_device_isoc_fs_**enter 2 0.0% >> vmm.ko`vm_exitinfo 2 0.0% >> zfs.ko`zfs_freebsd_write 2 0.0% >> vmm.ko`vcpu_stats 2 0.0% >> kernel`atomic_add_long 2 0.0% >> vmm.ko`0xffffffff81 2 0.0% >> kernel`vm_reserv_alloc_page 2 0.0% >> kernel`pagezero 2 0.0% >> kernel`bzero 2 0.0% >> zfs.ko`fletcher_2_native 2 0.0% >> vmm.ko`vmx_getreg 2 0.0% >> vmm.ko`vm_guest_msrs 3 0.0% >> vmm.ko`vmx_setreg 3 0.0% >> vmm.ko`vm_get_register 3 0.0% >> kernel`sx_try_xlock_ 3 0.0% >> kernel`vm_radix_lookup_le 3 0.0% >> vmm.ko`lapic_pending_intr 4 0.0% >> kernel`syscall_thread_enter 4 0.0% >> kernel`cpu_set_syscall_retval 4 0.0% >> kernel`__mnt_vnode_next_all 4 0.0% >> kernel`vfs_stdsync 4 0.0% >> vmm.ko`vmm_fetch_instruction 5 0.0% >> vmm.ko`vlapic_op_mem_read 6 0.0% >> kernel`_sx_xunlock 7 0.0% >> vmm.ko`vmm_emulate_instruction 7 0.0% >> vmm.ko`vlapic_op_mem_write 7 0.0% >> kernel`__mtx_lock_spin_flags 8 0.0% >> vmm.ko`lapic_timer_tick 8 0.0% >> zfs.ko`list_next 8 0.0% >> zfs.ko`l2arc_feed_thread 8 0.0% >> kernel`userret 9 0.0% >> vmm.ko`vm_gpa2hpa 9 0.0% >> vmm.ko`vmm_decode_instruction 11 0.0% >> vmm.ko`ept_vmmmap_get 11 0.0% >> vmm.ko`vm_lapic 11 0.0% >> kernel`spinlock_enter 14 0.0% >> kernel`__mtx_lock_sleep 15 0.0% >> vmm.ko`vcpu_set_state 16 0.0% >> vmm.ko`vm_nmi_pending 16 0.0% >> kernel`malloc_type_zone_**allocated 16 0.0% >> kernel`devfs_ioctl_f 16 0.0% >> kernel`pmap_kextract 17 0.0% >> kernel`sys_ioctl 18 0.0% >> kernel`malloc 21 0.0% >> kernel`cpu_fetch_syscall_args 24 0.0% >> kernel`uma_zfree_arg 28 0.0% >> vmm.ko`vlapic_pending_intr 29 0.0% >> kernel`trap 30 0.0% >> kernel`hpet_get_timecount 31 0.0% >> dtrace.ko`dtrace_trap 35 0.0% >> vmm.ko`vlapic_update_ppr 39 0.0% >> vmm.ko`vmmdev_ioctl 40 0.0% >> kernel`copyout 43 0.0% >> kernel`critical_enter 44 0.0% >> kernel`copyin 51 0.1% >> vmm.ko`vmcs_read 53 0.1% >> kernel`fpusave 61 0.1% >> kernel`uma_zalloc_arg 62 0.1% >> kernel`devvn_refthread 69 0.1% >> kernel`atomic_load_acq_int 70 0.1% >> kernel`fpurestore 70 0.1% >> vmm.ko`vlapic_timer_tick 75 0.1% >> kernel`amd64_syscall 77 0.1% >> kernel`bcopy 91 0.1% >> kernel`fpudna 103 0.1% >> kernel`free 103 0.1% >> kernel`critical_exit 105 0.1% >> kernel`dev_relthread 108 0.1% >> kernel`spinlock_exit 119 0.1% >> kernel`kern_ioctl 138 0.1% >> kernel`fpuexit 242 0.3% >> kernel`0xffffffff80 306 0.3% >> vmm.ko`restore_host_msrs 398 0.4% >> vmm.ko`vm_run 428 0.4% >> vmm.ko`restore_guest_msrs 433 0.5% >> kernel`cpu_idle 906 1.0% >> vmm.ko`vmx_run 9266 9.7% >> kernel`acpi_cpu_c1 81134 85.2% >> >> Hope this helps, let me know if you have any ideas on what to try next. >> >> > Could you try the following command to get vcpu stats? > sudo bhyvectl --vm=vm1 --get-stats --cpu=0 > sudo bhyvectl --vm=vm1 --get-stats --cpu=1 > > best > Neel > > >> Dan >> >> >> >> >> On Fri, 29 Mar 2013, Dan Mack wrote: >> >> On Thu, 28 Mar 2013, Peter Grehan wrote: >>> >>> Hi Dan, >>>> >>>> Any ideas? >>>>> >>>> >>>> Just to confirm: your host system is an i7 3930k, which looks like a 6 >>>> core system. Should be plenty beefy enough for a 2 vCPU guest. >>>> >>> >>> That's what I thought, 6 cores, 12 threads, 64 GB RAM. >>> >>> The issue you are seeing looks a bit like what happens when systems are >>>> heavily oversubscribed and vCPUs can't get enough run time. >>>> >>> >>> I was running in on an idle system; only bhyve was running. >>> >>> >>> How much spare RAM is on the host when you started the vm ? Also, is >>>> there much running on the host when the guest is started ? >>>> >>> >>> 64GB >>> >>> Here's some more information (sorry for the formatting): >>> >>> Case 1: single vCPU via: >>> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 >>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>> >>> Top shows it idle (note low load, 4GB memory given to guest, still 53GB >>> free RAM, nothing else clocking CPU cycles): >>> >>> root@olive:~ # top -bH >>> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 >>> 14:11:51 >>> 24 processes: 1 running, 23 sleeping >>> >>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free >>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other >>> Swap: >>> >>> >>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% >>> bhyve{coco vcpu 0} >>> 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd >>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% >>> bhyve{coco mevent} >>> >>> >>> So, case #1 all is normal. I was able to build a kernel with very good >>> performance. >>> >>> The boot sequence looks like the following: >>> >>> root@olive:/vms/coco # ./vmrun.sh coco >>> Launching virtual machine "coco" ... >>> Consoles: userboot >>> >>> FreeBSD/amd64 User boot, Revision 1.1 >>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+**0x12bb08] >>> / >>> ______ ____ _____ _____ >>> | ____| | _ \ / ____| __ \ >>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>> | | | | | __/ __/| |_) |____) | |__| | >>> | | | | | | || | | | >>> |_| |_| \___|\___||____/|_____/|_____/ ``` ` >>> s` `.....---.......--.``` -/ >>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. >>> | | yo`:. :o `+- >>> | 1. Boot Multi User [Enter] | y/ -/` -o/ >>> | 2. Boot [S]ingle User | .- ::/sy+:. >>> | 3. [Esc]ape to loader prompt | / `-- / >>> | 4. Reboot | `: :` >>> | | `: :` >>> | Options: | / / >>> | 5. Configure Boot [O]ptions... | .- -. >>> | | -- -. >>> | | `:` `:` >>> | | .-- `--. >>> | | .---.....----. >>> +-----------------------------**------------+ >>> >>> >>> Booting... >>> Copyright (c) 1992-2013 The FreeBSD Project. >>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >>> The Regents of the University of California. All rights reserved. >>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>> root@cocopuff.example.com:/**sys/amd64/compile/BHYVE amd64 >>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU) >>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>> Stepping = 7 >>> Features=0x8fa3ab7f>> SEP,PGE,CMOV,PAT,PSE36,DTS,**MMX,FXSR,SSE,SSE2,SS,PBE> >>> Features2=0x83bee217>> CX16,xTPR,PDCM,PCID,DCA,SSE4.**1,SSE4.2,x2APIC,POPCNT,TSCDLT,**AESNI,HV> >>> AMD Features=0x2c100800 >>> AMD Features2=0x1 >>> TSC: P-state invariant >>> real memory = 5368709120 (5120 MB) >>> avail memory = 3957534720 (3774 MB) >>> Event timer "LAPIC" quality 400 >>> ACPI APIC Table: >>> random device not loaded; using insecure entropy >>> ioapic0: Changing APIC ID to 1 >>> ioapic0 irqs 0-16 on motherboard >>> kbd0 at kbdmux0 >>> acpi0: on motherboard >>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>> Event timer "RTC" frequency 32768 Hz quality 0 >>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>> pcib0: port 0xcf8-0xcff on acpi0 >>> pci0: on pcib0 >>> pcib0: no PRT entry for 0.31.INTA >>> virtio_pci0: port 0x2000-0x201f mem >>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>> vtnet0: on virtio_pci0 >>> virtio_pci0: host features: 0x18020 >>> virtio_pci0: negotiated features: 0x18020 >>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>> virtio_pci1: port 0x2040-0x207f mem >>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>> vtblk0: on virtio_pci1 >>> virtio_pci1: host features: 0x10000004 >>> virtio_pci1: negotiated features: 0x10000004 >>> vtblk0: 8192MB (16777216 512 byte sectors) >>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 >>> uart2: console (9600,n,8,1) >>> Timecounters tick every 10.000 msec >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 >>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-**00a098ac0ddf. >>> Setting hostid: 0x24bfd025. >>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>> Starting file system checks: >>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% >>> fragmentation) >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> Mounting local file systems:. >>> Writing entropy file:. >>> Setting hostname: cocopuff.example.com. >>> vtnet0: link state changed to UP >>> Starting Network: lo0 vtnet0. >>> lo0: flags=8049 metric 0 mtu 16384 >>> options=600003 >>> inet6 ::1 prefixlen 128 >>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>> inet 127.0.0.1 netmask 0xff000000 >>> nd6 options=21 >>> vtnet0: flags=8943 >>> metric 0 mtu 1500 >>> options=80028 >>> ether 00:a0:98:ac:0d:df >>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>> nd6 options=29 >>> media: Ethernet 1000baseT >>> status: active >>> Starting devd. >>> add net default: gateway 172.22.22.252 >>> add net ::ffff:0.0.0.0: gateway ::1 >>> add net ::0.0.0.0: gateway ::1 >>> add net fe80::: gateway ::1 >>> add net ff02::: gateway ::1 >>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>> 32-bit compatibility ldconfig path: /usr/lib32 >>> Creating and/or trimming log files. >>> Starting syslogd. >>> Clearing /tmp (X related). >>> Updating motd:. >>> Performing sanity check on sshd configuration. >>> Starting sshd. >>> Starting cron. >>> Starting background file system checks in 60 seconds. >>> >>> Fri Mar 29 14:09:20 CDT 2013 >>> >>> FreeBSD/amd64 (cocopuff.example.com) (console) >>> >>> login: >>> >>> >>> ====== Next, boot the same image with two (2) vCPU: >>> >>> Command run by vmrun.sh: >>> >>> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 >>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>> >>> Top shows it running out of control: >>> >>> root@olive:~ # top -bH >>> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 >>> 14:17:32 >>> 25 processes: 3 running, 22 sleeping >>> >>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free >>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other >>> Swap: >>> >>> >>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% >>> bhyve{coco vcpu 1} >>> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% >>> bhyve{coco vcpu 0} >>> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd >>> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd >>> 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd >>> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron >>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% >>> bhyve{coco mevent} >>> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh >>> >>> Even though, it's just sitting at the login prompt: >>> >>> root@olive:/vms/coco # ./vmrun.sh coco >>> Launching virtual machine "coco" ... >>> Consoles: userboot >>> >>> FreeBSD/amd64 User boot, Revision 1.1 >>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+**0x12bb08] >>> / >>> ______ ____ _____ _____ >>> | ____| | _ \ / ____| __ \ >>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>> | | | | | __/ __/| |_) |____) | |__| | >>> | | | | | | || | | | >>> |_| |_| \___|\___||____/|_____/|_____/ ``` ` >>> s` `.....---.......--.``` -/ >>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. >>> | | yo`:. :o `+- >>> | 1. Boot Multi User [Enter] | y/ -/` -o/ >>> | 2. Boot [S]ingle User | .- ::/sy+:. >>> | 3. [Esc]ape to loader prompt | / `-- / >>> | 4. Reboot | `: :` >>> | | `: :` >>> | Options: | / / >>> | 5. Configure Boot [O]ptions... | .- -. >>> | | -- -. >>> | | `:` `:` >>> | | .-- `--. >>> | | .---.....----. >>> +-----------------------------**------------+ >>> >>> >>> Booting... >>> Copyright (c) 1992-2013 The FreeBSD Project. >>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >>> The Regents of the University of California. All rights reserved. >>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>> root@cocopuff.example.com:/**sys/amd64/compile/BHYVE amd64 >>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU) >>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>> Stepping = 7 >>> Features=0x8fa3ab7f>> SEP,PGE,CMOV,PAT,PSE36,DTS,**MMX,FXSR,SSE,SSE2,SS,PBE> >>> Features2=0x83bee217>> CX16,xTPR,PDCM,PCID,DCA,SSE4.**1,SSE4.2,x2APIC,POPCNT,TSCDLT,**AESNI,HV> >>> AMD Features=0x2c100800 >>> AMD Features2=0x1 >>> TSC: P-state invariant >>> real memory = 5368709120 (5120 MB) >>> avail memory = 3957534720 (3774 MB) >>> Event timer "LAPIC" quality 400 >>> ACPI APIC Table: >>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >>> FreeBSD/SMP: 2 package(s) x 1 core(s) >>> cpu0 (BSP): APIC ID: 0 >>> cpu1 (AP): APIC ID: 1 >>> random device not loaded; using insecure entropy >>> ioapic0: Changing APIC ID to 2 >>> ioapic0 irqs 0-16 on motherboard >>> kbd0 at kbdmux0 >>> acpi0: on motherboard >>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>> Event timer "RTC" frequency 32768 Hz quality 0 >>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>> pcib0: port 0xcf8-0xcff on acpi0 >>> pci0: on pcib0 >>> pcib0: no PRT entry for 0.31.INTA >>> virtio_pci0: port 0x2000-0x201f mem >>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>> vtnet0: on virtio_pci0 >>> virtio_pci0: host features: 0x18020 >>> virtio_pci0: negotiated features: 0x18020 >>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>> virtio_pci1: port 0x2040-0x207f mem >>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>> vtblk0: on virtio_pci1 >>> virtio_pci1: host features: 0x10000004 >>> virtio_pci1: negotiated features: 0x10000004 >>> vtblk0: 8192MB (16777216 512 byte sectors) >>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 >>> uart2: console (9600,n,8,1) >>> Timecounters tick every 10.000 msec >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> SMP: AP CPU #1 Launched! >>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-**00a098ac0ddf. >>> Setting hostid: 0x24bfd025. >>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>> Starting file system checks: >>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% >>> fragmentation) >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>> Mounting local file systems:. >>> Writing entropy file:. >>> Setting hostname: cocopuff.example.com. >>> vtnet0: link state changed to UP >>> Starting Network: lo0 vtnet0. >>> lo0: flags=8049 metric 0 mtu 16384 >>> options=600003 >>> inet6 ::1 prefixlen 128 >>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>> inet 127.0.0.1 netmask 0xff000000 >>> nd6 options=21 >>> vtnet0: flags=8943 >>> metric 0 mtu 1500 >>> options=80028 >>> ether 00:a0:98:ac:0d:df >>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>> nd6 options=29 >>> media: Ethernet 1000baseT >>> status: active >>> Starting devd. >>> add net default: gateway 172.22.22.252 >>> add net ::ffff:0.0.0.0: gateway ::1 >>> add net ::0.0.0.0: gateway ::1 >>> add net fe80::: gateway ::1 >>> add net ff02::: gateway ::1 >>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>> 32-bit compatibility ldconfig path: /usr/lib32 >>> Creating and/or trimming log files. >>> Starting syslogd. >>> syslogd: timed out waiting for child >>> /etc/rc: WARNING: failed to start syslogd >>> Clearing /tmp (X related). >>> Updating motd:. >>> Performing sanity check on sshd configuration. >>> Starting sshd. >>> Starting cron. >>> Starting background file system checks in 60 seconds. >>> >>> Thu Apr 18 02:06:43 CDT 2013 >>> >>> FreeBSD/amd64 (cocopuff.example.com) (console) >>> >>> login: >>> >>> >>> ========== end of boot sequence >>> >>> So, something is broken with SMP on the guest, I'd be happy to help just >>> let me know. >>> >>> Dan >>> >>> ______________________________**_________________ >>> freebsd-virtualization@**freebsd.org mailing list >>> http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization >>> To unsubscribe, send any mail to "freebsd-virtualization-** >>> unsubscribe@freebsd.org " >>> >>> ______________________________**_________________ >> freebsd-virtualization@**freebsd.org mailing list >> http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-** >> unsubscribe@freebsd.org " >> > From owner-freebsd-virtualization@FreeBSD.ORG Sat Mar 30 17:50:45 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6513A22B; Sat, 30 Mar 2013 17:50:45 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-ia0-x232.google.com (mail-ia0-x232.google.com [IPv6:2607:f8b0:4001:c02::232]) by mx1.freebsd.org (Postfix) with ESMTP id 251952A2; Sat, 30 Mar 2013 17:50:45 +0000 (UTC) Received: by mail-ia0-f178.google.com with SMTP id r13so1053452iar.37 for ; Sat, 30 Mar 2013 10:50:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=y6W3M+oTMCUncM0vzoZAvXPjjy43/Z+KaVELrcBuDng=; b=huWpBzfqW5QJI0u2iZMC3EOjBKaa5T56kqFOBcs1YZGItuIANraGoZ0VXgF07RR5g2 FRwoMb+eMlOBar0aIY5OPVCeom9qz3gCW9EhzsNKA3Nm0N2JUZD4tEhV6K3yzRkBkNaM EkemXqMIKSoQj2NZ8QvTXJr2uRXucJVqPyBlmk6ylZpGQWJCtmwCmuhORLhIRkpNMwbG fcyxbakras9vNBnxsebVf2tUem4wEqGZ9tXtW+tLAvyNyhVFNwZPed3PiuGVpTMPsQgM ukPpfw2nHGHnaM86rXVaysGYHDgWPnqn8IUUiJ76Pj1n56YkckW3/I2gxWPbAKojhvds AufQ== MIME-Version: 1.0 X-Received: by 10.42.145.137 with SMTP id f9mr3107594icv.52.1364665843965; Sat, 30 Mar 2013 10:50:43 -0700 (PDT) Received: by 10.42.23.132 with HTTP; Sat, 30 Mar 2013 10:50:43 -0700 (PDT) In-Reply-To: <20130329172023.L15559@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> <20130329172023.L15559@coco.macktronics.com> Date: Sat, 30 Mar 2013 10:50:43 -0700 Message-ID: Subject: Re: bhyve tty / login problems / panic From: Neel Natu To: Dan Mack Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Mar 2013 17:50:45 -0000 Hi Dan, On Fri, Mar 29, 2013 at 3:20 PM, Dan Mack wrote: > > > Here you go Neel: > > root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 > vcpu0 > vm exits due to external interrupt 45051 > number of times hlt was intercepted 2277 > number of NMIs delivered to vcpu 0 > vcpu total runtime 105001652482 > number of ticks vcpu was idle 919 > vcpu migration across host cpus 13 > number of times hlt was ignored 0 > total number of vm exits 9240925 > root@olive:~ # > root@olive:~ # > root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 > vcpu1 > vm exits due to external interrupt 149431 > number of times hlt was intercepted 4222 > number of NMIs delivered to vcpu 0 > vcpu total runtime 106876622528 > number of ticks vcpu was idle 869 > vcpu migration across host cpus 20 > number of times hlt was ignored 0 > total number of vm exits 8065909 > > That did not help a whole lot because we were not keeping track of all the reasons a vcpu could exit. I have fixed that in r248935: http://svnweb.freebsd.org/base?view=revision&revision=248935 Do you mind updating your vmm.ko with the change and getting the stats again? best Neel > Dan > > > > On Fri, 29 Mar 2013, Neel Natu wrote: > > Hi Dan, >> >> On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack wrote: >> >> >>> I ran the procsystime dtrace script on bhyve when it was chewing up all >>> the CPU in vCPU=2 mode, and this is what I see for about 10s of runtime: >>> >>> root@olive:/usr/share/dtrace/****toolkit # ./procsystime -n bhyve -aT >>> >>> Tracing... Hit Ctrl-C to end... >>> dtrace: 158536 dynamic variable drops with non-empty dirty list >>> dtrace: 207447 dynamic variable drops with non-empty dirty list >>> dtrace: 189205 dynamic variable drops with non-empty dirty list >>> dtrace: 164341 dynamic variable drops with non-empty dirty list >>> dtrace: 246307 dynamic variable drops with non-empty dirty list >>> dtrace: 187640 dynamic variable drops with non-empty dirty list >>> dtrace: 214771 dynamic variable drops with non-empty dirty list >>> dtrace: 221265 dynamic variable drops with non-empty dirty list >>> ^C >>> dtrace: 243468 dynamic variable drops with non-empty dirty list >>> >>> Elapsed Times for processes bhyve, >>> >>> SYSCALL TIME (ns) >>> _umtx_op 18349 >>> writev 135712 >>> preadv 16175267 >>> pwritev 22924378 >>> ioctl 4353897920 >>> TOTAL: 4393151626 >>> >>> CPU Times for processes bhyve, >>> >>> SYSCALL TIME (ns) >>> _umtx_op 8815 >>> writev 103145 >>> pwritev 10647023 >>> preadv 15159976 >>> ioctl 3943399889 >>> TOTAL: 3969318848 >>> >>> Syscall Counts for processes bhyve, >>> >>> SYSCALL COUNT >>> _umtx_op 17 >>> writev 43 >>> pwritev 639 >>> preadv 1183 >>> ioctl 652782 >>> TOTAL: 654664 >>> >>> Not sure if that helps or not. Hotkernel shows the cpu in the >>> kernel`acpi_cpu_c1 call most of the time, normal ? >>> >>> Dan >>> >>> root@olive:/usr/share/dtrace/****toolkit # ./hotkernel >>> >>> Sampling... Hit Ctrl-C to end. >>> ^C >>> FUNCTION COUNT PCNT >>> zfs.ko`space_map_sync 1 0.0% >>> kernel`syscall_thread_exit 1 0.0% >>> kernel`should_yield 1 0.0% >>> zfs.ko`lzjb_compress 1 0.0% >>> zfs.ko`dbuf_rele 1 0.0% >>> zfs.ko`dsl_pool_tempreserve_****space 1 0.0% >>> >>> kernel`_bus_dmamap_load_buffer 1 0.0% >>> kernel`cpu_idleclock 1 0.0% >>> kernel`_rw_runlock_cookie 1 0.0% >>> kernel`pmap_enter 1 0.0% >>> kernel`__lockmgr_args 1 0.0% >>> zfs.ko`vdev_geom_io_start 1 0.0% >>> kernel`ahci_ch_intr 1 0.0% >>> kernel`__rw_rlock 1 0.0% >>> kernel`VOP_UNLOCK_APV 1 0.0% >>> zfs.ko`arc_buf_thaw 1 0.0% >>> kernel`atomic_set_int 1 0.0% >>> kernel`g_io_schedule_down 1 0.0% >>> kernel`__mtx_lock_flags 1 0.0% >>> kernel`_callout_stop_safe 1 0.0% >>> kernel`sched_idletd 1 0.0% >>> kernel`_sx_slock 1 0.0% >>> vmm.ko`lapic_set_intr 1 0.0% >>> kernel`cpusetobj_ffs 1 0.0% >>> zfs.ko`dbuf_hold 1 0.0% >>> vmm.ko`vlapic_intr_accepted 1 0.0% >>> zfs.ko`sa_attr_op 1 0.0% >>> zfs.ko`dmu_buf_hold_array_by_****dnode 1 0.0% >>> >>> zfs.ko`taskq_run_safe 1 0.0% >>> kernel`vm_reserv_free_page 1 0.0% >>> kernel`_sx_sunlock 1 0.0% >>> kernel`vn_rlimit_fsize 1 0.0% >>> zfs.ko`vdev_queue_io_to_issue 1 0.0% >>> kernel`vn_write 1 0.0% >>> kernel`lock_init 1 0.0% >>> vmm.ko`vmx_resume 1 0.0% >>> zfs.ko`zio_gang_tree_free 1 0.0% >>> zfs.ko`dbuf_rele_and_unlock 1 0.0% >>> kernel`mnt_vnode_next_active 1 0.0% >>> kernel`vm_page_remove 1 0.0% >>> zfs.ko`vdev_queue_deadline_****compare 1 0.0% >>> >>> kernel`zone_free_item 1 0.0% >>> vmm.ko`vmx_launch 1 0.0% >>> kernel`m_pullup 1 0.0% >>> kernel`pmap_change_wiring 1 0.0% >>> kernel`sched_userret 1 0.0% >>> zfs.ko`dbuf_write_done 1 0.0% >>> zfs.ko`zio_execute 1 0.0% >>> kernel`PHYS_TO_VM_PAGE 2 0.0% >>> zfs.ko`avl_rotation 2 0.0% >>> kernel`ehci_device_isoc_fs_****enter 2 0.0% >>> >>> vmm.ko`vm_exitinfo 2 0.0% >>> zfs.ko`zfs_freebsd_write 2 0.0% >>> vmm.ko`vcpu_stats 2 0.0% >>> kernel`atomic_add_long 2 0.0% >>> vmm.ko`0xffffffff81 2 0.0% >>> kernel`vm_reserv_alloc_page 2 0.0% >>> kernel`pagezero 2 0.0% >>> kernel`bzero 2 0.0% >>> zfs.ko`fletcher_2_native 2 0.0% >>> vmm.ko`vmx_getreg 2 0.0% >>> vmm.ko`vm_guest_msrs 3 0.0% >>> vmm.ko`vmx_setreg 3 0.0% >>> vmm.ko`vm_get_register 3 0.0% >>> kernel`sx_try_xlock_ 3 0.0% >>> kernel`vm_radix_lookup_le 3 0.0% >>> vmm.ko`lapic_pending_intr 4 0.0% >>> kernel`syscall_thread_enter 4 0.0% >>> kernel`cpu_set_syscall_retval 4 0.0% >>> kernel`__mnt_vnode_next_all 4 0.0% >>> kernel`vfs_stdsync 4 0.0% >>> vmm.ko`vmm_fetch_instruction 5 0.0% >>> vmm.ko`vlapic_op_mem_read 6 0.0% >>> kernel`_sx_xunlock 7 0.0% >>> vmm.ko`vmm_emulate_instruction 7 0.0% >>> vmm.ko`vlapic_op_mem_write 7 0.0% >>> kernel`__mtx_lock_spin_flags 8 0.0% >>> vmm.ko`lapic_timer_tick 8 0.0% >>> zfs.ko`list_next 8 0.0% >>> zfs.ko`l2arc_feed_thread 8 0.0% >>> kernel`userret 9 0.0% >>> vmm.ko`vm_gpa2hpa 9 0.0% >>> vmm.ko`vmm_decode_instruction 11 0.0% >>> vmm.ko`ept_vmmmap_get 11 0.0% >>> vmm.ko`vm_lapic 11 0.0% >>> kernel`spinlock_enter 14 0.0% >>> kernel`__mtx_lock_sleep 15 0.0% >>> vmm.ko`vcpu_set_state 16 0.0% >>> vmm.ko`vm_nmi_pending 16 0.0% >>> kernel`malloc_type_zone_****allocated 16 0.0% >>> >>> kernel`devfs_ioctl_f 16 0.0% >>> kernel`pmap_kextract 17 0.0% >>> kernel`sys_ioctl 18 0.0% >>> kernel`malloc 21 0.0% >>> kernel`cpu_fetch_syscall_args 24 0.0% >>> kernel`uma_zfree_arg 28 0.0% >>> vmm.ko`vlapic_pending_intr 29 0.0% >>> kernel`trap 30 0.0% >>> kernel`hpet_get_timecount 31 0.0% >>> dtrace.ko`dtrace_trap 35 0.0% >>> vmm.ko`vlapic_update_ppr 39 0.0% >>> vmm.ko`vmmdev_ioctl 40 0.0% >>> kernel`copyout 43 0.0% >>> kernel`critical_enter 44 0.0% >>> kernel`copyin 51 0.1% >>> vmm.ko`vmcs_read 53 0.1% >>> kernel`fpusave 61 0.1% >>> kernel`uma_zalloc_arg 62 0.1% >>> kernel`devvn_refthread 69 0.1% >>> kernel`atomic_load_acq_int 70 0.1% >>> kernel`fpurestore 70 0.1% >>> vmm.ko`vlapic_timer_tick 75 0.1% >>> kernel`amd64_syscall 77 0.1% >>> kernel`bcopy 91 0.1% >>> kernel`fpudna 103 0.1% >>> kernel`free 103 0.1% >>> kernel`critical_exit 105 0.1% >>> kernel`dev_relthread 108 0.1% >>> kernel`spinlock_exit 119 0.1% >>> kernel`kern_ioctl 138 0.1% >>> kernel`fpuexit 242 0.3% >>> kernel`0xffffffff80 306 0.3% >>> vmm.ko`restore_host_msrs 398 0.4% >>> vmm.ko`vm_run 428 0.4% >>> vmm.ko`restore_guest_msrs 433 0.5% >>> kernel`cpu_idle 906 1.0% >>> vmm.ko`vmx_run 9266 9.7% >>> kernel`acpi_cpu_c1 81134 85.2% >>> >>> Hope this helps, let me know if you have any ideas on what to try next. >>> >>> >>> Could you try the following command to get vcpu stats? >> sudo bhyvectl --vm=vm1 --get-stats --cpu=0 >> sudo bhyvectl --vm=vm1 --get-stats --cpu=1 >> >> best >> Neel >> >> >> Dan >>> >>> >>> >>> >>> On Fri, 29 Mar 2013, Dan Mack wrote: >>> >>> On Thu, 28 Mar 2013, Peter Grehan wrote: >>> >>>> >>>> Hi Dan, >>>> >>>>> >>>>> Any ideas? >>>>> >>>>>> >>>>>> >>>>> Just to confirm: your host system is an i7 3930k, which looks like a 6 >>>>> core system. Should be plenty beefy enough for a 2 vCPU guest. >>>>> >>>>> >>>> That's what I thought, 6 cores, 12 threads, 64 GB RAM. >>>> >>>> The issue you are seeing looks a bit like what happens when systems are >>>> >>>>> heavily oversubscribed and vCPUs can't get enough run time. >>>>> >>>>> >>>> I was running in on an idle system; only bhyve was running. >>>> >>>> >>>> How much spare RAM is on the host when you started the vm ? Also, is >>>> >>>>> there much running on the host when the guest is started ? >>>>> >>>>> >>>> 64GB >>>> >>>> Here's some more information (sorry for the formatting): >>>> >>>> Case 1: single vCPU via: >>>> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 >>>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>> >>>> Top shows it idle (note low load, 4GB memory given to guest, still 53GB >>>> free RAM, nothing else clocking CPU cycles): >>>> >>>> root@olive:~ # top -bH >>>> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 >>>> 14:11:51 >>>> 24 processes: 1 running, 23 sleeping >>>> >>>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free >>>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other >>>> Swap: >>>> >>>> >>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>>> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% >>>> bhyve{coco vcpu 0} >>>> 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd >>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% >>>> bhyve{coco mevent} >>>> >>>> >>>> So, case #1 all is normal. I was able to build a kernel with very good >>>> performance. >>>> >>>> The boot sequence looks like the following: >>>> >>>> root@olive:/vms/coco # ./vmrun.sh coco >>>> Launching virtual machine "coco" ... >>>> Consoles: userboot >>>> >>>> FreeBSD/amd64 User boot, Revision 1.1 >>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08] >>>> >>>> / >>>> ______ ____ _____ _____ >>>> | ____| | _ \ / ____| __ \ >>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>> | | | | | __/ __/| |_) |____) | |__| | >>>> | | | | | | || | | | >>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>> ` >>>> s` `.....---.......--.``` >>>> -/ >>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>> +. >>>> | | yo`:. :o >>>> `+- >>>> | 1. Boot Multi User [Enter] | y/ -/` -o/ >>>> | 2. Boot [S]ingle User | .- >>>> ::/sy+:. >>>> | 3. [Esc]ape to loader prompt | / `-- >>>> / >>>> | 4. Reboot | `: >>>> :` >>>> | | `: >>>> :` >>>> | Options: | / >>>> / >>>> | 5. Configure Boot [O]ptions... | .- >>>> -. >>>> | | -- -. >>>> | | `:` `:` >>>> | | .-- `--. >>>> | | .---.....----. >>>> +-----------------------------****------------+ >>>> >>>> >>>> >>>> Booting... >>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >>>> The Regents of the University of California. All rights >>>> reserved. >>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>> root@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64 >>>> >>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU) >>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>> Stepping = 7 >>>> Features=0x8fa3ab7f>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE> >>>> Features2=0x83bee217>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,** >>>> TSCDLT,**AESNI,HV> >>>> AMD Features=0x2c100800 >>>> >>>> AMD Features2=0x1 >>>> TSC: P-state invariant >>>> real memory = 5368709120 (5120 MB) >>>> avail memory = 3957534720 (3774 MB) >>>> Event timer "LAPIC" quality 400 >>>> ACPI APIC Table: >>>> random device not loaded; using insecure entropy >>>> ioapic0: Changing APIC ID to 1 >>>> ioapic0 irqs 0-16 on motherboard >>>> kbd0 at kbdmux0 >>>> acpi0: on motherboard >>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>> pcib0: port 0xcf8-0xcff on acpi0 >>>> pci0: on pcib0 >>>> pcib0: no PRT entry for 0.31.INTA >>>> virtio_pci0: port 0x2000-0x201f mem >>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>> vtnet0: on virtio_pci0 >>>> virtio_pci0: host features: 0x18020 >>>> virtio_pci0: negotiated features: 0x18020 >>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>> virtio_pci1: port 0x2040-0x207f mem >>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>> vtblk0: on virtio_pci1 >>>> virtio_pci1: host features: 0x10000004 >>>> virtio_pci1: negotiated features: 0x10000004 >>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>> pci0 >>>> uart2: console (9600,n,8,1) >>>> Timecounters tick every 10.000 msec >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 >>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf. >>>> >>>> Setting hostid: 0x24bfd025. >>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>> Starting file system checks: >>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% >>>> fragmentation) >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> Mounting local file systems:. >>>> Writing entropy file:. >>>> Setting hostname: cocopuff.example.com. >>>> vtnet0: link state changed to UP >>>> Starting Network: lo0 vtnet0. >>>> lo0: flags=8049 metric 0 mtu 16384 >>>> options=600003 >>>> >>>> inet6 ::1 prefixlen 128 >>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>> inet 127.0.0.1 netmask 0xff000000 >>>> nd6 options=21 >>>> vtnet0: flags=8943>>> MULTICAST> >>>> metric 0 mtu 1500 >>>> options=80028 >>>> >>>> ether 00:a0:98:ac:0d:df >>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>> nd6 options=29 >>>> >>>> media: Ethernet 1000baseT >>>> status: active >>>> Starting devd. >>>> add net default: gateway 172.22.22.252 >>>> add net ::ffff:0.0.0.0: gateway ::1 >>>> add net ::0.0.0.0: gateway ::1 >>>> add net fe80::: gateway ::1 >>>> add net ff02::: gateway ::1 >>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>> Creating and/or trimming log files. >>>> Starting syslogd. >>>> Clearing /tmp (X related). >>>> Updating motd:. >>>> Performing sanity check on sshd configuration. >>>> Starting sshd. >>>> Starting cron. >>>> Starting background file system checks in 60 seconds. >>>> >>>> Fri Mar 29 14:09:20 CDT 2013 >>>> >>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>> >>>> login: >>>> >>>> >>>> ====== Next, boot the same image with two (2) vCPU: >>>> >>>> Command run by vmrun.sh: >>>> >>>> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 >>>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>> >>>> Top shows it running out of control: >>>> >>>> root@olive:~ # top -bH >>>> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 >>>> 14:17:32 >>>> 25 processes: 3 running, 22 sleeping >>>> >>>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free >>>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other >>>> Swap: >>>> >>>> >>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>>> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% >>>> bhyve{coco vcpu 1} >>>> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% >>>> bhyve{coco vcpu 0} >>>> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd >>>> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd >>>> 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd >>>> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron >>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% >>>> bhyve{coco mevent} >>>> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh >>>> >>>> Even though, it's just sitting at the login prompt: >>>> >>>> root@olive:/vms/coco # ./vmrun.sh coco >>>> Launching virtual machine "coco" ... >>>> Consoles: userboot >>>> >>>> FreeBSD/amd64 User boot, Revision 1.1 >>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08] >>>> >>>> / >>>> ______ ____ _____ _____ >>>> | ____| | _ \ / ____| __ \ >>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>> | | | | | __/ __/| |_) |____) | |__| | >>>> | | | | | | || | | | >>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>> ` >>>> s` `.....---.......--.``` >>>> -/ >>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>> +. >>>> | | yo`:. :o >>>> `+- >>>> | 1. Boot Multi User [Enter] | y/ -/` -o/ >>>> | 2. Boot [S]ingle User | .- >>>> ::/sy+:. >>>> | 3. [Esc]ape to loader prompt | / `-- >>>> / >>>> | 4. Reboot | `: >>>> :` >>>> | | `: >>>> :` >>>> | Options: | / >>>> / >>>> | 5. Configure Boot [O]ptions... | .- >>>> -. >>>> | | -- -. >>>> | | `:` `:` >>>> | | .-- `--. >>>> | | .---.....----. >>>> +-----------------------------****------------+ >>>> >>>> >>>> >>>> Booting... >>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >>>> The Regents of the University of California. All rights >>>> reserved. >>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>> root@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64 >>>> >>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU) >>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>> Stepping = 7 >>>> Features=0x8fa3ab7f>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE> >>>> Features2=0x83bee217>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,** >>>> TSCDLT,**AESNI,HV> >>>> AMD Features=0x2c100800 >>>> >>>> AMD Features2=0x1 >>>> TSC: P-state invariant >>>> real memory = 5368709120 (5120 MB) >>>> avail memory = 3957534720 (3774 MB) >>>> Event timer "LAPIC" quality 400 >>>> ACPI APIC Table: >>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >>>> FreeBSD/SMP: 2 package(s) x 1 core(s) >>>> cpu0 (BSP): APIC ID: 0 >>>> cpu1 (AP): APIC ID: 1 >>>> random device not loaded; using insecure entropy >>>> ioapic0: Changing APIC ID to 2 >>>> ioapic0 irqs 0-16 on motherboard >>>> kbd0 at kbdmux0 >>>> acpi0: on motherboard >>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>> pcib0: port 0xcf8-0xcff on acpi0 >>>> pci0: on pcib0 >>>> pcib0: no PRT entry for 0.31.INTA >>>> virtio_pci0: port 0x2000-0x201f mem >>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>> vtnet0: on virtio_pci0 >>>> virtio_pci0: host features: 0x18020 >>>> virtio_pci0: negotiated features: 0x18020 >>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>> virtio_pci1: port 0x2040-0x207f mem >>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>> vtblk0: on virtio_pci1 >>>> virtio_pci1: host features: 0x10000004 >>>> virtio_pci1: negotiated features: 0x10000004 >>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>> pci0 >>>> uart2: console (9600,n,8,1) >>>> Timecounters tick every 10.000 msec >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> SMP: AP CPU #1 Launched! >>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf. >>>> >>>> Setting hostid: 0x24bfd025. >>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>> Starting file system checks: >>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% >>>> fragmentation) >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> Mounting local file systems:. >>>> Writing entropy file:. >>>> Setting hostname: cocopuff.example.com. >>>> vtnet0: link state changed to UP >>>> Starting Network: lo0 vtnet0. >>>> lo0: flags=8049 metric 0 mtu 16384 >>>> options=600003 >>>> >>>> inet6 ::1 prefixlen 128 >>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>> inet 127.0.0.1 netmask 0xff000000 >>>> nd6 options=21 >>>> vtnet0: flags=8943>>> MULTICAST> >>>> metric 0 mtu 1500 >>>> options=80028 >>>> >>>> ether 00:a0:98:ac:0d:df >>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>> nd6 options=29 >>>> >>>> media: Ethernet 1000baseT >>>> status: active >>>> Starting devd. >>>> add net default: gateway 172.22.22.252 >>>> add net ::ffff:0.0.0.0: gateway ::1 >>>> add net ::0.0.0.0: gateway ::1 >>>> add net fe80::: gateway ::1 >>>> add net ff02::: gateway ::1 >>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>> Creating and/or trimming log files. >>>> Starting syslogd. >>>> syslogd: timed out waiting for child >>>> /etc/rc: WARNING: failed to start syslogd >>>> Clearing /tmp (X related). >>>> Updating motd:. >>>> Performing sanity check on sshd configuration. >>>> Starting sshd. >>>> Starting cron. >>>> Starting background file system checks in 60 seconds. >>>> >>>> Thu Apr 18 02:06:43 CDT 2013 >>>> >>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>> >>>> login: >>>> >>>> >>>> ========== end of boot sequence >>>> >>>> So, something is broken with SMP on the guest, I'd be happy to help just >>>> let me know. >>>> >>>> Dan >>>> >>>> ______________________________****_________________ >>>> freebsd-virtualization@**freeb**sd.org < >>>> freebsd-virtualization@**freebsd.org>mailing >>>> list >>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-**** >>>> virtualization >>>> >>>> > >>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>> unsubscribe@freebsd.org >>> unsubscribe@freebsd.org >>>> >" >>>> >>>> ______________________________****_________________ >>>> >>> freebsd-virtualization@**freeb**sd.org < >>> freebsd-virtualization@**freebsd.org>mailing >>> list >>> http://lists.freebsd.org/****mailman/listinfo/freebsd-****virtualization >>> >>> > >>> To unsubscribe, send any mail to "freebsd-virtualization-** >>> unsubscribe@freebsd.org >> unsubscribe@freebsd.org >>> >" >>> >>> >> From owner-freebsd-virtualization@FreeBSD.ORG Sat Mar 30 18:00:28 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id DC68D510; Sat, 30 Mar 2013 18:00:28 +0000 (UTC) (envelope-from mack@macktronics.com) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by mx1.freebsd.org (Postfix) with ESMTP id 46159370; Sat, 30 Mar 2013 18:00:28 +0000 (UTC) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by coco.macktronics.com (Postfix) with ESMTP id 7ABB44AC40; Sat, 30 Mar 2013 13:00:27 -0500 (CDT) Date: Sat, 30 Mar 2013 13:00:27 -0500 (CDT) From: Dan Mack To: Neel Natu Subject: Re: bhyve tty / login problems / panic In-Reply-To: Message-ID: <20130330125954.G17063@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> <20130329172023.L15559@coco.macktronics.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Mar 2013 18:00:28 -0000 Yep, no problem. I saw the checkin and started a rebuild a few minutes ago :-) Thanks, Dan On Sat, 30 Mar 2013, Neel Natu wrote: > Hi Dan, > > On Fri, Mar 29, 2013 at 3:20 PM, Dan Mack wrote: > >> >> >> Here you go Neel: >> >> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 >> vcpu0 >> vm exits due to external interrupt 45051 >> number of times hlt was intercepted 2277 >> number of NMIs delivered to vcpu 0 >> vcpu total runtime 105001652482 >> number of ticks vcpu was idle 919 >> vcpu migration across host cpus 13 >> number of times hlt was ignored 0 >> total number of vm exits 9240925 >> root@olive:~ # >> root@olive:~ # >> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 >> vcpu1 >> vm exits due to external interrupt 149431 >> number of times hlt was intercepted 4222 >> number of NMIs delivered to vcpu 0 >> vcpu total runtime 106876622528 >> number of ticks vcpu was idle 869 >> vcpu migration across host cpus 20 >> number of times hlt was ignored 0 >> total number of vm exits 8065909 >> >> > That did not help a whole lot because we were not keeping track of all the > reasons a vcpu could exit. > > I have fixed that in r248935: > http://svnweb.freebsd.org/base?view=revision&revision=248935 > > Do you mind updating your vmm.ko with the change and getting the stats > again? > > best > Neel > > >> Dan >> >> >> >> On Fri, 29 Mar 2013, Neel Natu wrote: >> >> Hi Dan, >>> >>> On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack wrote: >>> >>> >>>> I ran the procsystime dtrace script on bhyve when it was chewing up all >>>> the CPU in vCPU=2 mode, and this is what I see for about 10s of runtime: >>>> >>>> root@olive:/usr/share/dtrace/****toolkit # ./procsystime -n bhyve -aT >>>> >>>> Tracing... Hit Ctrl-C to end... >>>> dtrace: 158536 dynamic variable drops with non-empty dirty list >>>> dtrace: 207447 dynamic variable drops with non-empty dirty list >>>> dtrace: 189205 dynamic variable drops with non-empty dirty list >>>> dtrace: 164341 dynamic variable drops with non-empty dirty list >>>> dtrace: 246307 dynamic variable drops with non-empty dirty list >>>> dtrace: 187640 dynamic variable drops with non-empty dirty list >>>> dtrace: 214771 dynamic variable drops with non-empty dirty list >>>> dtrace: 221265 dynamic variable drops with non-empty dirty list >>>> ^C >>>> dtrace: 243468 dynamic variable drops with non-empty dirty list >>>> >>>> Elapsed Times for processes bhyve, >>>> >>>> SYSCALL TIME (ns) >>>> _umtx_op 18349 >>>> writev 135712 >>>> preadv 16175267 >>>> pwritev 22924378 >>>> ioctl 4353897920 >>>> TOTAL: 4393151626 >>>> >>>> CPU Times for processes bhyve, >>>> >>>> SYSCALL TIME (ns) >>>> _umtx_op 8815 >>>> writev 103145 >>>> pwritev 10647023 >>>> preadv 15159976 >>>> ioctl 3943399889 >>>> TOTAL: 3969318848 >>>> >>>> Syscall Counts for processes bhyve, >>>> >>>> SYSCALL COUNT >>>> _umtx_op 17 >>>> writev 43 >>>> pwritev 639 >>>> preadv 1183 >>>> ioctl 652782 >>>> TOTAL: 654664 >>>> >>>> Not sure if that helps or not. Hotkernel shows the cpu in the >>>> kernel`acpi_cpu_c1 call most of the time, normal ? >>>> >>>> Dan >>>> >>>> root@olive:/usr/share/dtrace/****toolkit # ./hotkernel >>>> >>>> Sampling... Hit Ctrl-C to end. >>>> ^C >>>> FUNCTION COUNT PCNT >>>> zfs.ko`space_map_sync 1 0.0% >>>> kernel`syscall_thread_exit 1 0.0% >>>> kernel`should_yield 1 0.0% >>>> zfs.ko`lzjb_compress 1 0.0% >>>> zfs.ko`dbuf_rele 1 0.0% >>>> zfs.ko`dsl_pool_tempreserve_****space 1 0.0% >>>> >>>> kernel`_bus_dmamap_load_buffer 1 0.0% >>>> kernel`cpu_idleclock 1 0.0% >>>> kernel`_rw_runlock_cookie 1 0.0% >>>> kernel`pmap_enter 1 0.0% >>>> kernel`__lockmgr_args 1 0.0% >>>> zfs.ko`vdev_geom_io_start 1 0.0% >>>> kernel`ahci_ch_intr 1 0.0% >>>> kernel`__rw_rlock 1 0.0% >>>> kernel`VOP_UNLOCK_APV 1 0.0% >>>> zfs.ko`arc_buf_thaw 1 0.0% >>>> kernel`atomic_set_int 1 0.0% >>>> kernel`g_io_schedule_down 1 0.0% >>>> kernel`__mtx_lock_flags 1 0.0% >>>> kernel`_callout_stop_safe 1 0.0% >>>> kernel`sched_idletd 1 0.0% >>>> kernel`_sx_slock 1 0.0% >>>> vmm.ko`lapic_set_intr 1 0.0% >>>> kernel`cpusetobj_ffs 1 0.0% >>>> zfs.ko`dbuf_hold 1 0.0% >>>> vmm.ko`vlapic_intr_accepted 1 0.0% >>>> zfs.ko`sa_attr_op 1 0.0% >>>> zfs.ko`dmu_buf_hold_array_by_****dnode 1 0.0% >>>> >>>> zfs.ko`taskq_run_safe 1 0.0% >>>> kernel`vm_reserv_free_page 1 0.0% >>>> kernel`_sx_sunlock 1 0.0% >>>> kernel`vn_rlimit_fsize 1 0.0% >>>> zfs.ko`vdev_queue_io_to_issue 1 0.0% >>>> kernel`vn_write 1 0.0% >>>> kernel`lock_init 1 0.0% >>>> vmm.ko`vmx_resume 1 0.0% >>>> zfs.ko`zio_gang_tree_free 1 0.0% >>>> zfs.ko`dbuf_rele_and_unlock 1 0.0% >>>> kernel`mnt_vnode_next_active 1 0.0% >>>> kernel`vm_page_remove 1 0.0% >>>> zfs.ko`vdev_queue_deadline_****compare 1 0.0% >>>> >>>> kernel`zone_free_item 1 0.0% >>>> vmm.ko`vmx_launch 1 0.0% >>>> kernel`m_pullup 1 0.0% >>>> kernel`pmap_change_wiring 1 0.0% >>>> kernel`sched_userret 1 0.0% >>>> zfs.ko`dbuf_write_done 1 0.0% >>>> zfs.ko`zio_execute 1 0.0% >>>> kernel`PHYS_TO_VM_PAGE 2 0.0% >>>> zfs.ko`avl_rotation 2 0.0% >>>> kernel`ehci_device_isoc_fs_****enter 2 0.0% >>>> >>>> vmm.ko`vm_exitinfo 2 0.0% >>>> zfs.ko`zfs_freebsd_write 2 0.0% >>>> vmm.ko`vcpu_stats 2 0.0% >>>> kernel`atomic_add_long 2 0.0% >>>> vmm.ko`0xffffffff81 2 0.0% >>>> kernel`vm_reserv_alloc_page 2 0.0% >>>> kernel`pagezero 2 0.0% >>>> kernel`bzero 2 0.0% >>>> zfs.ko`fletcher_2_native 2 0.0% >>>> vmm.ko`vmx_getreg 2 0.0% >>>> vmm.ko`vm_guest_msrs 3 0.0% >>>> vmm.ko`vmx_setreg 3 0.0% >>>> vmm.ko`vm_get_register 3 0.0% >>>> kernel`sx_try_xlock_ 3 0.0% >>>> kernel`vm_radix_lookup_le 3 0.0% >>>> vmm.ko`lapic_pending_intr 4 0.0% >>>> kernel`syscall_thread_enter 4 0.0% >>>> kernel`cpu_set_syscall_retval 4 0.0% >>>> kernel`__mnt_vnode_next_all 4 0.0% >>>> kernel`vfs_stdsync 4 0.0% >>>> vmm.ko`vmm_fetch_instruction 5 0.0% >>>> vmm.ko`vlapic_op_mem_read 6 0.0% >>>> kernel`_sx_xunlock 7 0.0% >>>> vmm.ko`vmm_emulate_instruction 7 0.0% >>>> vmm.ko`vlapic_op_mem_write 7 0.0% >>>> kernel`__mtx_lock_spin_flags 8 0.0% >>>> vmm.ko`lapic_timer_tick 8 0.0% >>>> zfs.ko`list_next 8 0.0% >>>> zfs.ko`l2arc_feed_thread 8 0.0% >>>> kernel`userret 9 0.0% >>>> vmm.ko`vm_gpa2hpa 9 0.0% >>>> vmm.ko`vmm_decode_instruction 11 0.0% >>>> vmm.ko`ept_vmmmap_get 11 0.0% >>>> vmm.ko`vm_lapic 11 0.0% >>>> kernel`spinlock_enter 14 0.0% >>>> kernel`__mtx_lock_sleep 15 0.0% >>>> vmm.ko`vcpu_set_state 16 0.0% >>>> vmm.ko`vm_nmi_pending 16 0.0% >>>> kernel`malloc_type_zone_****allocated 16 0.0% >>>> >>>> kernel`devfs_ioctl_f 16 0.0% >>>> kernel`pmap_kextract 17 0.0% >>>> kernel`sys_ioctl 18 0.0% >>>> kernel`malloc 21 0.0% >>>> kernel`cpu_fetch_syscall_args 24 0.0% >>>> kernel`uma_zfree_arg 28 0.0% >>>> vmm.ko`vlapic_pending_intr 29 0.0% >>>> kernel`trap 30 0.0% >>>> kernel`hpet_get_timecount 31 0.0% >>>> dtrace.ko`dtrace_trap 35 0.0% >>>> vmm.ko`vlapic_update_ppr 39 0.0% >>>> vmm.ko`vmmdev_ioctl 40 0.0% >>>> kernel`copyout 43 0.0% >>>> kernel`critical_enter 44 0.0% >>>> kernel`copyin 51 0.1% >>>> vmm.ko`vmcs_read 53 0.1% >>>> kernel`fpusave 61 0.1% >>>> kernel`uma_zalloc_arg 62 0.1% >>>> kernel`devvn_refthread 69 0.1% >>>> kernel`atomic_load_acq_int 70 0.1% >>>> kernel`fpurestore 70 0.1% >>>> vmm.ko`vlapic_timer_tick 75 0.1% >>>> kernel`amd64_syscall 77 0.1% >>>> kernel`bcopy 91 0.1% >>>> kernel`fpudna 103 0.1% >>>> kernel`free 103 0.1% >>>> kernel`critical_exit 105 0.1% >>>> kernel`dev_relthread 108 0.1% >>>> kernel`spinlock_exit 119 0.1% >>>> kernel`kern_ioctl 138 0.1% >>>> kernel`fpuexit 242 0.3% >>>> kernel`0xffffffff80 306 0.3% >>>> vmm.ko`restore_host_msrs 398 0.4% >>>> vmm.ko`vm_run 428 0.4% >>>> vmm.ko`restore_guest_msrs 433 0.5% >>>> kernel`cpu_idle 906 1.0% >>>> vmm.ko`vmx_run 9266 9.7% >>>> kernel`acpi_cpu_c1 81134 85.2% >>>> >>>> Hope this helps, let me know if you have any ideas on what to try next. >>>> >>>> >>>> Could you try the following command to get vcpu stats? >>> sudo bhyvectl --vm=vm1 --get-stats --cpu=0 >>> sudo bhyvectl --vm=vm1 --get-stats --cpu=1 >>> >>> best >>> Neel >>> >>> >>> Dan >>>> >>>> >>>> >>>> >>>> On Fri, 29 Mar 2013, Dan Mack wrote: >>>> >>>> On Thu, 28 Mar 2013, Peter Grehan wrote: >>>> >>>>> >>>>> Hi Dan, >>>>> >>>>>> >>>>>> Any ideas? >>>>>> >>>>>>> >>>>>>> >>>>>> Just to confirm: your host system is an i7 3930k, which looks like a 6 >>>>>> core system. Should be plenty beefy enough for a 2 vCPU guest. >>>>>> >>>>>> >>>>> That's what I thought, 6 cores, 12 threads, 64 GB RAM. >>>>> >>>>> The issue you are seeing looks a bit like what happens when systems are >>>>> >>>>>> heavily oversubscribed and vCPUs can't get enough run time. >>>>>> >>>>>> >>>>> I was running in on an idle system; only bhyve was running. >>>>> >>>>> >>>>> How much spare RAM is on the host when you started the vm ? Also, is >>>>> >>>>>> there much running on the host when the guest is started ? >>>>>> >>>>>> >>>>> 64GB >>>>> >>>>> Here's some more information (sorry for the formatting): >>>>> >>>>> Case 1: single vCPU via: >>>>> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 >>>>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>>> >>>>> Top shows it idle (note low load, 4GB memory given to guest, still 53GB >>>>> free RAM, nothing else clocking CPU cycles): >>>>> >>>>> root@olive:~ # top -bH >>>>> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 >>>>> 14:11:51 >>>>> 24 processes: 1 running, 23 sleeping >>>>> >>>>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free >>>>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other >>>>> Swap: >>>>> >>>>> >>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>>>> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% >>>>> bhyve{coco vcpu 0} >>>>> 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd >>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% >>>>> bhyve{coco mevent} >>>>> >>>>> >>>>> So, case #1 all is normal. I was able to build a kernel with very good >>>>> performance. >>>>> >>>>> The boot sequence looks like the following: >>>>> >>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>> Launching virtual machine "coco" ... >>>>> Consoles: userboot >>>>> >>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08] >>>>> >>>>> / >>>>> ______ ____ _____ _____ >>>>> | ____| | _ \ / ____| __ \ >>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>> | | | | | | || | | | >>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>> ` >>>>> s` `.....---.......--.``` >>>>> -/ >>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>> +. >>>>> | | yo`:. :o >>>>> `+- >>>>> | 1. Boot Multi User [Enter] | y/ -/` -o/ >>>>> | 2. Boot [S]ingle User | .- >>>>> ::/sy+:. >>>>> | 3. [Esc]ape to loader prompt | / `-- >>>>> / >>>>> | 4. Reboot | `: >>>>> :` >>>>> | | `: >>>>> :` >>>>> | Options: | / >>>>> / >>>>> | 5. Configure Boot [O]ptions... | .- >>>>> -. >>>>> | | -- -. >>>>> | | `:` `:` >>>>> | | .-- `--. >>>>> | | .---.....----. >>>>> +-----------------------------****------------+ >>>>> >>>>> >>>>> >>>>> Booting... >>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >>>>> The Regents of the University of California. All rights >>>>> reserved. >>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>> root@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64 >>>>> >>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU) >>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>> Stepping = 7 >>>>> Features=0x8fa3ab7f>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE> >>>>> Features2=0x83bee217>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,** >>>>> TSCDLT,**AESNI,HV> >>>>> AMD Features=0x2c100800 >>>>> >>>>> AMD Features2=0x1 >>>>> TSC: P-state invariant >>>>> real memory = 5368709120 (5120 MB) >>>>> avail memory = 3957534720 (3774 MB) >>>>> Event timer "LAPIC" quality 400 >>>>> ACPI APIC Table: >>>>> random device not loaded; using insecure entropy >>>>> ioapic0: Changing APIC ID to 1 >>>>> ioapic0 irqs 0-16 on motherboard >>>>> kbd0 at kbdmux0 >>>>> acpi0: on motherboard >>>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>> pcib0: port 0xcf8-0xcff on acpi0 >>>>> pci0: on pcib0 >>>>> pcib0: no PRT entry for 0.31.INTA >>>>> virtio_pci0: port 0x2000-0x201f mem >>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>> vtnet0: on virtio_pci0 >>>>> virtio_pci0: host features: 0x18020 >>>>> virtio_pci0: negotiated features: 0x18020 >>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>> virtio_pci1: port 0x2040-0x207f mem >>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>> vtblk0: on virtio_pci1 >>>>> virtio_pci1: host features: 0x10000004 >>>>> virtio_pci1: negotiated features: 0x10000004 >>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>>> pci0 >>>>> uart2: console (9600,n,8,1) >>>>> Timecounters tick every 10.000 msec >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 >>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf. >>>>> >>>>> Setting hostid: 0x24bfd025. >>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>> Starting file system checks: >>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% >>>>> fragmentation) >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> Mounting local file systems:. >>>>> Writing entropy file:. >>>>> Setting hostname: cocopuff.example.com. >>>>> vtnet0: link state changed to UP >>>>> Starting Network: lo0 vtnet0. >>>>> lo0: flags=8049 metric 0 mtu 16384 >>>>> options=600003 >>>>> >>>>> inet6 ::1 prefixlen 128 >>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>> inet 127.0.0.1 netmask 0xff000000 >>>>> nd6 options=21 >>>>> vtnet0: flags=8943>>>> MULTICAST> >>>>> metric 0 mtu 1500 >>>>> options=80028 >>>>> >>>>> ether 00:a0:98:ac:0d:df >>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>>> nd6 options=29 >>>>> >>>>> media: Ethernet 1000baseT >>>>> status: active >>>>> Starting devd. >>>>> add net default: gateway 172.22.22.252 >>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>> add net ::0.0.0.0: gateway ::1 >>>>> add net fe80::: gateway ::1 >>>>> add net ff02::: gateway ::1 >>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>> Creating and/or trimming log files. >>>>> Starting syslogd. >>>>> Clearing /tmp (X related). >>>>> Updating motd:. >>>>> Performing sanity check on sshd configuration. >>>>> Starting sshd. >>>>> Starting cron. >>>>> Starting background file system checks in 60 seconds. >>>>> >>>>> Fri Mar 29 14:09:20 CDT 2013 >>>>> >>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>> >>>>> login: >>>>> >>>>> >>>>> ====== Next, boot the same image with two (2) vCPU: >>>>> >>>>> Command run by vmrun.sh: >>>>> >>>>> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 >>>>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>>> >>>>> Top shows it running out of control: >>>>> >>>>> root@olive:~ # top -bH >>>>> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 >>>>> 14:17:32 >>>>> 25 processes: 3 running, 22 sleeping >>>>> >>>>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free >>>>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other >>>>> Swap: >>>>> >>>>> >>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>>>> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% >>>>> bhyve{coco vcpu 1} >>>>> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% >>>>> bhyve{coco vcpu 0} >>>>> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd >>>>> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd >>>>> 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd >>>>> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron >>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% >>>>> bhyve{coco mevent} >>>>> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh >>>>> >>>>> Even though, it's just sitting at the login prompt: >>>>> >>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>> Launching virtual machine "coco" ... >>>>> Consoles: userboot >>>>> >>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08] >>>>> >>>>> / >>>>> ______ ____ _____ _____ >>>>> | ____| | _ \ / ____| __ \ >>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>> | | | | | | || | | | >>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>> ` >>>>> s` `.....---.......--.``` >>>>> -/ >>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>> +. >>>>> | | yo`:. :o >>>>> `+- >>>>> | 1. Boot Multi User [Enter] | y/ -/` -o/ >>>>> | 2. Boot [S]ingle User | .- >>>>> ::/sy+:. >>>>> | 3. [Esc]ape to loader prompt | / `-- >>>>> / >>>>> | 4. Reboot | `: >>>>> :` >>>>> | | `: >>>>> :` >>>>> | Options: | / >>>>> / >>>>> | 5. Configure Boot [O]ptions... | .- >>>>> -. >>>>> | | -- -. >>>>> | | `:` `:` >>>>> | | .-- `--. >>>>> | | .---.....----. >>>>> +-----------------------------****------------+ >>>>> >>>>> >>>>> >>>>> Booting... >>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >>>>> The Regents of the University of California. All rights >>>>> reserved. >>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>> root@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64 >>>>> >>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU) >>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>> Stepping = 7 >>>>> Features=0x8fa3ab7f>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE> >>>>> Features2=0x83bee217>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,** >>>>> TSCDLT,**AESNI,HV> >>>>> AMD Features=0x2c100800 >>>>> >>>>> AMD Features2=0x1 >>>>> TSC: P-state invariant >>>>> real memory = 5368709120 (5120 MB) >>>>> avail memory = 3957534720 (3774 MB) >>>>> Event timer "LAPIC" quality 400 >>>>> ACPI APIC Table: >>>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >>>>> FreeBSD/SMP: 2 package(s) x 1 core(s) >>>>> cpu0 (BSP): APIC ID: 0 >>>>> cpu1 (AP): APIC ID: 1 >>>>> random device not loaded; using insecure entropy >>>>> ioapic0: Changing APIC ID to 2 >>>>> ioapic0 irqs 0-16 on motherboard >>>>> kbd0 at kbdmux0 >>>>> acpi0: on motherboard >>>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>> pcib0: port 0xcf8-0xcff on acpi0 >>>>> pci0: on pcib0 >>>>> pcib0: no PRT entry for 0.31.INTA >>>>> virtio_pci0: port 0x2000-0x201f mem >>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>> vtnet0: on virtio_pci0 >>>>> virtio_pci0: host features: 0x18020 >>>>> virtio_pci0: negotiated features: 0x18020 >>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>> virtio_pci1: port 0x2040-0x207f mem >>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>> vtblk0: on virtio_pci1 >>>>> virtio_pci1: host features: 0x10000004 >>>>> virtio_pci1: negotiated features: 0x10000004 >>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>>> pci0 >>>>> uart2: console (9600,n,8,1) >>>>> Timecounters tick every 10.000 msec >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> SMP: AP CPU #1 Launched! >>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf. >>>>> >>>>> Setting hostid: 0x24bfd025. >>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>> Starting file system checks: >>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% >>>>> fragmentation) >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>> Mounting local file systems:. >>>>> Writing entropy file:. >>>>> Setting hostname: cocopuff.example.com. >>>>> vtnet0: link state changed to UP >>>>> Starting Network: lo0 vtnet0. >>>>> lo0: flags=8049 metric 0 mtu 16384 >>>>> options=600003 >>>>> >>>>> inet6 ::1 prefixlen 128 >>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>> inet 127.0.0.1 netmask 0xff000000 >>>>> nd6 options=21 >>>>> vtnet0: flags=8943>>>> MULTICAST> >>>>> metric 0 mtu 1500 >>>>> options=80028 >>>>> >>>>> ether 00:a0:98:ac:0d:df >>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>>> nd6 options=29 >>>>> >>>>> media: Ethernet 1000baseT >>>>> status: active >>>>> Starting devd. >>>>> add net default: gateway 172.22.22.252 >>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>> add net ::0.0.0.0: gateway ::1 >>>>> add net fe80::: gateway ::1 >>>>> add net ff02::: gateway ::1 >>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>> Creating and/or trimming log files. >>>>> Starting syslogd. >>>>> syslogd: timed out waiting for child >>>>> /etc/rc: WARNING: failed to start syslogd >>>>> Clearing /tmp (X related). >>>>> Updating motd:. >>>>> Performing sanity check on sshd configuration. >>>>> Starting sshd. >>>>> Starting cron. >>>>> Starting background file system checks in 60 seconds. >>>>> >>>>> Thu Apr 18 02:06:43 CDT 2013 >>>>> >>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>> >>>>> login: >>>>> >>>>> >>>>> ========== end of boot sequence >>>>> >>>>> So, something is broken with SMP on the guest, I'd be happy to help just >>>>> let me know. >>>>> >>>>> Dan >>>>> >>>>> ______________________________****_________________ >>>>> freebsd-virtualization@**freeb**sd.org < >>>>> freebsd-virtualization@**freebsd.org>mailing >>>>> list >>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-**** >>>>> virtualization >>>>> >>>>>> >>>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>>> unsubscribe@freebsd.org >>>> unsubscribe@freebsd.org >>>>>> " >>>>> >>>>> ______________________________****_________________ >>>>> >>>> freebsd-virtualization@**freeb**sd.org < >>>> freebsd-virtualization@**freebsd.org>mailing >>>> list >>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-****virtualization >>>> >>>>> >>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>> unsubscribe@freebsd.org >>> unsubscribe@freebsd.org >>>>> " >>>> >>>> >>> > From owner-freebsd-virtualization@FreeBSD.ORG Sat Mar 30 19:17:45 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2C66515A for ; Sat, 30 Mar 2013 19:17:45 +0000 (UTC) (envelope-from mack@macktronics.com) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by mx1.freebsd.org (Postfix) with ESMTP id 7CF7E833 for ; Sat, 30 Mar 2013 19:17:44 +0000 (UTC) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by coco.macktronics.com (Postfix) with ESMTP id 96BF94AC40; Sat, 30 Mar 2013 14:17:43 -0500 (CDT) Date: Sat, 30 Mar 2013 14:17:43 -0500 (CDT) From: Dan Mack To: Neel Natu Subject: Re: bhyve tty / login problems / panic In-Reply-To: <20130330125954.G17063@coco.macktronics.com> Message-ID: <20130330141141.B17063@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> <20130329172023.L15559@coco.macktronics.com> <20130330125954.G17063@coco.macktronics.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Mar 2013 19:17:45 -0000 Here you go: - uni-processor output: root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 vcpu0 vm exits due to external interrupt 1330 number of times hlt was intercepted 676 number of times %cr access was intercepted 1 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 4 number of monitor trap exits 0 number of times pause was intercepted 428499 vm exits due to interrupt window opening 1231 vm exits due to nmi window opening 0 number of times in/out was intercepted 423985 number of times cpuid was intercepted 22 vm exits due to nested page fault 7523 number of vm exits for unknown reason 0 number of times astpending at exit 38 number of vm exits handled in userspace 853290 number of NMIs delivered to vcpu 0 vcpu total runtime 6954314530 number of ticks vcpu was idle 59572 vcpu migration across host cpus 29 number of times hlt was ignored 0 total number of vm exits 863271 And here is when running with vcpu==2: root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 vcpu0 vm exits due to external interrupt 19234 number of times hlt was intercepted 1965 number of times %cr access was intercepted 1 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 4 number of monitor trap exits 0 number of times pause was intercepted 3907709 vm exits due to interrupt window opening 8105 vm exits due to nmi window opening 0 number of times in/out was intercepted 457998 number of times cpuid was intercepted 50 vm exits due to nested page fault 940909 number of vm exits for unknown reason 0 number of times astpending at exit 102 number of vm exits handled in userspace 4367825 number of NMIs delivered to vcpu 0 vcpu total runtime 51834598990 number of ticks vcpu was idle 887 vcpu migration across host cpus 25 number of times hlt was ignored 0 total number of vm exits 5335975 root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 vcpu1 vm exits due to external interrupt 51249 number of times hlt was intercepted 3676 number of times %cr access was intercepted 2 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 4 number of monitor trap exits 0 number of times pause was intercepted 4667527 vm exits due to interrupt window opening 32453 vm exits due to nmi window opening 0 number of times in/out was intercepted 57278 number of times cpuid was intercepted 63 vm exits due to nested page fault 266091 number of vm exits for unknown reason 0 number of times astpending at exit 164 number of vm exits handled in userspace 4728481 number of NMIs delivered to vcpu 0 vcpu total runtime 53833466076 number of ticks vcpu was idle 965 vcpu migration across host cpus 16 number of times hlt was ignored 0 total number of vm exits 5078343 The panic comes eventually like this: login: panic: deadlkres: possible deadlock detected for 0xfffffe0090fb5490, blocked for 217798 ticks cpuid = 0 Uptime: 255d6h29m56s Automatic reboot in 15 seconds - press a key on the console to abort Hope this helps, and FYI, here's a dmesg from the host: Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #7 r248935: Sat Mar 30 13:19:43 CDT 2013 root@olive.example.com:/usr/obj/usr/src/sys/MACKGEN amd64 FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3200.19-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0xbfebfbff Features2=0x1fbee3bf AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 63123800064 (60199 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 8 cpu9 (AP): APIC ID: 9 cpu10 (AP): APIC ID: 10 cpu11 (AP): APIC ID: 11 ioapic1 irqs 24-47 on motherboard ioapic0 irqs 0-23 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) acpi0: reservation of f0000000, 18000000 (3) failed cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed03fff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 1.1 on pci0 pci2: on pcib2 xhci0: mem 0xe3400000-0xe3401fff irq 16 at device 0.0 on pci2 usbus0: waiting for BIOS to give up control xhci0: 32 byte context size. usbus0 on xhci0 pcib3: at device 2.0 on pci0 pci3: on pcib3 vgapci0: port 0x3000-0x307f mem 0xe2000000-0xe2ffffff,0xd0000000-0xdfffffff,0xe0000000-0xe1ffffff irq 16 at device 0.0 on pci3 pci3: at device 0.1 (no driver attached) pcib4: at device 3.0 on pci0 pci4: on pcib4 pci0: at device 5.0 (no driver attached) pci0: at device 5.2 (no driver attached) pcib5: at device 17.0 on pci0 pci5: on pcib5 pci0: at device 22.0 (no driver attached) em0: port 0x4040-0x405f mem 0xe3500000-0xe351ffff,0xe3521000-0xe3521fff irq 20 at device 25.0 on pci0 em0: Using an MSI interrupt em0: Ethernet address: 4c:72:b9:d1:b2:d2 ehci0: mem 0xe3601000-0xe36013ff irq 16 at device 26.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci0 pcib6: at device 28.0 on pci0 pci6: on pcib6 pcib7: at device 28.2 on pci0 pci7: on pcib7 pci7: at device 0.0 (no driver attached) pcib8: at device 28.3 on pci0 pci8: on pcib8 xhci1: mem 0xe3200000-0xe3201fff irq 19 at device 0.0 on pci8 usbus2: waiting for BIOS to give up control xhci1: 32 byte context size. usbus2 on xhci1 pcib9: at device 28.5 on pci0 pci9: on pcib9 em1: port 0x1000-0x101f mem 0xe3100000-0xe311ffff,0xe3120000-0xe3123fff irq 17 at device 0.0 on pci9 em1: Using MSIX interrupts with 3 vectors em1: Ethernet address: 4c:72:b9:d1:b2:d3 ehci1: mem 0xe3602000-0xe36023ff irq 23 at device 29.0 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci1 pcib10: at device 30.0 on pci0 pci10: on pcib10 isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0x4068-0x406f,0x4074-0x4077,0x4060-0x4067,0x4070-0x4073,0x4020-0x403f mem 0xe3523000-0xe35237ff irq 18 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 ahciem0: on ahci0 pci0: at device 31.3 (no driver attached) orm0: at iomem 0xd1000-0xd1fff,0xd2000-0xd2fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: cannot reserve I/O port range est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 est2: on cpu2 p4tcc2: on cpu2 est3: on cpu3 p4tcc3: on cpu3 est4: on cpu4 p4tcc4: on cpu4 est5: on cpu5 p4tcc5: on cpu5 est6: on cpu6 p4tcc6: on cpu6 est7: on cpu7 p4tcc7: on cpu7 est8: on cpu8 p4tcc8: on cpu8 est9: on cpu9 p4tcc9: on cpu9 est10: on cpu10 p4tcc10: on cpu10 est11: on cpu11 p4tcc11: on cpu11 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec usbus0: 5.0Gbps Super Speed USB v3.0 usbus1: 480Mbps High Speed USB v2.0 usbus2: 5.0Gbps Super Speed USB v3.0 usbus3: 480Mbps High Speed USB v2.0 ugen1.1: at usbus1 uhub0: on usbus1 ugen0.1: <0x1033> at usbus0 uhub1: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 ugen3.1: at usbus3 uhub2: on usbus3 ugen2.1: <0x1033> at usbus2 uhub3: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus2 ses0 at ahciem0 bus 0 scbus6 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-9 SATA 3.x device ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 122104MB (250069680 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: ATA-9 SATA 3.x device ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C) ada1: Previously was known as ad6 ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 ada2: ATA-8 SATA 3.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada2: Previously was known as ad8 ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 ada3: ATA-8 SATA 3.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada3: Previously was known as ad10 ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 ada4: ATA-8 SATA 3.x device ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada4: Previously was known as ad12 ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 ada5: ATA-8 SATA 3.x device ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada5: Previously was known as ad14 lapic1: Forcing LINT1 to edge trigger SMP: AP CPU #1 Launched! lapic4: Forcing LINT1 to edge trigger SMP: AP CPU #4 Launched! lapic11: Forcing LINT1 to edge trigger SMP: AP CPU #11 Launched! lapic7: Forcing LINT1 to edge trigger SMP: AP CPU #7 Launched! lapic5: Forcing LINT1 to edge trigger SMP: AP CPU #5 Launched! lapic9: Forcing LINT1 to edge trigger SMP: AP CPU #9 Launched! lapic2: Forcing LINT1 to edge trigger SMP: AP CPU #2 Launched! lapic3: Forcing LINT1 to edge trigger SMP: AP CPU #3 Launched! lapic10: Forcing LINT1 to edge trigger SMP: AP CPU #10 Launched! lapic6: Forcing LINT1 to edge trigger SMP: AP CPU #6 Launched! lapic8: Forcing LINT1 to edge trigger SMP: AP CPU #8 Launched! uhub3: 4 ports with 4 removable, self powered uhub1: 4 ports with 4 removable, self powered Root mount waiting for: usbus3 usbus1 uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered Root mount waiting for: usbus3 usbus1 ugen1.2: at usbus1 uhub4: on usbus1 ugen3.2: at usbus3 uhub5: on usbus3 Root mount waiting for: usbus3 usbus1 uhub4: 6 ports with 6 removable, self powered uhub5: 8 ports with 8 removable, self powered ugen1.3: at usbus1 ukbd0: on usbus1 kbd2 at ukbd0 ugen3.3: at usbus3 Root mount waiting for: usbus3 ugen3.4: at usbus3 Trying to mount root from zfs:tron []... ums0: on usbus1 ums0: 5 buttons and [XYZ] coordinates ID=0 run0: <1.0> on usbus3 run0: MAC/BBP RT3070 (rev 0x0201), RF RT3020 (MIMO 1T1R), address 94:db:c9:e3:6e:c8 ubt0: on usbus3 WARNING: attempt to domain_add(bluetooth) after domainfinalize() WARNING: attempt to domain_add(netgraph) after domainfinalize() tap0: Ethernet address: 00:bd:8c:54:07:00 tap0: link state changed to UP bridge0: Ethernet address: 02:4c:89:ce:33:00 tap0: promiscuous mode enabled bridge0: link state changed to UP em1: link state changed to DOWN em1: promiscuous mode enabled em1: link state changed to UP tap0: link state changed to DOWN tap0: link state changed to UP tap0: link state changed to DOWN tap0: link state changed to UP > > Yep, no problem. I saw the checkin and started a rebuild a few minutes ago > :-) > > Thanks, > > Dan > > On Sat, 30 Mar 2013, Neel Natu wrote: > >> Hi Dan, >> >> On Fri, Mar 29, 2013 at 3:20 PM, Dan Mack wrote: >> >>> >>> >>> Here you go Neel: >>> >>> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 >>> vcpu0 >>> vm exits due to external interrupt 45051 >>> number of times hlt was intercepted 2277 >>> number of NMIs delivered to vcpu 0 >>> vcpu total runtime 105001652482 >>> number of ticks vcpu was idle 919 >>> vcpu migration across host cpus 13 >>> number of times hlt was ignored 0 >>> total number of vm exits 9240925 >>> root@olive:~ # >>> root@olive:~ # >>> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 >>> vcpu1 >>> vm exits due to external interrupt 149431 >>> number of times hlt was intercepted 4222 >>> number of NMIs delivered to vcpu 0 >>> vcpu total runtime 106876622528 >>> number of ticks vcpu was idle 869 >>> vcpu migration across host cpus 20 >>> number of times hlt was ignored 0 >>> total number of vm exits 8065909 >>> >>> >> That did not help a whole lot because we were not keeping track of all the >> reasons a vcpu could exit. >> >> I have fixed that in r248935: >> http://svnweb.freebsd.org/base?view=revision&revision=248935 >> >> Do you mind updating your vmm.ko with the change and getting the stats >> again? >> >> best >> Neel >> >> >>> Dan >>> >>> >>> >>> On Fri, 29 Mar 2013, Neel Natu wrote: >>> >>> Hi Dan, >>>> >>>> On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack wrote: >>>> >>>> >>>>> I ran the procsystime dtrace script on bhyve when it was chewing up all >>>>> the CPU in vCPU=2 mode, and this is what I see for about 10s of runtime: >>>>> >>>>> root@olive:/usr/share/dtrace/****toolkit # ./procsystime -n bhyve -aT >>>>> >>>>> Tracing... Hit Ctrl-C to end... >>>>> dtrace: 158536 dynamic variable drops with non-empty dirty list >>>>> dtrace: 207447 dynamic variable drops with non-empty dirty list >>>>> dtrace: 189205 dynamic variable drops with non-empty dirty list >>>>> dtrace: 164341 dynamic variable drops with non-empty dirty list >>>>> dtrace: 246307 dynamic variable drops with non-empty dirty list >>>>> dtrace: 187640 dynamic variable drops with non-empty dirty list >>>>> dtrace: 214771 dynamic variable drops with non-empty dirty list >>>>> dtrace: 221265 dynamic variable drops with non-empty dirty list >>>>> ^C >>>>> dtrace: 243468 dynamic variable drops with non-empty dirty list >>>>> >>>>> Elapsed Times for processes bhyve, >>>>> >>>>> SYSCALL TIME (ns) >>>>> _umtx_op 18349 >>>>> writev 135712 >>>>> preadv 16175267 >>>>> pwritev 22924378 >>>>> ioctl 4353897920 >>>>> TOTAL: 4393151626 >>>>> >>>>> CPU Times for processes bhyve, >>>>> >>>>> SYSCALL TIME (ns) >>>>> _umtx_op 8815 >>>>> writev 103145 >>>>> pwritev 10647023 >>>>> preadv 15159976 >>>>> ioctl 3943399889 >>>>> TOTAL: 3969318848 >>>>> >>>>> Syscall Counts for processes bhyve, >>>>> >>>>> SYSCALL COUNT >>>>> _umtx_op 17 >>>>> writev 43 >>>>> pwritev 639 >>>>> preadv 1183 >>>>> ioctl 652782 >>>>> TOTAL: 654664 >>>>> >>>>> Not sure if that helps or not. Hotkernel shows the cpu in the >>>>> kernel`acpi_cpu_c1 call most of the time, normal ? >>>>> >>>>> Dan >>>>> >>>>> root@olive:/usr/share/dtrace/****toolkit # ./hotkernel >>>>> >>>>> Sampling... Hit Ctrl-C to end. >>>>> ^C >>>>> FUNCTION COUNT PCNT >>>>> zfs.ko`space_map_sync 1 0.0% >>>>> kernel`syscall_thread_exit 1 0.0% >>>>> kernel`should_yield 1 0.0% >>>>> zfs.ko`lzjb_compress 1 0.0% >>>>> zfs.ko`dbuf_rele 1 0.0% >>>>> zfs.ko`dsl_pool_tempreserve_****space 1 0.0% >>>>> >>>>> kernel`_bus_dmamap_load_buffer 1 0.0% >>>>> kernel`cpu_idleclock 1 0.0% >>>>> kernel`_rw_runlock_cookie 1 0.0% >>>>> kernel`pmap_enter 1 0.0% >>>>> kernel`__lockmgr_args 1 0.0% >>>>> zfs.ko`vdev_geom_io_start 1 0.0% >>>>> kernel`ahci_ch_intr 1 0.0% >>>>> kernel`__rw_rlock 1 0.0% >>>>> kernel`VOP_UNLOCK_APV 1 0.0% >>>>> zfs.ko`arc_buf_thaw 1 0.0% >>>>> kernel`atomic_set_int 1 0.0% >>>>> kernel`g_io_schedule_down 1 0.0% >>>>> kernel`__mtx_lock_flags 1 0.0% >>>>> kernel`_callout_stop_safe 1 0.0% >>>>> kernel`sched_idletd 1 0.0% >>>>> kernel`_sx_slock 1 0.0% >>>>> vmm.ko`lapic_set_intr 1 0.0% >>>>> kernel`cpusetobj_ffs 1 0.0% >>>>> zfs.ko`dbuf_hold 1 0.0% >>>>> vmm.ko`vlapic_intr_accepted 1 0.0% >>>>> zfs.ko`sa_attr_op 1 0.0% >>>>> zfs.ko`dmu_buf_hold_array_by_****dnode 1 0.0% >>>>> >>>>> zfs.ko`taskq_run_safe 1 0.0% >>>>> kernel`vm_reserv_free_page 1 0.0% >>>>> kernel`_sx_sunlock 1 0.0% >>>>> kernel`vn_rlimit_fsize 1 0.0% >>>>> zfs.ko`vdev_queue_io_to_issue 1 0.0% >>>>> kernel`vn_write 1 0.0% >>>>> kernel`lock_init 1 0.0% >>>>> vmm.ko`vmx_resume 1 0.0% >>>>> zfs.ko`zio_gang_tree_free 1 0.0% >>>>> zfs.ko`dbuf_rele_and_unlock 1 0.0% >>>>> kernel`mnt_vnode_next_active 1 0.0% >>>>> kernel`vm_page_remove 1 0.0% >>>>> zfs.ko`vdev_queue_deadline_****compare 1 0.0% >>>>> >>>>> kernel`zone_free_item 1 0.0% >>>>> vmm.ko`vmx_launch 1 0.0% >>>>> kernel`m_pullup 1 0.0% >>>>> kernel`pmap_change_wiring 1 0.0% >>>>> kernel`sched_userret 1 0.0% >>>>> zfs.ko`dbuf_write_done 1 0.0% >>>>> zfs.ko`zio_execute 1 0.0% >>>>> kernel`PHYS_TO_VM_PAGE 2 0.0% >>>>> zfs.ko`avl_rotation 2 0.0% >>>>> kernel`ehci_device_isoc_fs_****enter 2 0.0% >>>>> >>>>> vmm.ko`vm_exitinfo 2 0.0% >>>>> zfs.ko`zfs_freebsd_write 2 0.0% >>>>> vmm.ko`vcpu_stats 2 0.0% >>>>> kernel`atomic_add_long 2 0.0% >>>>> vmm.ko`0xffffffff81 2 0.0% >>>>> kernel`vm_reserv_alloc_page 2 0.0% >>>>> kernel`pagezero 2 0.0% >>>>> kernel`bzero 2 0.0% >>>>> zfs.ko`fletcher_2_native 2 0.0% >>>>> vmm.ko`vmx_getreg 2 0.0% >>>>> vmm.ko`vm_guest_msrs 3 0.0% >>>>> vmm.ko`vmx_setreg 3 0.0% >>>>> vmm.ko`vm_get_register 3 0.0% >>>>> kernel`sx_try_xlock_ 3 0.0% >>>>> kernel`vm_radix_lookup_le 3 0.0% >>>>> vmm.ko`lapic_pending_intr 4 0.0% >>>>> kernel`syscall_thread_enter 4 0.0% >>>>> kernel`cpu_set_syscall_retval 4 0.0% >>>>> kernel`__mnt_vnode_next_all 4 0.0% >>>>> kernel`vfs_stdsync 4 0.0% >>>>> vmm.ko`vmm_fetch_instruction 5 0.0% >>>>> vmm.ko`vlapic_op_mem_read 6 0.0% >>>>> kernel`_sx_xunlock 7 0.0% >>>>> vmm.ko`vmm_emulate_instruction 7 0.0% >>>>> vmm.ko`vlapic_op_mem_write 7 0.0% >>>>> kernel`__mtx_lock_spin_flags 8 0.0% >>>>> vmm.ko`lapic_timer_tick 8 0.0% >>>>> zfs.ko`list_next 8 0.0% >>>>> zfs.ko`l2arc_feed_thread 8 0.0% >>>>> kernel`userret 9 0.0% >>>>> vmm.ko`vm_gpa2hpa 9 0.0% >>>>> vmm.ko`vmm_decode_instruction 11 0.0% >>>>> vmm.ko`ept_vmmmap_get 11 0.0% >>>>> vmm.ko`vm_lapic 11 0.0% >>>>> kernel`spinlock_enter 14 0.0% >>>>> kernel`__mtx_lock_sleep 15 0.0% >>>>> vmm.ko`vcpu_set_state 16 0.0% >>>>> vmm.ko`vm_nmi_pending 16 0.0% >>>>> kernel`malloc_type_zone_****allocated 16 0.0% >>>>> >>>>> kernel`devfs_ioctl_f 16 0.0% >>>>> kernel`pmap_kextract 17 0.0% >>>>> kernel`sys_ioctl 18 0.0% >>>>> kernel`malloc 21 0.0% >>>>> kernel`cpu_fetch_syscall_args 24 0.0% >>>>> kernel`uma_zfree_arg 28 0.0% >>>>> vmm.ko`vlapic_pending_intr 29 0.0% >>>>> kernel`trap 30 0.0% >>>>> kernel`hpet_get_timecount 31 0.0% >>>>> dtrace.ko`dtrace_trap 35 0.0% >>>>> vmm.ko`vlapic_update_ppr 39 0.0% >>>>> vmm.ko`vmmdev_ioctl 40 0.0% >>>>> kernel`copyout 43 0.0% >>>>> kernel`critical_enter 44 0.0% >>>>> kernel`copyin 51 0.1% >>>>> vmm.ko`vmcs_read 53 0.1% >>>>> kernel`fpusave 61 0.1% >>>>> kernel`uma_zalloc_arg 62 0.1% >>>>> kernel`devvn_refthread 69 0.1% >>>>> kernel`atomic_load_acq_int 70 0.1% >>>>> kernel`fpurestore 70 0.1% >>>>> vmm.ko`vlapic_timer_tick 75 0.1% >>>>> kernel`amd64_syscall 77 0.1% >>>>> kernel`bcopy 91 0.1% >>>>> kernel`fpudna 103 0.1% >>>>> kernel`free 103 0.1% >>>>> kernel`critical_exit 105 0.1% >>>>> kernel`dev_relthread 108 0.1% >>>>> kernel`spinlock_exit 119 0.1% >>>>> kernel`kern_ioctl 138 0.1% >>>>> kernel`fpuexit 242 0.3% >>>>> kernel`0xffffffff80 306 0.3% >>>>> vmm.ko`restore_host_msrs 398 0.4% >>>>> vmm.ko`vm_run 428 0.4% >>>>> vmm.ko`restore_guest_msrs 433 0.5% >>>>> kernel`cpu_idle 906 1.0% >>>>> vmm.ko`vmx_run 9266 9.7% >>>>> kernel`acpi_cpu_c1 81134 85.2% >>>>> >>>>> Hope this helps, let me know if you have any ideas on what to try next. >>>>> >>>>> >>>>> Could you try the following command to get vcpu stats? >>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=0 >>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=1 >>>> >>>> best >>>> Neel >>>> >>>> >>>> Dan >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, 29 Mar 2013, Dan Mack wrote: >>>>> >>>>> On Thu, 28 Mar 2013, Peter Grehan wrote: >>>>> >>>>>> >>>>>> Hi Dan, >>>>>> >>>>>>> >>>>>>> Any ideas? >>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Just to confirm: your host system is an i7 3930k, which looks like a 6 >>>>>>> core system. Should be plenty beefy enough for a 2 vCPU guest. >>>>>>> >>>>>>> >>>>>> That's what I thought, 6 cores, 12 threads, 64 GB RAM. >>>>>> >>>>>> The issue you are seeing looks a bit like what happens when systems >>>>>> are >>>>>> >>>>>>> heavily oversubscribed and vCPUs can't get enough run time. >>>>>>> >>>>>>> >>>>>> I was running in on an idle system; only bhyve was running. >>>>>> >>>>>> >>>>>> How much spare RAM is on the host when you started the vm ? Also, is >>>>>> >>>>>>> there much running on the host when the guest is started ? >>>>>>> >>>>>>> >>>>>> 64GB >>>>>> >>>>>> Here's some more information (sorry for the formatting): >>>>>> >>>>>> Case 1: single vCPU via: >>>>>> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 >>>>>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>>>> >>>>>> Top shows it idle (note low load, 4GB memory given to guest, still 53GB >>>>>> free RAM, nothing else clocking CPU cycles): >>>>>> >>>>>> root@olive:~ # top -bH >>>>>> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 >>>>>> 14:11:51 >>>>>> 24 processes: 1 running, 23 sleeping >>>>>> >>>>>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free >>>>>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other >>>>>> Swap: >>>>>> >>>>>> >>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>>>>> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% >>>>>> bhyve{coco vcpu 0} >>>>>> 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd >>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>>> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% >>>>>> bhyve{coco mevent} >>>>>> >>>>>> >>>>>> So, case #1 all is normal. I was able to build a kernel with very good >>>>>> performance. >>>>>> >>>>>> The boot sequence looks like the following: >>>>>> >>>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>>> Launching virtual machine "coco" ... >>>>>> Consoles: userboot >>>>>> >>>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08] >>>>>> >>>>>> / >>>>>> ______ ____ _____ _____ >>>>>> | ____| | _ \ / ____| __ \ >>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>>> | | | | | | || | | | >>>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>>> ` >>>>>> s` `.....---.......--.``` >>>>>> -/ >>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>>> +. >>>>>> | | yo`:. :o >>>>>> `+- >>>>>> | 1. Boot Multi User [Enter] | y/ -/` >>>>>> -o/ >>>>>> | 2. Boot [S]ingle User | .- >>>>>> ::/sy+:. >>>>>> | 3. [Esc]ape to loader prompt | / `-- >>>>>> / >>>>>> | 4. Reboot | `: >>>>>> :` >>>>>> | | `: >>>>>> :` >>>>>> | Options: | / >>>>>> / >>>>>> | 5. Configure Boot [O]ptions... | .- >>>>>> -. >>>>>> | | -- >>>>>> -. >>>>>> | | `:` `:` >>>>>> | | .-- `--. >>>>>> | | .---.....----. >>>>>> +-----------------------------****------------+ >>>>>> >>>>>> >>>>>> >>>>>> Booting... >>>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >>>>>> 1994 >>>>>> The Regents of the University of California. All rights >>>>>> reserved. >>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>>> root@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64 >>>>>> >>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class >>>>>> CPU) >>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>>> Stepping = 7 >>>>>> Features=0x8fa3ab7f>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE> >>>>>> Features2=0x83bee217>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,** >>>>>> TSCDLT,**AESNI,HV> >>>>>> AMD Features=0x2c100800 >>>>>> >>>>>> AMD Features2=0x1 >>>>>> TSC: P-state invariant >>>>>> real memory = 5368709120 (5120 MB) >>>>>> avail memory = 3957534720 (3774 MB) >>>>>> Event timer "LAPIC" quality 400 >>>>>> ACPI APIC Table: >>>>>> random device not loaded; using insecure entropy >>>>>> ioapic0: Changing APIC ID to 1 >>>>>> ioapic0 irqs 0-16 on motherboard >>>>>> kbd0 at kbdmux0 >>>>>> acpi0: on motherboard >>>>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>>> pcib0: port 0xcf8-0xcff on acpi0 >>>>>> pci0: on pcib0 >>>>>> pcib0: no PRT entry for 0.31.INTA >>>>>> virtio_pci0: port 0x2000-0x201f mem >>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>>> vtnet0: on virtio_pci0 >>>>>> virtio_pci0: host features: 0x18020 >>>>>> virtio_pci0: negotiated features: 0x18020 >>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>>> virtio_pci1: port 0x2040-0x207f mem >>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>>> vtblk0: on virtio_pci1 >>>>>> virtio_pci1: host features: 0x10000004 >>>>>> virtio_pci1: negotiated features: 0x10000004 >>>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>>>> pci0 >>>>>> uart2: console (9600,n,8,1) >>>>>> Timecounters tick every 10.000 msec >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 >>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf. >>>>>> >>>>>> Setting hostid: 0x24bfd025. >>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>>> Starting file system checks: >>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% >>>>>> fragmentation) >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> Mounting local file systems:. >>>>>> Writing entropy file:. >>>>>> Setting hostname: cocopuff.example.com. >>>>>> vtnet0: link state changed to UP >>>>>> Starting Network: lo0 vtnet0. >>>>>> lo0: flags=8049 metric 0 mtu 16384 >>>>>> options=600003 >>>>>> >>>>>> inet6 ::1 prefixlen 128 >>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>>> inet 127.0.0.1 netmask 0xff000000 >>>>>> nd6 options=21 >>>>>> vtnet0: flags=8943>>>>> MULTICAST> >>>>>> metric 0 mtu 1500 >>>>>> options=80028 >>>>>> >>>>>> ether 00:a0:98:ac:0d:df >>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>>>> nd6 options=29 >>>>>> >>>>>> media: Ethernet 1000baseT >>>>>> status: active >>>>>> Starting devd. >>>>>> add net default: gateway 172.22.22.252 >>>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>>> add net ::0.0.0.0: gateway ::1 >>>>>> add net fe80::: gateway ::1 >>>>>> add net ff02::: gateway ::1 >>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>>> Creating and/or trimming log files. >>>>>> Starting syslogd. >>>>>> Clearing /tmp (X related). >>>>>> Updating motd:. >>>>>> Performing sanity check on sshd configuration. >>>>>> Starting sshd. >>>>>> Starting cron. >>>>>> Starting background file system checks in 60 seconds. >>>>>> >>>>>> Fri Mar 29 14:09:20 CDT 2013 >>>>>> >>>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>>> >>>>>> login: >>>>>> >>>>>> >>>>>> ====== Next, boot the same image with two (2) vCPU: >>>>>> >>>>>> Command run by vmrun.sh: >>>>>> >>>>>> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 >>>>>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>>>> >>>>>> Top shows it running out of control: >>>>>> >>>>>> root@olive:~ # top -bH >>>>>> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 >>>>>> 14:17:32 >>>>>> 25 processes: 3 running, 22 sleeping >>>>>> >>>>>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free >>>>>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M >>>>>> Other >>>>>> Swap: >>>>>> >>>>>> >>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>>>>> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% >>>>>> bhyve{coco vcpu 1} >>>>>> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% >>>>>> bhyve{coco vcpu 0} >>>>>> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd >>>>>> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd >>>>>> 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd >>>>>> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron >>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>>> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% >>>>>> bhyve{coco mevent} >>>>>> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh >>>>>> >>>>>> Even though, it's just sitting at the login prompt: >>>>>> >>>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>>> Launching virtual machine "coco" ... >>>>>> Consoles: userboot >>>>>> >>>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08] >>>>>> >>>>>> / >>>>>> ______ ____ _____ _____ >>>>>> | ____| | _ \ / ____| __ \ >>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>>> | | | | | | || | | | >>>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>>> ` >>>>>> s` `.....---.......--.``` >>>>>> -/ >>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>>> +. >>>>>> | | yo`:. :o >>>>>> `+- >>>>>> | 1. Boot Multi User [Enter] | y/ -/` >>>>>> -o/ >>>>>> | 2. Boot [S]ingle User | .- >>>>>> ::/sy+:. >>>>>> | 3. [Esc]ape to loader prompt | / `-- >>>>>> / >>>>>> | 4. Reboot | `: >>>>>> :` >>>>>> | | `: >>>>>> :` >>>>>> | Options: | / >>>>>> / >>>>>> | 5. Configure Boot [O]ptions... | .- >>>>>> -. >>>>>> | | -- >>>>>> -. >>>>>> | | `:` `:` >>>>>> | | .-- `--. >>>>>> | | .---.....----. >>>>>> +-----------------------------****------------+ >>>>>> >>>>>> >>>>>> >>>>>> Booting... >>>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >>>>>> 1994 >>>>>> The Regents of the University of California. All rights >>>>>> reserved. >>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>>> root@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64 >>>>>> >>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class >>>>>> CPU) >>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>>> Stepping = 7 >>>>>> Features=0x8fa3ab7f>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE> >>>>>> Features2=0x83bee217>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,** >>>>>> TSCDLT,**AESNI,HV> >>>>>> AMD Features=0x2c100800 >>>>>> >>>>>> AMD Features2=0x1 >>>>>> TSC: P-state invariant >>>>>> real memory = 5368709120 (5120 MB) >>>>>> avail memory = 3957534720 (3774 MB) >>>>>> Event timer "LAPIC" quality 400 >>>>>> ACPI APIC Table: >>>>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >>>>>> FreeBSD/SMP: 2 package(s) x 1 core(s) >>>>>> cpu0 (BSP): APIC ID: 0 >>>>>> cpu1 (AP): APIC ID: 1 >>>>>> random device not loaded; using insecure entropy >>>>>> ioapic0: Changing APIC ID to 2 >>>>>> ioapic0 irqs 0-16 on motherboard >>>>>> kbd0 at kbdmux0 >>>>>> acpi0: on motherboard >>>>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>>> pcib0: port 0xcf8-0xcff on acpi0 >>>>>> pci0: on pcib0 >>>>>> pcib0: no PRT entry for 0.31.INTA >>>>>> virtio_pci0: port 0x2000-0x201f mem >>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>>> vtnet0: on virtio_pci0 >>>>>> virtio_pci0: host features: 0x18020 >>>>>> virtio_pci0: negotiated features: 0x18020 >>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>>> virtio_pci1: port 0x2040-0x207f mem >>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>>> vtblk0: on virtio_pci1 >>>>>> virtio_pci1: host features: 0x10000004 >>>>>> virtio_pci1: negotiated features: 0x10000004 >>>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>>>> pci0 >>>>>> uart2: console (9600,n,8,1) >>>>>> Timecounters tick every 10.000 msec >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> SMP: AP CPU #1 Launched! >>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf. >>>>>> >>>>>> Setting hostid: 0x24bfd025. >>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>>> Starting file system checks: >>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% >>>>>> fragmentation) >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> Mounting local file systems:. >>>>>> Writing entropy file:. >>>>>> Setting hostname: cocopuff.example.com. >>>>>> vtnet0: link state changed to UP >>>>>> Starting Network: lo0 vtnet0. >>>>>> lo0: flags=8049 metric 0 mtu 16384 >>>>>> options=600003 >>>>>> >>>>>> inet6 ::1 prefixlen 128 >>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>>> inet 127.0.0.1 netmask 0xff000000 >>>>>> nd6 options=21 >>>>>> vtnet0: flags=8943>>>>> MULTICAST> >>>>>> metric 0 mtu 1500 >>>>>> options=80028 >>>>>> >>>>>> ether 00:a0:98:ac:0d:df >>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>>>> nd6 options=29 >>>>>> >>>>>> media: Ethernet 1000baseT >>>>>> status: active >>>>>> Starting devd. >>>>>> add net default: gateway 172.22.22.252 >>>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>>> add net ::0.0.0.0: gateway ::1 >>>>>> add net fe80::: gateway ::1 >>>>>> add net ff02::: gateway ::1 >>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>>> Creating and/or trimming log files. >>>>>> Starting syslogd. >>>>>> syslogd: timed out waiting for child >>>>>> /etc/rc: WARNING: failed to start syslogd >>>>>> Clearing /tmp (X related). >>>>>> Updating motd:. >>>>>> Performing sanity check on sshd configuration. >>>>>> Starting sshd. >>>>>> Starting cron. >>>>>> Starting background file system checks in 60 seconds. >>>>>> >>>>>> Thu Apr 18 02:06:43 CDT 2013 >>>>>> >>>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>>> >>>>>> login: >>>>>> >>>>>> >>>>>> ========== end of boot sequence >>>>>> >>>>>> So, something is broken with SMP on the guest, I'd be happy to help >>>>>> just >>>>>> let me know. >>>>>> >>>>>> Dan >>>>>> >>>>>> ______________________________****_________________ >>>>>> freebsd-virtualization@**freeb**sd.org < >>>>>> freebsd-virtualization@**freebsd.org>mailing >>>>>> list >>>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-**** >>>>>> virtualization >>>>>> >>>>>>> >>>>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>>>> unsubscribe@freebsd.org >>>>> unsubscribe@freebsd.org >>>>>>> " >>>>>> >>>>>> ______________________________****_________________ >>>>>> >>>>> freebsd-virtualization@**freeb**sd.org < >>>>> freebsd-virtualization@**freebsd.org>mailing >>>>> list >>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-****virtualization >>>>> >>>>>> >>>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>>> unsubscribe@freebsd.org >>>> unsubscribe@freebsd.org >>>>>> " >>>>> >>>>> >>>> >> > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org" >