From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 09:27:07 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 322B5BD7 for ; Mon, 22 Jun 2015 09:27:07 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 6895A9D8 for ; Mon, 22 Jun 2015 09:27:05 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA24571 for ; Mon, 22 Jun 2015 12:27:03 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z6y0N-0004xY-BN for freebsd-virtualization@freebsd.org; Mon, 22 Jun 2015 12:27:03 +0300 Message-ID: <5587D4AE.3050508@FreeBSD.org> Date: Mon, 22 Jun 2015 12:26:06 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "freebsd-virtualization@freebsd.org" Subject: header type for bhyve host-pci bridge Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 09:27:07 -0000 diff --git a/usr.sbin/bhyve/pci_hostbridge.c b/usr.sbin/bhyve/pci_hostbridge.c index 54a25ae..5c9ea28 100644 --- a/usr.sbin/bhyve/pci_hostbridge.c +++ b/usr.sbin/bhyve/pci_hostbridge.c @@ -38,7 +38,7 @@ pci_hostbridge_init(struct vmctx *ctx, struct pci_devinst *pi, char *opts) /* config space */ pci_set_cfgdata16(pi, PCIR_VENDOR, 0x1275); /* NetApp */ pci_set_cfgdata16(pi, PCIR_DEVICE, 0x1275); /* NetApp */ - pci_set_cfgdata8(pi, PCIR_HDRTYPE, PCIM_HDRTYPE_BRIDGE); + pci_set_cfgdata8(pi, PCIR_HDRTYPE, PCIM_HDRTYPE_NORMAL); pci_set_cfgdata8(pi, PCIR_CLASS, PCIC_BRIDGE); pci_set_cfgdata8(pi, PCIR_SUBCLASS, PCIS_BRIDGE_HOST); It seems that the normal header type is expected for a Host-PCI bridge. I see that on real hardware. Also, Linux complains about the current header type: pci 0000:00:00.0: ignoring class 0x060000 (doesn't match header type 01) pci 0000:00:00.0: bridge configuration invalid ([bus 00-00]), reconfiguring pci 0000:00:00.0: can't allocate child bus 00 from [bus 00] Same with lspci: $ lspci -v 00:00.0 Non-VGA unclassified device: Network Appliance Corporation Device 1275 !!! Invalid class 0000 for header type 01 Flags: bus master, fast devsel, latency 0 Bus: primary=00, secondary=00, subordinate=00, sec-latency=0 I/O behind bridge: 00000000-00000fff Memory behind bridge: 00000000-000fffff Prefetchable memory behind bridge: 00000000-000fffff Capabilities: [40] Express Root Port (Slot-), MSI 00 I think that PCIM_HDRTYPE_BRIDGE is for PCI-PCI bridges. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 11:15:25 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 05ADCF77 for ; Mon, 22 Jun 2015 11:15:25 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 234F61A1 for ; Mon, 22 Jun 2015 11:15:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA25992 for ; Mon, 22 Jun 2015 14:15:21 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z6zhB-00053Z-FW for freebsd-virtualization@freebsd.org; Mon, 22 Jun 2015 14:15:21 +0300 Message-ID: <5587EE05.2020001@FreeBSD.org> Date: Mon, 22 Jun 2015 14:14:13 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "freebsd-virtualization@freebsd.org" Subject: bhyve: centos 7.1 with multiple virtual processors Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 11:15:25 -0000 If I run a CentOS 7.1 VM with more than one CPU more often than not it would hang on startup and bhyve would start spinning. The following are the last messages seen in the VM: Switching to clocksource hpet ------------[ cut here ]------------ WARNING: at kernel/time/clockevents.c:239 clockevents_program_event+0xdb/0xf0() Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1 Hardware name: BHYVE, BIOS 1.00 03/14/2014 0000000000000000 00000000cab5bdb6 ffff88003fc03e08 ffffffff81604eaa ffff88003fc03e40 ffffffff8106e34b 80000000000f423f 80000000000f423f ffffffff81915440 0000000000000000 0000000000000000 ffff88003fc03e50 Call Trace: [] dump_stack+0x19/0x1b [] warn_slowpath_common+0x6b/0xb0 [] warn_slowpath_null+0x1a/0x20 [] clockevents_program_event+0xdb/0xf0 [] tick_handle_periodic_broadcast+0x41/0x50 [] timer_interrupt+0x15/0x20 [] handle_irq_event_percpu+0x3e/0x1e0 [] handle_irq_event+0x3d/0x60 [] handle_edge_irq+0x77/0x130 [] handle_irq+0xbf/0x150 [] ? irq_enter+0x17/0xa0 [] do_IRQ+0x4f/0xf0 [] common_interrupt+0x6d/0x6d [] ? selinux_inode_alloc_security+0x59/0xa0 [] ? __d_instantiate+0xbf/0x100 [] ? __d_instantiate+0x9f/0x100 [] d_instantiate+0x3d/0x70 [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130 [] __create_file+0x1c2/0x2c0 [] ? set_graph_function+0x1f/0x1f [] debugfs_create_dir+0x1b/0x20 [] tracing_init_dentry_tr+0x7e/0x90 [] tracing_init_dentry+0x10/0x20 [] ftrace_init_debugfs+0x13/0x1fd [] ? set_graph_function+0x1f/0x1f [] do_one_initcall+0xb8/0x230 [] kernel_init_freeable+0x18b/0x22a [] ? initcall_blacklist+0xb0/0xb0 [] ? rest_init+0x80/0x80 [] kernel_init+0xe/0xf0 [] ret_from_fork+0x7c/0xb0 [] ? rest_init+0x80/0x80 ---[ end trace d5caa1cab8e7e98d ]--- At the same time sometimes there is one or more of spurious NMIs on the _host_ system: NMI ISA c, EISA ff NMI ... going to debugger bhyve seems to spin here: vmm.ko`svm_vmrun+0x894 vmm.ko`vm_run+0xbb7 vmm.ko`vmmdev_ioctl+0x5a4 kernel`devfs_ioctl_f+0x13b kernel`kern_ioctl+0x1e1 kernel`sys_ioctl+0x16a kernel`amd64_syscall+0x3ca kernel`0xffffffff8088997b (kgdb) list *svm_vmrun+0x894 0xffffffff813c9194 is in svm_vmrun (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895). 1890 1891 static __inline void 1892 enable_gintr(void) 1893 { 1894 1895 __asm __volatile("stgi"); 1896 } 1897 1898 /* 1899 * Start vcpu with specified RIP. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 14:23:26 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A17C9B36 for ; Mon, 22 Jun 2015 14:23:26 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id 60097930 for ; Mon, 22 Jun 2015 14:23:25 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id 8DCBF1241E for ; Tue, 23 Jun 2015 00:23:17 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 88E59281FDF for ; Tue, 23 Jun 2015 00:23:17 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id yyWpd8lzOtM7 for ; Tue, 23 Jun 2015 00:23:17 +1000 (AEST) Received: from Peters-MacBook-Pro.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 958862804FA; Tue, 23 Jun 2015 00:23:14 +1000 (AEST) Message-ID: <55881A50.6000608@freebsd.org> Date: Mon, 22 Jun 2015 07:23:12 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Andriy Gapon CC: freebsd-virtualization@freebsd.org Subject: Re: header type for bhyve host-pci bridge References: <5587D4AE.3050508@FreeBSD.org> In-Reply-To: <5587D4AE.3050508@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 14:23:26 -0000 Hi Andriy, > diff --git a/usr.sbin/bhyve/pci_hostbridge.c b/usr.sbin/bhyve/pci_hostbridge.c > index 54a25ae..5c9ea28 100644 > --- a/usr.sbin/bhyve/pci_hostbridge.c > +++ b/usr.sbin/bhyve/pci_hostbridge.c > @@ -38,7 +38,7 @@ pci_hostbridge_init(struct vmctx *ctx, struct pci_devinst *pi, > char *opts) > /* config space */ > pci_set_cfgdata16(pi, PCIR_VENDOR, 0x1275); /* NetApp */ > pci_set_cfgdata16(pi, PCIR_DEVICE, 0x1275); /* NetApp */ > - pci_set_cfgdata8(pi, PCIR_HDRTYPE, PCIM_HDRTYPE_BRIDGE); > + pci_set_cfgdata8(pi, PCIR_HDRTYPE, PCIM_HDRTYPE_NORMAL); > pci_set_cfgdata8(pi, PCIR_CLASS, PCIC_BRIDGE); > pci_set_cfgdata8(pi, PCIR_SUBCLASS, PCIS_BRIDGE_HOST); This change is already in CURRENT as of r283264 and will be MFCd soon. later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 14:41:18 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1B302D81 for ; Mon, 22 Jun 2015 14:41:18 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id CE1D31E0 for ; Mon, 22 Jun 2015 14:41:17 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id 15AE81269E for ; Tue, 23 Jun 2015 00:41:17 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 09D3F281FDE for ; Tue, 23 Jun 2015 00:41:17 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Bq68T6qhvmMy for ; Tue, 23 Jun 2015 00:41:16 +1000 (AEST) Received: from Peters-MacBook-Pro.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 50FA5281FDF; Tue, 23 Jun 2015 00:41:15 +1000 (AEST) Message-ID: <55881E89.9020204@freebsd.org> Date: Mon, 22 Jun 2015 07:41:13 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Andriy Gapon CC: "freebsd-virtualization@freebsd.org" Subject: Re: bhyve: centos 7.1 with multiple virtual processors References: <5587EE05.2020001@FreeBSD.org> In-Reply-To: <5587EE05.2020001@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 14:41:18 -0000 Hi Andriy, > If I run a CentOS 7.1 VM with more than one CPU more often than not it would > hang on startup and bhyve would start spinning. Looks like an AMD host - what's the version of FreeBSD you are running there ? later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 14:46:55 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A565ADDF for ; Mon, 22 Jun 2015 14:46:55 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 877B7682; Mon, 22 Jun 2015 14:46:53 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA28472; Mon, 22 Jun 2015 17:46:51 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z72zr-0005Ew-Gx; Mon, 22 Jun 2015 17:46:51 +0300 Message-ID: <55881FA1.5080109@FreeBSD.org> Date: Mon, 22 Jun 2015 17:45:53 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Peter Grehan CC: "freebsd-virtualization@freebsd.org" Subject: Re: bhyve: centos 7.1 with multiple virtual processors References: <5587EE05.2020001@FreeBSD.org> <55881E89.9020204@freebsd.org> In-Reply-To: <55881E89.9020204@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 14:46:55 -0000 On 22/06/2015 17:41, Peter Grehan wrote: > Hi Andriy, > >> If I run a CentOS 7.1 VM with more than one CPU more often than not it would >> hang on startup and bhyve would start spinning. > > Looks like an AMD host - what's the version of FreeBSD you are running there ? Yes, this is an AMD host, the version is head@283188. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 16:03:04 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 848842C5 for ; Mon, 22 Jun 2015 16:03:04 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id BC5B3D14 for ; Mon, 22 Jun 2015 16:03:03 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA29361 for ; Mon, 22 Jun 2015 19:03:01 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z74BY-0005Iu-Tt for freebsd-virtualization@freebsd.org; Mon, 22 Jun 2015 19:03:00 +0300 Message-ID: <5588317C.8020203@FreeBSD.org> Date: Mon, 22 Jun 2015 19:02:04 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "freebsd-virtualization@freebsd.org" Subject: dtrace fbt vs vmm Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 16:03:04 -0000 It seems that vmm contains functions for which it is not safe to enable fbt probes. At least, if I activate all fbt return probes with e.g. fbt:::return /self->trace && (int)arg1 == 28/ { printf("ret = %#x\n", arg0); stack(); (this is a part of larger script) while having an active bhyve VM, then I instantly get what looks like a possible triple-fault. I haven't investigated this issue further, but if there are any unsafe functions in vmm it would be nice to blacklist them in the fbt code. Perhaps blacklisting everything from vmm module could be a stopgap solution. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 16:08:24 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5CE9432B for ; Mon, 22 Jun 2015 16:08:24 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id 1C106E9E for ; Mon, 22 Jun 2015 16:08:23 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id BF34012670 for ; Tue, 23 Jun 2015 02:08:21 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id B93F4281FDE for ; Tue, 23 Jun 2015 02:08:21 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id c91v9jB16_rB for ; Tue, 23 Jun 2015 02:08:21 +1000 (AEST) Received: from Peters-MacBook-Pro.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id B189F281FE8; Tue, 23 Jun 2015 02:08:11 +1000 (AEST) Message-ID: <558832E8.9080407@freebsd.org> Date: Mon, 22 Jun 2015 09:08:08 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Andriy Gapon CC: "freebsd-virtualization@freebsd.org" Subject: Re: dtrace fbt vs vmm References: <5588317C.8020203@FreeBSD.org> In-Reply-To: <5588317C.8020203@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 16:08:24 -0000 Hi Andriy, > I haven't investigated this issue further, but if there are any unsafe functions > in vmm it would be nice to blacklist them in the fbt code. > Perhaps blacklisting everything from vmm module could be a stopgap solution. Intel or AMD ? We've been able to run dtrace on Intel; less (to no) coverage on AMD so there's quite possibly bugs there. later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 16:18:38 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3CDD7499 for ; Mon, 22 Jun 2015 16:18:38 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 39FD4855; Mon, 22 Jun 2015 16:18:36 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA29546; Mon, 22 Jun 2015 19:18:35 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z74Qd-0005K5-89; Mon, 22 Jun 2015 19:18:35 +0300 Message-ID: <55883537.3030400@FreeBSD.org> Date: Mon, 22 Jun 2015 19:17:59 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Peter Grehan CC: "freebsd-virtualization@freebsd.org" Subject: Re: dtrace fbt vs vmm References: <5588317C.8020203@FreeBSD.org> <558832E8.9080407@freebsd.org> In-Reply-To: <558832E8.9080407@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 16:18:38 -0000 On 22/06/2015 19:08, Peter Grehan wrote: > Hi Andriy, > >> I haven't investigated this issue further, but if there are any unsafe functions >> in vmm it would be nice to blacklist them in the fbt code. >> Perhaps blacklisting everything from vmm module could be a stopgap solution. > > Intel or AMD ? We've been able to run dtrace on Intel; less (to no) coverage on > AMD so there's quite possibly bugs there. Peter, it's AMD again. Thanks! -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 17:49:41 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DE6F2A6E for ; Mon, 22 Jun 2015 17:49:41 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 0431BB6C for ; Mon, 22 Jun 2015 17:49:40 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id UAA00673 for ; Mon, 22 Jun 2015 20:49:38 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z75qk-0005Oi-Ch for freebsd-virtualization@freebsd.org; Mon, 22 Jun 2015 20:49:38 +0300 Message-ID: <55884A61.4080106@FreeBSD.org> Date: Mon, 22 Jun 2015 20:48:17 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "freebsd-virtualization@freebsd.org" Subject: bhyve -> grub-bhyve without intermediate destroy Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 17:49:42 -0000 This is what happens if I create and run a VM by using grub-bhyve and bhyve, then exit from the VM via shutdown -r within it, and then try to run it again by using grub-bhyve and bhyve: Assertion failed: (error == 0), function fbsdrun_addcpu, file /usr/src/usr.sbin/bhyve/bhyverun.c, line 261 grub-bhyve apparently succeeds, but bhyve can't start up. Both invocation of bhyve are with "-c 1". -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 19:08:41 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3AA46134 for ; Mon, 22 Jun 2015 19:08:41 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id ED920EC5 for ; Mon, 22 Jun 2015 19:08:40 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id 5862B126ED for ; Tue, 23 Jun 2015 05:08:39 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 48ED6281FDF for ; Tue, 23 Jun 2015 05:08:39 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2ikIASz_g9Ng for ; Tue, 23 Jun 2015 05:08:39 +1000 (AEST) Received: from Peters-MacBook-Pro.local (unknown [64.245.0.210]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 8C8FD281FDE; Tue, 23 Jun 2015 05:08:36 +1000 (AEST) Message-ID: <55885D32.7040003@freebsd.org> Date: Mon, 22 Jun 2015 12:08:34 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Andriy Gapon CC: "freebsd-virtualization@freebsd.org" Subject: Re: bhyve -> grub-bhyve without intermediate destroy References: <55884A61.4080106@FreeBSD.org> In-Reply-To: <55884A61.4080106@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 19:08:41 -0000 Hi Andriy, > This is what happens if I create and run a VM by using grub-bhyve and bhyve, > then exit from the VM via shutdown -r within it, and then try to run it again by > using grub-bhyve and bhyve: > > Assertion failed: (error == 0), function fbsdrun_addcpu, file > /usr/src/usr.sbin/bhyve/bhyverun.c, line 261 > > grub-bhyve apparently succeeds, but bhyve can't start up. > Both invocation of bhyve are with "-c 1". This was fixed in grub2-bhyve upstream with https://github.com/grehan-freebsd/grub2-bhyve/commit/370fa455d41212282bf63cea7b048e87a821a31a I'm gathering a few more fixes before doing a point release of grub2-bhyve, at which point the FreeBSD port will be updated. You'll need to do a 'bhyvectl --destroy' until then, or build grub2-bhyve from upstream. later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 20:00:45 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5C5EF397 for ; Mon, 22 Jun 2015 20:00:45 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 5672779F; Mon, 22 Jun 2015 20:00:43 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id XAA02364; Mon, 22 Jun 2015 23:00:42 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z77tZ-0005W4-Vq; Mon, 22 Jun 2015 23:00:42 +0300 Message-ID: <55886931.1090600@FreeBSD.org> Date: Mon, 22 Jun 2015 22:59:45 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Peter Grehan CC: "freebsd-virtualization@freebsd.org" Subject: Re: bhyve -> grub-bhyve without intermediate destroy References: <55884A61.4080106@FreeBSD.org> <55885D32.7040003@freebsd.org> In-Reply-To: <55885D32.7040003@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 20:00:45 -0000 On 22/06/2015 22:08, Peter Grehan wrote: > Hi Andriy, > >> This is what happens if I create and run a VM by using grub-bhyve and bhyve, >> then exit from the VM via shutdown -r within it, and then try to run it again by >> using grub-bhyve and bhyve: >> >> Assertion failed: (error == 0), function fbsdrun_addcpu, file >> /usr/src/usr.sbin/bhyve/bhyverun.c, line 261 >> >> grub-bhyve apparently succeeds, but bhyve can't start up. >> Both invocation of bhyve are with "-c 1". > > This was fixed in grub2-bhyve upstream with > > https://github.com/grehan-freebsd/grub2-bhyve/commit/370fa455d41212282bf63cea7b048e87a821a31a > > > I'm gathering a few more fixes before doing a point release of grub2-bhyve, at > which point the FreeBSD port will be updated. > > You'll need to do a 'bhyvectl --destroy' until then, or build grub2-bhyve from > upstream. Thank you! Will the release that you are planning include the ext4 64-bit patch? -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 23 02:24:06 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 28725C5D for ; Tue, 23 Jun 2015 02:24:06 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wg0-x22a.google.com (mail-wg0-x22a.google.com [IPv6:2a00:1450:400c:c00::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BA2327CA; Tue, 23 Jun 2015 02:24:05 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wgck11 with SMTP id k11so28632558wgc.0; Mon, 22 Jun 2015 19:24:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=DtjOAGTMuta3yuS52rI/OYzuNKAeNgv1fSFPthhlxBA=; b=xaP+CzwziciSySuXzFE+Uehyz2zWTOi99K+P83Urt8ORbGeO3kKm+fna7ns6unTPzQ JaoQrUIcAohqD/uh0LVQV4kXzBbknpgU2SAhIp9f1AC+RPIUH3pO/Xq6cInBEP6ymicm aJK73xVqrwUyb8BfMn6FFf0MA6fOjzGMjyeXRgrt4u/8zYSwLLtNYVPbhylDm/5ZyB+h LAdgrHmVWvwNBbG8CnGRPPWzeAi7P3g31HQKpPxX7BFz5vpuJpUxVkfyNPBReNvXZCro P42EtRTzqjogMUVf54ZcvuGl7e30IeevyZwsdouK7XEKco7wpg+12xkoF5iV88AmQfgU u2lQ== MIME-Version: 1.0 X-Received: by 10.194.179.10 with SMTP id dc10mr45032583wjc.63.1435026242952; Mon, 22 Jun 2015 19:24:02 -0700 (PDT) Received: by 10.27.52.18 with HTTP; Mon, 22 Jun 2015 19:24:02 -0700 (PDT) In-Reply-To: <55883537.3030400@FreeBSD.org> References: <5588317C.8020203@FreeBSD.org> <558832E8.9080407@freebsd.org> <55883537.3030400@FreeBSD.org> Date: Mon, 22 Jun 2015 19:24:02 -0700 Message-ID: Subject: Re: dtrace fbt vs vmm From: Neel Natu To: Andriy Gapon Cc: Peter Grehan , "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 02:24:06 -0000 Hi Andriy, On Mon, Jun 22, 2015 at 9:17 AM, Andriy Gapon wrote: > On 22/06/2015 19:08, Peter Grehan wrote: >> Hi Andriy, >> >>> I haven't investigated this issue further, but if there are any unsafe functions >>> in vmm it would be nice to blacklist them in the fbt code. >>> Perhaps blacklisting everything from vmm module could be a stopgap solution. >> >> Intel or AMD ? We've been able to run dtrace on Intel; less (to no) coverage on >> AMD so there's quite possibly bugs there. > > Peter, > > it's AMD again. > Thanks! > This was an issue with 'svm_launch()' FBT tracing and is fixed in r284712. Details are in the commit message: https://svnweb.freebsd.org/base?view=revision&revision=284712 best Neel > -- > Andriy Gapon > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 23 02:37:10 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DE412CA9 for ; Tue, 23 Jun 2015 02:37:10 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-qc0-x22e.google.com (mail-qc0-x22e.google.com [IPv6:2607:f8b0:400d:c01::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 98E4EAEC; Tue, 23 Jun 2015 02:37:10 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by qcji3 with SMTP id i3so12716114qcj.1; Mon, 22 Jun 2015 19:37:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=FlMx9rQTmZ2UmpgvE37cj0jwT+BOxn9tjJo4AwcYAXk=; b=pgoT5vuz5Ndob40XJjC4hfcREHt+SXXa5tcxDUsuHqmYkrRTgds5XGIm9oNaYkWgNH WbpJlbVdJaDSplOBQnKGhxj5ONpvjiWV3IhWwdru06VPE9wcahdE7d6gjQseP1D/pK6b 8dbi1/kOLzMsSfvdydTukATpF8HpONf71Xv+QDjHx6mXDSftS6vLXyHGbaaLn+bXFnbm LFOV3CHxEaSBgO16cFtgDni56zAvHEtbzvSpgBtWRGroC1uVpukGUAIpDqnwEq5mhhNV wlwS9lS4Ms+T23BRbF9On1YmpYxLJuQR20Bb+DnCfAqjiTnxTC4IN2L+exYkTqaWe5pV cwDQ== MIME-Version: 1.0 X-Received: by 10.55.19.25 with SMTP id d25mr68593063qkh.19.1435027029665; Mon, 22 Jun 2015 19:37:09 -0700 (PDT) Received: by 10.140.40.200 with HTTP; Mon, 22 Jun 2015 19:37:09 -0700 (PDT) In-Reply-To: <5587EE05.2020001@FreeBSD.org> References: <5587EE05.2020001@FreeBSD.org> Date: Mon, 22 Jun 2015 19:37:09 -0700 Message-ID: Subject: Re: bhyve: centos 7.1 with multiple virtual processors From: Neel Natu To: Andriy Gapon Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 02:37:11 -0000 Hi Andriy, FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus fine on my host with 8 physical cores. I have some questions about your setup inline. On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon wrote: > > If I run a CentOS 7.1 VM with more than one CPU more often than not it would > hang on startup and bhyve would start spinning. > > The following are the last messages seen in the VM: > > Switching to clocksource hpet > ------------[ cut here ]------------ > WARNING: at kernel/time/clockevents.c:239 clockevents_program_event+0xdb/0xf0() > Modules linked in: > CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1 > Hardware name: BHYVE, BIOS 1.00 03/14/2014 > 0000000000000000 00000000cab5bdb6 ffff88003fc03e08 ffffffff81604eaa > ffff88003fc03e40 ffffffff8106e34b 80000000000f423f 80000000000f423f > ffffffff81915440 0000000000000000 0000000000000000 ffff88003fc03e50 > Call Trace: > [] dump_stack+0x19/0x1b > [] warn_slowpath_common+0x6b/0xb0 > [] warn_slowpath_null+0x1a/0x20 > [] clockevents_program_event+0xdb/0xf0 > [] tick_handle_periodic_broadcast+0x41/0x50 > [] timer_interrupt+0x15/0x20 > [] handle_irq_event_percpu+0x3e/0x1e0 > [] handle_irq_event+0x3d/0x60 > [] handle_edge_irq+0x77/0x130 > [] handle_irq+0xbf/0x150 > [] ? irq_enter+0x17/0xa0 > [] do_IRQ+0x4f/0xf0 > [] common_interrupt+0x6d/0x6d > [] ? selinux_inode_alloc_security+0x59/0xa0 > [] ? __d_instantiate+0xbf/0x100 > [] ? __d_instantiate+0x9f/0x100 > [] d_instantiate+0x3d/0x70 > [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130 > [] __create_file+0x1c2/0x2c0 > [] ? set_graph_function+0x1f/0x1f > [] debugfs_create_dir+0x1b/0x20 > [] tracing_init_dentry_tr+0x7e/0x90 > [] tracing_init_dentry+0x10/0x20 > [] ftrace_init_debugfs+0x13/0x1fd > [] ? set_graph_function+0x1f/0x1f > [] do_one_initcall+0xb8/0x230 > [] kernel_init_freeable+0x18b/0x22a > [] ? initcall_blacklist+0xb0/0xb0 > [] ? rest_init+0x80/0x80 > [] kernel_init+0xe/0xf0 > [] ret_from_fork+0x7c/0xb0 > [] ? rest_init+0x80/0x80 > ---[ end trace d5caa1cab8e7e98d ]--- > A few questions to narrow this down: - Is the host very busy when the VM is started (or what is the host doing when this happened)? - How many vcpus are you giving to the VM? - How many cores on the host? > > At the same time sometimes there is one or more of spurious NMIs on the _host_ > system: > NMI ISA c, EISA ff > NMI ... going to debugger > Hmm, that's interesting. Are you using hwpmc to do instruction sampling? > bhyve seems to spin here: > vmm.ko`svm_vmrun+0x894 > vmm.ko`vm_run+0xbb7 > vmm.ko`vmmdev_ioctl+0x5a4 > kernel`devfs_ioctl_f+0x13b > kernel`kern_ioctl+0x1e1 > kernel`sys_ioctl+0x16a > kernel`amd64_syscall+0x3ca > kernel`0xffffffff8088997b > > (kgdb) list *svm_vmrun+0x894 > 0xffffffff813c9194 is in svm_vmrun > (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895). > 1890 > 1891 static __inline void > 1892 enable_gintr(void) > 1893 { > 1894 > 1895 __asm __volatile("stgi"); > 1896 } > 1897 > 1898 /* > 1899 * Start vcpu with specified RIP. > Yeah, that's not surprising because host interrupts are blocked when the cpu is executing in guest context. The 'enable_gintr()' enables interrupts so it gets blamed by the interrupt-based sampling. In this case it just means that the cpu was in guest context when a host-interrupt fired. best Neel > -- > Andriy Gapon > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 23 06:47:05 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9023EB94 for ; Tue, 23 Jun 2015 06:47:05 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id CE7DFAE6 for ; Tue, 23 Jun 2015 06:47:04 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id JAA11222; Tue, 23 Jun 2015 09:46:56 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z7Hyy-0006Ax-Cu; Tue, 23 Jun 2015 09:46:56 +0300 Message-ID: <558900A7.40609@FreeBSD.org> Date: Tue, 23 Jun 2015 09:45:59 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Neel Natu CC: "freebsd-virtualization@freebsd.org" Subject: Re: bhyve: centos 7.1 with multiple virtual processors References: <5587EE05.2020001@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 06:47:05 -0000 On 23/06/2015 05:37, Neel Natu wrote: > Hi Andriy, > > FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus > fine on my host with 8 physical cores. > > I have some questions about your setup inline. > > On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon wrote: >> >> If I run a CentOS 7.1 VM with more than one CPU more often than not it would >> hang on startup and bhyve would start spinning. >> >> The following are the last messages seen in the VM: >> >> Switching to clocksource hpet >> ------------[ cut here ]------------ >> WARNING: at kernel/time/clockevents.c:239 clockevents_program_event+0xdb/0xf0() >> Modules linked in: >> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1 >> Hardware name: BHYVE, BIOS 1.00 03/14/2014 >> 0000000000000000 00000000cab5bdb6 ffff88003fc03e08 ffffffff81604eaa >> ffff88003fc03e40 ffffffff8106e34b 80000000000f423f 80000000000f423f >> ffffffff81915440 0000000000000000 0000000000000000 ffff88003fc03e50 >> Call Trace: >> [] dump_stack+0x19/0x1b >> [] warn_slowpath_common+0x6b/0xb0 >> [] warn_slowpath_null+0x1a/0x20 >> [] clockevents_program_event+0xdb/0xf0 >> [] tick_handle_periodic_broadcast+0x41/0x50 >> [] timer_interrupt+0x15/0x20 >> [] handle_irq_event_percpu+0x3e/0x1e0 >> [] handle_irq_event+0x3d/0x60 >> [] handle_edge_irq+0x77/0x130 >> [] handle_irq+0xbf/0x150 >> [] ? irq_enter+0x17/0xa0 >> [] do_IRQ+0x4f/0xf0 >> [] common_interrupt+0x6d/0x6d >> [] ? selinux_inode_alloc_security+0x59/0xa0 >> [] ? __d_instantiate+0xbf/0x100 >> [] ? __d_instantiate+0x9f/0x100 >> [] d_instantiate+0x3d/0x70 >> [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130 >> [] __create_file+0x1c2/0x2c0 >> [] ? set_graph_function+0x1f/0x1f >> [] debugfs_create_dir+0x1b/0x20 >> [] tracing_init_dentry_tr+0x7e/0x90 >> [] tracing_init_dentry+0x10/0x20 >> [] ftrace_init_debugfs+0x13/0x1fd >> [] ? set_graph_function+0x1f/0x1f >> [] do_one_initcall+0xb8/0x230 >> [] kernel_init_freeable+0x18b/0x22a >> [] ? initcall_blacklist+0xb0/0xb0 >> [] ? rest_init+0x80/0x80 >> [] kernel_init+0xe/0xf0 >> [] ret_from_fork+0x7c/0xb0 >> [] ? rest_init+0x80/0x80 >> ---[ end trace d5caa1cab8e7e98d ]--- >> > > A few questions to narrow this down: > - Is the host very busy when the VM is started (or what is the host > doing when this happened)? The host typically is not heavily loaded. There is X server running and some applications. I'd imagine that those could cause some additional latency but not CPU starvation. > - How many vcpus are you giving to the VM? > - How many cores on the host? I tried only 2 / 2. >> >> At the same time sometimes there is one or more of spurious NMIs on the _host_ >> system: >> NMI ISA c, EISA ff >> NMI ... going to debugger >> > > Hmm, that's interesting. Are you using hwpmc to do instruction sampling? hwpmc driver is in the kernel, but it was not used. >> bhyve seems to spin here: >> vmm.ko`svm_vmrun+0x894 >> vmm.ko`vm_run+0xbb7 >> vmm.ko`vmmdev_ioctl+0x5a4 >> kernel`devfs_ioctl_f+0x13b >> kernel`kern_ioctl+0x1e1 >> kernel`sys_ioctl+0x16a >> kernel`amd64_syscall+0x3ca >> kernel`0xffffffff8088997b >> >> (kgdb) list *svm_vmrun+0x894 >> 0xffffffff813c9194 is in svm_vmrun >> (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895). >> 1890 >> 1891 static __inline void >> 1892 enable_gintr(void) >> 1893 { >> 1894 >> 1895 __asm __volatile("stgi"); >> 1896 } >> 1897 >> 1898 /* >> 1899 * Start vcpu with specified RIP. >> > > Yeah, that's not surprising because host interrupts are blocked when > the cpu is executing in guest context. The 'enable_gintr()' enables > interrupts so it gets blamed by the interrupt-based sampling. > > In this case it just means that the cpu was in guest context when a > host-interrupt fired. I see. FWIW, that was captured with DTrace. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 23 06:47:47 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id ED43EBBD for ; Tue, 23 Jun 2015 06:47:47 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id E8FE8B4E; Tue, 23 Jun 2015 06:47:46 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id JAA11230; Tue, 23 Jun 2015 09:47:45 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z7Hzk-0006B0-Sl; Tue, 23 Jun 2015 09:47:44 +0300 Message-ID: <558900ED.8060804@FreeBSD.org> Date: Tue, 23 Jun 2015 09:47:09 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Neel Natu CC: Peter Grehan , "freebsd-virtualization@freebsd.org" Subject: Re: dtrace fbt vs vmm References: <5588317C.8020203@FreeBSD.org> <558832E8.9080407@freebsd.org> <55883537.3030400@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 06:47:48 -0000 On 23/06/2015 05:24, Neel Natu wrote: > Hi Andriy, > > On Mon, Jun 22, 2015 at 9:17 AM, Andriy Gapon wrote: >> On 22/06/2015 19:08, Peter Grehan wrote: >>> Hi Andriy, >>> >>>> I haven't investigated this issue further, but if there are any unsafe functions >>>> in vmm it would be nice to blacklist them in the fbt code. >>>> Perhaps blacklisting everything from vmm module could be a stopgap solution. >>> >>> Intel or AMD ? We've been able to run dtrace on Intel; less (to no) coverage on >>> AMD so there's quite possibly bugs there. >> >> Peter, >> >> it's AMD again. >> Thanks! >> > > This was an issue with 'svm_launch()' FBT tracing and is fixed in r284712. > > Details are in the commit message: > https://svnweb.freebsd.org/base?view=revision&revision=284712 Thank you! -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 23 07:26:44 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BDDA0E96 for ; Tue, 23 Jun 2015 07:26:44 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wg0-x22a.google.com (mail-wg0-x22a.google.com [IPv6:2a00:1450:400c:c00::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 76C4B2BD; Tue, 23 Jun 2015 07:26:44 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wgbhy7 with SMTP id hy7so1426127wgb.2; Tue, 23 Jun 2015 00:26:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=pNWGC1SdjAjybulzCKHAuzt1CXTftPr+dRGN0dKZzF0=; b=MITG+k0WvwoNjTVkSRPEm8b9r0UDhHhWFV+4j4G+Gt1Qmefay6U1pBZZ8UhHc70em+ dMVx+H2Gp0ofM0WZbdDJ0+ptw8H1FLbafkmJuilmRt0qeBas4roKLmRgBjYfcpRDb1T/ OCqchk+cDoKRjKRoEXJjbgItex9MO+WA+rgK19INIj580PkizLiWPxCCXt22ePIBUxEx UPLjn5k6iBScNXzHLpuzXYOJF7oGdZDo751JJrUibdmy6VLgjdGHfaUIpeJ/K3xvKHNU YCmShr9Baikg9O2XC7IABUZ4Frtu2ICq4rL/j0ypgknHQgV+yyQEyLOAJlzw40Wc0pDv 59Zw== MIME-Version: 1.0 X-Received: by 10.194.192.166 with SMTP id hh6mr57287450wjc.127.1435044402811; Tue, 23 Jun 2015 00:26:42 -0700 (PDT) Received: by 10.27.52.79 with HTTP; Tue, 23 Jun 2015 00:26:42 -0700 (PDT) In-Reply-To: <558900A7.40609@FreeBSD.org> References: <5587EE05.2020001@FreeBSD.org> <558900A7.40609@FreeBSD.org> Date: Tue, 23 Jun 2015 00:26:42 -0700 Message-ID: Subject: Re: bhyve: centos 7.1 with multiple virtual processors From: Neel Natu To: Andriy Gapon Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 07:26:44 -0000 Hi Andriy, On Mon, Jun 22, 2015 at 11:45 PM, Andriy Gapon wrote: > On 23/06/2015 05:37, Neel Natu wrote: >> Hi Andriy, >> >> FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus >> fine on my host with 8 physical cores. >> >> I have some questions about your setup inline. >> >> On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon wrote: >>> >>> If I run a CentOS 7.1 VM with more than one CPU more often than not it would >>> hang on startup and bhyve would start spinning. >>> >>> The following are the last messages seen in the VM: >>> >>> Switching to clocksource hpet >>> ------------[ cut here ]------------ >>> WARNING: at kernel/time/clockevents.c:239 clockevents_program_event+0xdb/0xf0() >>> Modules linked in: >>> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1 >>> Hardware name: BHYVE, BIOS 1.00 03/14/2014 >>> 0000000000000000 00000000cab5bdb6 ffff88003fc03e08 ffffffff81604eaa >>> ffff88003fc03e40 ffffffff8106e34b 80000000000f423f 80000000000f423f >>> ffffffff81915440 0000000000000000 0000000000000000 ffff88003fc03e50 >>> Call Trace: >>> [] dump_stack+0x19/0x1b >>> [] warn_slowpath_common+0x6b/0xb0 >>> [] warn_slowpath_null+0x1a/0x20 >>> [] clockevents_program_event+0xdb/0xf0 >>> [] tick_handle_periodic_broadcast+0x41/0x50 >>> [] timer_interrupt+0x15/0x20 >>> [] handle_irq_event_percpu+0x3e/0x1e0 >>> [] handle_irq_event+0x3d/0x60 >>> [] handle_edge_irq+0x77/0x130 >>> [] handle_irq+0xbf/0x150 >>> [] ? irq_enter+0x17/0xa0 >>> [] do_IRQ+0x4f/0xf0 >>> [] common_interrupt+0x6d/0x6d >>> [] ? selinux_inode_alloc_security+0x59/0xa0 >>> [] ? __d_instantiate+0xbf/0x100 >>> [] ? __d_instantiate+0x9f/0x100 >>> [] d_instantiate+0x3d/0x70 >>> [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130 >>> [] __create_file+0x1c2/0x2c0 >>> [] ? set_graph_function+0x1f/0x1f >>> [] debugfs_create_dir+0x1b/0x20 >>> [] tracing_init_dentry_tr+0x7e/0x90 >>> [] tracing_init_dentry+0x10/0x20 >>> [] ftrace_init_debugfs+0x13/0x1fd >>> [] ? set_graph_function+0x1f/0x1f >>> [] do_one_initcall+0xb8/0x230 >>> [] kernel_init_freeable+0x18b/0x22a >>> [] ? initcall_blacklist+0xb0/0xb0 >>> [] ? rest_init+0x80/0x80 >>> [] kernel_init+0xe/0xf0 >>> [] ret_from_fork+0x7c/0xb0 >>> [] ? rest_init+0x80/0x80 >>> ---[ end trace d5caa1cab8e7e98d ]--- >>> >> >> A few questions to narrow this down: >> - Is the host very busy when the VM is started (or what is the host >> doing when this happened)? > > The host typically is not heavily loaded. There is X server running and some > applications. I'd imagine that those could cause some additional latency but > not CPU starvation. > Yup, I agree. Does this ever happen with a single vcpu guest? The other mystery is the NMIs the host is receiving. I (re)verified to make sure that bhyve/vmm.ko do not assert NMIs so it has to be something else on the host that's doing it ... best Neel >> - How many vcpus are you giving to the VM? >> - How many cores on the host? > > I tried only 2 / 2. > >>> >>> At the same time sometimes there is one or more of spurious NMIs on the _host_ >>> system: >>> NMI ISA c, EISA ff >>> NMI ... going to debugger >>> >> >> Hmm, that's interesting. Are you using hwpmc to do instruction sampling? > > hwpmc driver is in the kernel, but it was not used. > >>> bhyve seems to spin here: >>> vmm.ko`svm_vmrun+0x894 >>> vmm.ko`vm_run+0xbb7 >>> vmm.ko`vmmdev_ioctl+0x5a4 >>> kernel`devfs_ioctl_f+0x13b >>> kernel`kern_ioctl+0x1e1 >>> kernel`sys_ioctl+0x16a >>> kernel`amd64_syscall+0x3ca >>> kernel`0xffffffff8088997b >>> >>> (kgdb) list *svm_vmrun+0x894 >>> 0xffffffff813c9194 is in svm_vmrun >>> (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895). >>> 1890 >>> 1891 static __inline void >>> 1892 enable_gintr(void) >>> 1893 { >>> 1894 >>> 1895 __asm __volatile("stgi"); >>> 1896 } >>> 1897 >>> 1898 /* >>> 1899 * Start vcpu with specified RIP. >>> >> >> Yeah, that's not surprising because host interrupts are blocked when >> the cpu is executing in guest context. The 'enable_gintr()' enables >> interrupts so it gets blamed by the interrupt-based sampling. >> >> In this case it just means that the cpu was in guest context when a >> host-interrupt fired. > > I see. FWIW, that was captured with DTrace. > > -- > Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 23 07:57:31 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B3A15BE for ; Tue, 23 Jun 2015 07:57:31 +0000 (UTC) (envelope-from xmj@freebsd.org) Received: from hub.freebsd.org (hub.freebsd.org [8.8.178.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hub.freebsd.org", Issuer "hub.freebsd.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 97610EF0 for ; Tue, 23 Jun 2015 07:57:31 +0000 (UTC) (envelope-from xmj@freebsd.org) Received: by hub.freebsd.org (Postfix) id 8D2A5BD; Tue, 23 Jun 2015 07:57:31 +0000 (UTC) Delivered-To: virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8C70DBC for ; Tue, 23 Jun 2015 07:57:31 +0000 (UTC) (envelope-from xmj@freebsd.org) Received: from dd16522.kasserver.com (dd16522.kasserver.com [85.13.137.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4F2C8EED; Tue, 23 Jun 2015 07:57:30 +0000 (UTC) (envelope-from xmj@freebsd.org) Received: from mx12.chaot.net (82.131.85.156.cable.starman.ee [82.131.85.156]) by dd16522.kasserver.com (Postfix) with ESMTPSA id AE9378D4002; Tue, 23 Jun 2015 09:57:21 +0200 (CEST) Received: from mx12.chaot.net (localhost [127.0.0.1]); by mx12.chaot.net (OpenSMTPD) with ESMTP id 5d6816a5; Tue, 23 Jun 2015 10:57:20 +0300 (EEST) Message-ID: <55891160.2030502@FreeBSD.org> Date: Tue, 23 Jun 2015 10:57:20 +0300 From: Johannes Jost Meixner User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Peter Grehan , "Leo R. Lundgren" CC: virtualization@FreeBSD.org Subject: Re: Windows support status References: <55881E08.5010906@freebsd.org> <19CB2CC1-FB10-42CF-A10B-DACF009CC65F@finalresort.org> <55885E1D.1010402@freebsd.org> In-Reply-To: <55885E1D.1010402@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 07:57:31 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Peter, Leo, On 06/22/2015 22:12, Peter Grehan wrote: > Hi Leo, > >> Forgive my ignorance, but when you talk about the UEFI build I >> don't suppose you mean that one must run the UEFI "version" of >> FreeBSD, right? I assume you mean that bhyve is presenting UEFI >> to the guest. > > That's right. We have a custom build of Intel UEFI firmware that > gets placed into guest ROM space. This is different than the > previous approach of having external user-space loaders > (bhyveload/grub-bhyve). Cool! Does that UEFI Firmware come with a way to running very old things, e.g. DOS 6.2.2, Windows 3.11 or Windows 95? > >> Will one be able to run this on 10.1-RELEASE, or must one use >> 11-CURRENT? > > It'll be on 11-CURRENT initially and will be backported to > 10-STABLE. I don't know think it's going to make the 10.2 cutoff > :( I remember asking you in Ottawa about it, but forgot your answer. What's the timeline for bringing it into 11.0-CURRENT? > > later, > > Peter. - -- Johannes Meixner | FreeBSD Committer xmj@FreeBSD.org | http://people.freebsd.org/~xmj -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJViRFZAAoJEPyeKTcbGw0LBgsH/ishP4gmsE1Mrbu9tmtsednz 84iYg8Nn4TfJ0bOR0kGjXl5CctcfNJR3eF1CJunJrnfgcdxVfghlXCs0Fu3wMs0p VrpqKQK+UHu88Fv9y7l0O1Ly+Acx6lr4lsXXUk2D9XpgivKwZw5rylxcOtu6YHlw 9d2l7KR062jMHiya0EKvtzqxS2e61I4JGTP2ygRgsQELo2bYcpVWt9BsTcNQdGvv hm9ynQqGk141x4kY+UJyag8UH9INfB7Fr+GfAYOIE2MQ/UgRWEEHW3v42T/djTuX EyQwRZllKYQlKp6p1vA2jSDW0eDlY2/BgZpZFHahghzn9pdUHE5cyMtV2hCz/Eg= =8Z0T -----END PGP SIGNATURE----- From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 23 08:06:08 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D791F1D8 for ; Tue, 23 Jun 2015 08:06:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 2674F616 for ; Tue, 23 Jun 2015 08:06:07 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id LAA12607; Tue, 23 Jun 2015 11:06:05 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z7JDZ-0006Fg-Ag; Tue, 23 Jun 2015 11:06:05 +0300 Message-ID: <55891335.5090403@FreeBSD.org> Date: Tue, 23 Jun 2015 11:05:09 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Neel Natu CC: "freebsd-virtualization@freebsd.org" Subject: Re: bhyve: centos 7.1 with multiple virtual processors References: <5587EE05.2020001@FreeBSD.org> <558900A7.40609@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 08:06:08 -0000 On 23/06/2015 10:26, Neel Natu wrote: > Hi Andriy, > > On Mon, Jun 22, 2015 at 11:45 PM, Andriy Gapon wrote: >> On 23/06/2015 05:37, Neel Natu wrote: >>> Hi Andriy, >>> >>> FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus >>> fine on my host with 8 physical cores. >>> >>> I have some questions about your setup inline. >>> >>> On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon wrote: >>>> >>>> If I run a CentOS 7.1 VM with more than one CPU more often than not it would >>>> hang on startup and bhyve would start spinning. >>>> >>>> The following are the last messages seen in the VM: >>>> >>>> Switching to clocksource hpet >>>> ------------[ cut here ]------------ >>>> WARNING: at kernel/time/clockevents.c:239 clockevents_program_event+0xdb/0xf0() >>>> Modules linked in: >>>> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1 >>>> Hardware name: BHYVE, BIOS 1.00 03/14/2014 >>>> 0000000000000000 00000000cab5bdb6 ffff88003fc03e08 ffffffff81604eaa >>>> ffff88003fc03e40 ffffffff8106e34b 80000000000f423f 80000000000f423f >>>> ffffffff81915440 0000000000000000 0000000000000000 ffff88003fc03e50 >>>> Call Trace: >>>> [] dump_stack+0x19/0x1b >>>> [] warn_slowpath_common+0x6b/0xb0 >>>> [] warn_slowpath_null+0x1a/0x20 >>>> [] clockevents_program_event+0xdb/0xf0 >>>> [] tick_handle_periodic_broadcast+0x41/0x50 >>>> [] timer_interrupt+0x15/0x20 >>>> [] handle_irq_event_percpu+0x3e/0x1e0 >>>> [] handle_irq_event+0x3d/0x60 >>>> [] handle_edge_irq+0x77/0x130 >>>> [] handle_irq+0xbf/0x150 >>>> [] ? irq_enter+0x17/0xa0 >>>> [] do_IRQ+0x4f/0xf0 >>>> [] common_interrupt+0x6d/0x6d >>>> [] ? selinux_inode_alloc_security+0x59/0xa0 >>>> [] ? __d_instantiate+0xbf/0x100 >>>> [] ? __d_instantiate+0x9f/0x100 >>>> [] d_instantiate+0x3d/0x70 >>>> [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130 >>>> [] __create_file+0x1c2/0x2c0 >>>> [] ? set_graph_function+0x1f/0x1f >>>> [] debugfs_create_dir+0x1b/0x20 >>>> [] tracing_init_dentry_tr+0x7e/0x90 >>>> [] tracing_init_dentry+0x10/0x20 >>>> [] ftrace_init_debugfs+0x13/0x1fd >>>> [] ? set_graph_function+0x1f/0x1f >>>> [] do_one_initcall+0xb8/0x230 >>>> [] kernel_init_freeable+0x18b/0x22a >>>> [] ? initcall_blacklist+0xb0/0xb0 >>>> [] ? rest_init+0x80/0x80 >>>> [] kernel_init+0xe/0xf0 >>>> [] ret_from_fork+0x7c/0xb0 >>>> [] ? rest_init+0x80/0x80 >>>> ---[ end trace d5caa1cab8e7e98d ]--- >>>> >>> >>> A few questions to narrow this down: >>> - Is the host very busy when the VM is started (or what is the host >>> doing when this happened)? >> >> The host typically is not heavily loaded. There is X server running and some >> applications. I'd imagine that those could cause some additional latency but >> not CPU starvation. >> > > Yup, I agree. > > Does this ever happen with a single vcpu guest? Never seen the problem with a single CPU so far. Also, never had that problem with FreeBSD guests. > The other mystery is the NMIs the host is receiving. I (re)verified to > make sure that bhyve/vmm.ko do not assert NMIs so it has to be > something else on the host that's doing it ... But the correlation with the multi-CPU non-FreeBSD guests seems to be significant. P.S. meanwhile I found this old-ish thread that seems to describe exactly the problem I am seeing but on real hardware: http://thread.gmane.org/gmane.linux.kernel/1483297 -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 23 17:25:00 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 768897D9 for ; Tue, 23 Jun 2015 17:25:00 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from hub.freebsd.org (hub.freebsd.org [8.8.178.136]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hub.freebsd.org", Issuer "hub.freebsd.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 539DFA2 for ; Tue, 23 Jun 2015 17:25:00 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: by hub.freebsd.org (Postfix) id 493C57D8; Tue, 23 Jun 2015 17:25:00 +0000 (UTC) Delivered-To: virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 489497D7 for ; Tue, 23 Jun 2015 17:25:00 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id 0642E9F for ; Tue, 23 Jun 2015 17:24:59 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id 550191272E for ; Wed, 24 Jun 2015 03:24:52 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 4C3802809D5 for ; Wed, 24 Jun 2015 03:24:52 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kJ1RPWhlfjog for ; Wed, 24 Jun 2015 03:24:52 +1000 (AEST) Received: from Peters-MacBook-Pro.local (c-50-184-135-69.hsd1.ca.comcast.net [50.184.135.69]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id C9DAA280901; Wed, 24 Jun 2015 03:24:47 +1000 (AEST) Message-ID: <5589965C.10503@freebsd.org> Date: Tue, 23 Jun 2015 10:24:44 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Johannes Jost Meixner , "Leo R. Lundgren" CC: virtualization@FreeBSD.org Subject: Re: Windows support status References: <55881E08.5010906@freebsd.org> <19CB2CC1-FB10-42CF-A10B-DACF009CC65F@finalresort.org> <55885E1D.1010402@freebsd.org> <55891160.2030502@FreeBSD.org> In-Reply-To: <55891160.2030502@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 17:25:00 -0000 > Does that UEFI Firmware come with a way to running very old > things, e.g. DOS 6.2.2, Windows 3.11 or Windows 95? Eventually :) > What's the timeline for bringing it into 11.0-CURRENT? Headless operation will be supported shortly. The native graphics work will take longer to trickle in. later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 23 18:15:53 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6B75BA5F for ; Tue, 23 Jun 2015 18:15:53 +0000 (UTC) (envelope-from xmj@freebsd.org) Received: from hub.freebsd.org (hub.freebsd.org [IPv6:2001:1900:2254:206c::16:88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hub.freebsd.org", Issuer "hub.freebsd.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4E327348 for ; Tue, 23 Jun 2015 18:15:53 +0000 (UTC) (envelope-from xmj@freebsd.org) Received: by hub.freebsd.org (Postfix) id 43B38A5E; Tue, 23 Jun 2015 18:15:53 +0000 (UTC) Delivered-To: virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 42F8EA5D for ; Tue, 23 Jun 2015 18:15:53 +0000 (UTC) (envelope-from xmj@freebsd.org) Received: from dd16522.kasserver.com (dd16522.kasserver.com [85.13.137.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 055F7346; Tue, 23 Jun 2015 18:15:52 +0000 (UTC) (envelope-from xmj@freebsd.org) Received: from mx12.chaot.net (82.131.85.156.cable.starman.ee [82.131.85.156]) by dd16522.kasserver.com (Postfix) with ESMTPSA id 661978D4002; Tue, 23 Jun 2015 20:15:49 +0200 (CEST) Received: from mx12.chaot.net (localhost [127.0.0.1]); by mx12.chaot.net (OpenSMTPD) with ESMTP id bdf087b9; Tue, 23 Jun 2015 21:15:48 +0300 (EEST) Message-ID: <5589A254.7000803@FreeBSD.org> Date: Tue, 23 Jun 2015 21:15:48 +0300 From: Johannes Jost Meixner User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Peter Grehan , "Leo R. Lundgren" CC: virtualization@FreeBSD.org Subject: Re: Windows support status References: <55881E08.5010906@freebsd.org> <19CB2CC1-FB10-42CF-A10B-DACF009CC65F@finalresort.org> <55885E1D.1010402@freebsd.org> <55891160.2030502@FreeBSD.org> <5589965C.10503@freebsd.org> In-Reply-To: <5589965C.10503@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 18:15:53 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 06/23/2015 20:24, Peter Grehan wrote: >> Does that UEFI Firmware come with a way to running very old >> things, e.g. DOS 6.2.2, Windows 3.11 or Windows 95? > > Eventually :) > >> What's the timeline for bringing it into 11.0-CURRENT? > > Headless operation will be supported shortly. The native graphics > work will take longer to trickle in. Awesome. Keep on rocking! - -Johannes - -- Johannes Meixner | FreeBSD Committer xmj@FreeBSD.org | http://people.freebsd.org/~xmj -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBCAAGBQJViaJQAAoJEPyeKTcbGw0Lp54H/37HEduDcV625xzBQG/TpkJ/ K9/SYuy1/5LLQMLiKMpdCkf3C1yZ5gNrM9PB/W9ZFCFd5tO+YmlEk7V7HRaTh+wI QKWeIVMBk0Tj8B60rO2aAnoI9L/MDvpp7WAKpjAeBvrk0RpRNM0n6RBDwJiMR/8i TrAaO1uN7eTTfMMwXNHcsDhguXCX/+Fq8zER5BBr2w8vgIRvHiyt0P0R79h/CqJ2 JuLohGwlBMrNL0de7/RCAkq0F+sbJKTYaH9gOW4HCSQ9BuEQJVM3Wgktpe5F3yjx rFhvg9UkDX1ZOGESSwjb/VldmZT7BHtV4U/KyhtGfaULD/yjQblFmZ/dXvN2ZZM= =PRqo -----END PGP SIGNATURE----- From owner-freebsd-virtualization@freebsd.org Wed Jun 24 22:49:18 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4560C916F84 for ; Wed, 24 Jun 2015 22:49:18 +0000 (UTC) (envelope-from javocado@gmail.com) Received: from mail-lb0-x230.google.com (mail-lb0-x230.google.com [IPv6:2a00:1450:4010:c04::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C5C0E2ED1 for ; Wed, 24 Jun 2015 22:49:17 +0000 (UTC) (envelope-from javocado@gmail.com) Received: by lbnk3 with SMTP id k3so35079505lbn.1 for ; Wed, 24 Jun 2015 15:49:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=xGZeEU9aeGY27KNmtfEh4WLKe3cig3vLV42nAskk2KY=; b=kzT2zdDnAIDw6qKni5JRg/cGHG38brWJx5Qh7hXwXrNOIN34h3gBBnc949NBbcFo6c Qrjharfnw7HjHrXJLPS7X5922AC2tku1+isaN1CeGJFidGF+GwvkXc8ZWLndjqn0kYri 4pXIkGxsaYyx72R18xCLhvyheVtzRGZYMFSxWTTP4Oi28BHvP2T9W/Ao/bLo3GIorCcQ kV0Vku7YFgERAxasRsBdQdZYemI6wN+X/2Ub4wvcmpZ2vHwkMM2XLFrR30k2Spkm4cow iXS5GrsaPsHDDObSZPed9QfeFguPiZlpxYE4IPgfnGdS95ZifJ58MdaAf6rP2TWF4Qsg UPng== MIME-Version: 1.0 X-Received: by 10.112.50.148 with SMTP id c20mr41528792lbo.27.1435186155804; Wed, 24 Jun 2015 15:49:15 -0700 (PDT) Received: by 10.114.96.8 with HTTP; Wed, 24 Jun 2015 15:49:15 -0700 (PDT) Date: Wed, 24 Jun 2015 15:49:15 -0700 Message-ID: Subject: ZFS pool within FreeBSD bhyve guest From: javocado To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jun 2015 22:49:18 -0000 Hi, I would like to create a zfs filesystem within my bhyve (FreeBSD 10.1 as the guest and host) allowing users of the VM to run zfs send/receive commands on the zfs filesystem within their bhyve VM. Is this possible and what is/are the methods and options for creating the zfs filesystem (or volume) within the VM? If there is a way to do this, would any of the proposed methods depend on whether the VM lives in a file versus a zfs volume? My VM is file-based. Thanks! From owner-freebsd-virtualization@freebsd.org Wed Jun 24 22:58:26 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9599C9160B6 for ; Wed, 24 Jun 2015 22:58:26 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id 6D53411E0 for ; Wed, 24 Jun 2015 22:58:25 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [192.168.1.10] (unknown [192.168.1.10]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 0FCFC9371 for ; Wed, 24 Jun 2015 22:58:19 +0000 (UTC) Message-ID: <558B3605.4070600@freebsd.org> Date: Wed, 24 Jun 2015 18:58:13 -0400 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-virtualization@freebsd.org Subject: Re: ZFS pool within FreeBSD bhyve guest References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jun 2015 22:58:26 -0000 On 2015-06-24 18:49, javocado wrote: > Hi, > > I would like to create a zfs filesystem within my bhyve (FreeBSD 10.1 as > the guest and host) allowing users of the VM to run zfs send/receive > commands on the zfs filesystem within their bhyve VM. > > Is this possible and what is/are the methods and options for creating the > zfs filesystem (or volume) within the VM? If there is a way to do this, > would any of the proposed methods depend on whether the VM lives in a file > versus a zfs volume? My VM is file-based. > > Thanks! > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > It should just work. You create the zpool inside the VM the same as you would on a bare metal machine. bsdinstall inside the VM should be able to handle it without issue. If you are using a ZFS zvol, you'll want to make sure the 'volmode' is set to dev. For a file-backed VM, there should not be any special steps required. -- Allan Jude From owner-freebsd-virtualization@freebsd.org Fri Jun 26 07:19:54 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 867B998C80A; Fri, 26 Jun 2015 07:19:54 +0000 (UTC) (envelope-from royger@gmail.com) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E4161BAE; Fri, 26 Jun 2015 07:19:54 +0000 (UTC) (envelope-from royger@gmail.com) Received: by wguu7 with SMTP id u7so80738100wgu.3; Fri, 26 Jun 2015 00:19:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; bh=dI5l+vYPy0A2W/O+enVGfAkIGn0TNJp63QIKsF4fw9U=; b=inbfguTK553c/xdofgCm0j/ceUcu9lgDhYfPMJPOr1DMG3c510hzso3jfjuEpAsDba nGwmOfpDG1TCfQRTIheqn5O6+XUsejGHPF1FEtkAZoR7ry40XQQIoBezmpqb3gGRE7lm 33Lfpi7WxLOBzQ0RMBFrzdnGiN+3bal/ypGCh3YpIc/nkVM8vZkSP6TxhhL86yhVp9fC ZHJZ/SLj1V4sw15akHFISfWnfxZQq6EAmvQwwJQF7gRCgU79wBFEg16SSW/oiI/VqxHF lzuQmqUfm8UH1ydQwmAMnBU8EQIh7jOpg3OAX+gJdQ9yI1V4wRf0Jvuzy7XwPzb98BhD RJWQ== X-Received: by 10.180.11.174 with SMTP id r14mr2160453wib.72.1435303192451; Fri, 26 Jun 2015 00:19:52 -0700 (PDT) Received: from [172.16.1.30] (5.Red-79-156-191.staticIP.rima-tde.net. [79.156.191.5]) by mx.google.com with ESMTPSA id u6sm49138434wja.40.2015.06.26.00.19.50 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 26 Jun 2015 00:19:51 -0700 (PDT) Sender: =?UTF-8?Q?Roger_Pau_Monn=C3=A9?= Message-ID: <558CFD11.9050807@FreeBSD.org> Date: Fri, 26 Jun 2015 09:19:45 +0200 From: =?UTF-8?B?Um9nZXIgUGF1IE1vbm7DqQ==?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-virtualization@freebsd.org, FreeBSD XEN Subject: HEAD UP: notice to FreeBSD/Xen Dom0 users Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jun 2015 07:19:54 -0000 Hello, If you are running a FreeBSD/Xen Dom0 and update FreeBSD to r284870 or any later revision make sure you are using the xen-kernel-4.5.0_4 and xen-tools-4.5.0_7 packages (or any later version), or else you won't be able to boot. Sorry for the trouble this might cause. Be aware that the Xen PVH interface (which FreeBSD uses) is still not marked as stable, so other incompatible changes will happen before this is fully stabilized. Users of FreeBSD as a Xen guest (PVHVM mode) are not affected and should ignore this message. Roger.