From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 22 11:15:25 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 05ADCF77 for ; Mon, 22 Jun 2015 11:15:25 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 234F61A1 for ; Mon, 22 Jun 2015 11:15:23 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA25992 for ; Mon, 22 Jun 2015 14:15:21 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z6zhB-00053Z-FW for freebsd-virtualization@freebsd.org; Mon, 22 Jun 2015 14:15:21 +0300 Message-ID: <5587EE05.2020001@FreeBSD.org> Date: Mon, 22 Jun 2015 14:14:13 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "freebsd-virtualization@freebsd.org" Subject: bhyve: centos 7.1 with multiple virtual processors Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jun 2015 11:15:25 -0000 If I run a CentOS 7.1 VM with more than one CPU more often than not it would hang on startup and bhyve would start spinning. The following are the last messages seen in the VM: Switching to clocksource hpet ------------[ cut here ]------------ WARNING: at kernel/time/clockevents.c:239 clockevents_program_event+0xdb/0xf0() Modules linked in: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1 Hardware name: BHYVE, BIOS 1.00 03/14/2014 0000000000000000 00000000cab5bdb6 ffff88003fc03e08 ffffffff81604eaa ffff88003fc03e40 ffffffff8106e34b 80000000000f423f 80000000000f423f ffffffff81915440 0000000000000000 0000000000000000 ffff88003fc03e50 Call Trace: [] dump_stack+0x19/0x1b [] warn_slowpath_common+0x6b/0xb0 [] warn_slowpath_null+0x1a/0x20 [] clockevents_program_event+0xdb/0xf0 [] tick_handle_periodic_broadcast+0x41/0x50 [] timer_interrupt+0x15/0x20 [] handle_irq_event_percpu+0x3e/0x1e0 [] handle_irq_event+0x3d/0x60 [] handle_edge_irq+0x77/0x130 [] handle_irq+0xbf/0x150 [] ? irq_enter+0x17/0xa0 [] do_IRQ+0x4f/0xf0 [] common_interrupt+0x6d/0x6d [] ? selinux_inode_alloc_security+0x59/0xa0 [] ? __d_instantiate+0xbf/0x100 [] ? __d_instantiate+0x9f/0x100 [] d_instantiate+0x3d/0x70 [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130 [] __create_file+0x1c2/0x2c0 [] ? set_graph_function+0x1f/0x1f [] debugfs_create_dir+0x1b/0x20 [] tracing_init_dentry_tr+0x7e/0x90 [] tracing_init_dentry+0x10/0x20 [] ftrace_init_debugfs+0x13/0x1fd [] ? set_graph_function+0x1f/0x1f [] do_one_initcall+0xb8/0x230 [] kernel_init_freeable+0x18b/0x22a [] ? initcall_blacklist+0xb0/0xb0 [] ? rest_init+0x80/0x80 [] kernel_init+0xe/0xf0 [] ret_from_fork+0x7c/0xb0 [] ? rest_init+0x80/0x80 ---[ end trace d5caa1cab8e7e98d ]--- At the same time sometimes there is one or more of spurious NMIs on the _host_ system: NMI ISA c, EISA ff NMI ... going to debugger bhyve seems to spin here: vmm.ko`svm_vmrun+0x894 vmm.ko`vm_run+0xbb7 vmm.ko`vmmdev_ioctl+0x5a4 kernel`devfs_ioctl_f+0x13b kernel`kern_ioctl+0x1e1 kernel`sys_ioctl+0x16a kernel`amd64_syscall+0x3ca kernel`0xffffffff8088997b (kgdb) list *svm_vmrun+0x894 0xffffffff813c9194 is in svm_vmrun (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895). 1890 1891 static __inline void 1892 enable_gintr(void) 1893 { 1894 1895 __asm __volatile("stgi"); 1896 } 1897 1898 /* 1899 * Start vcpu with specified RIP. -- Andriy Gapon