From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 23 07:26:44 2015 Return-Path: Delivered-To: freebsd-virtualization@nevdull.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BDDA0E96 for ; Tue, 23 Jun 2015 07:26:44 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wg0-x22a.google.com (mail-wg0-x22a.google.com [IPv6:2a00:1450:400c:c00::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 76C4B2BD; Tue, 23 Jun 2015 07:26:44 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wgbhy7 with SMTP id hy7so1426127wgb.2; Tue, 23 Jun 2015 00:26:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=pNWGC1SdjAjybulzCKHAuzt1CXTftPr+dRGN0dKZzF0=; b=MITG+k0WvwoNjTVkSRPEm8b9r0UDhHhWFV+4j4G+Gt1Qmefay6U1pBZZ8UhHc70em+ dMVx+H2Gp0ofM0WZbdDJ0+ptw8H1FLbafkmJuilmRt0qeBas4roKLmRgBjYfcpRDb1T/ OCqchk+cDoKRjKRoEXJjbgItex9MO+WA+rgK19INIj580PkizLiWPxCCXt22ePIBUxEx UPLjn5k6iBScNXzHLpuzXYOJF7oGdZDo751JJrUibdmy6VLgjdGHfaUIpeJ/K3xvKHNU YCmShr9Baikg9O2XC7IABUZ4Frtu2ICq4rL/j0ypgknHQgV+yyQEyLOAJlzw40Wc0pDv 59Zw== MIME-Version: 1.0 X-Received: by 10.194.192.166 with SMTP id hh6mr57287450wjc.127.1435044402811; Tue, 23 Jun 2015 00:26:42 -0700 (PDT) Received: by 10.27.52.79 with HTTP; Tue, 23 Jun 2015 00:26:42 -0700 (PDT) In-Reply-To: <558900A7.40609@FreeBSD.org> References: <5587EE05.2020001@FreeBSD.org> <558900A7.40609@FreeBSD.org> Date: Tue, 23 Jun 2015 00:26:42 -0700 Message-ID: Subject: Re: bhyve: centos 7.1 with multiple virtual processors From: Neel Natu To: Andriy Gapon Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jun 2015 07:26:44 -0000 Hi Andriy, On Mon, Jun 22, 2015 at 11:45 PM, Andriy Gapon wrote: > On 23/06/2015 05:37, Neel Natu wrote: >> Hi Andriy, >> >> FWIW I can boot up a Centos 7.1 virtual machine with 2 and 4 vcpus >> fine on my host with 8 physical cores. >> >> I have some questions about your setup inline. >> >> On Mon, Jun 22, 2015 at 4:14 AM, Andriy Gapon wrote: >>> >>> If I run a CentOS 7.1 VM with more than one CPU more often than not it would >>> hang on startup and bhyve would start spinning. >>> >>> The following are the last messages seen in the VM: >>> >>> Switching to clocksource hpet >>> ------------[ cut here ]------------ >>> WARNING: at kernel/time/clockevents.c:239 clockevents_program_event+0xdb/0xf0() >>> Modules linked in: >>> CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-229.4.2.el7.x86_64 #1 >>> Hardware name: BHYVE, BIOS 1.00 03/14/2014 >>> 0000000000000000 00000000cab5bdb6 ffff88003fc03e08 ffffffff81604eaa >>> ffff88003fc03e40 ffffffff8106e34b 80000000000f423f 80000000000f423f >>> ffffffff81915440 0000000000000000 0000000000000000 ffff88003fc03e50 >>> Call Trace: >>> [] dump_stack+0x19/0x1b >>> [] warn_slowpath_common+0x6b/0xb0 >>> [] warn_slowpath_null+0x1a/0x20 >>> [] clockevents_program_event+0xdb/0xf0 >>> [] tick_handle_periodic_broadcast+0x41/0x50 >>> [] timer_interrupt+0x15/0x20 >>> [] handle_irq_event_percpu+0x3e/0x1e0 >>> [] handle_irq_event+0x3d/0x60 >>> [] handle_edge_irq+0x77/0x130 >>> [] handle_irq+0xbf/0x150 >>> [] ? irq_enter+0x17/0xa0 >>> [] do_IRQ+0x4f/0xf0 >>> [] common_interrupt+0x6d/0x6d >>> [] ? selinux_inode_alloc_security+0x59/0xa0 >>> [] ? __d_instantiate+0xbf/0x100 >>> [] ? __d_instantiate+0x9f/0x100 >>> [] d_instantiate+0x3d/0x70 >>> [] debugfs_mknod.isra.5.part.6.constprop.15+0x98/0x130 >>> [] __create_file+0x1c2/0x2c0 >>> [] ? set_graph_function+0x1f/0x1f >>> [] debugfs_create_dir+0x1b/0x20 >>> [] tracing_init_dentry_tr+0x7e/0x90 >>> [] tracing_init_dentry+0x10/0x20 >>> [] ftrace_init_debugfs+0x13/0x1fd >>> [] ? set_graph_function+0x1f/0x1f >>> [] do_one_initcall+0xb8/0x230 >>> [] kernel_init_freeable+0x18b/0x22a >>> [] ? initcall_blacklist+0xb0/0xb0 >>> [] ? rest_init+0x80/0x80 >>> [] kernel_init+0xe/0xf0 >>> [] ret_from_fork+0x7c/0xb0 >>> [] ? rest_init+0x80/0x80 >>> ---[ end trace d5caa1cab8e7e98d ]--- >>> >> >> A few questions to narrow this down: >> - Is the host very busy when the VM is started (or what is the host >> doing when this happened)? > > The host typically is not heavily loaded. There is X server running and some > applications. I'd imagine that those could cause some additional latency but > not CPU starvation. > Yup, I agree. Does this ever happen with a single vcpu guest? The other mystery is the NMIs the host is receiving. I (re)verified to make sure that bhyve/vmm.ko do not assert NMIs so it has to be something else on the host that's doing it ... best Neel >> - How many vcpus are you giving to the VM? >> - How many cores on the host? > > I tried only 2 / 2. > >>> >>> At the same time sometimes there is one or more of spurious NMIs on the _host_ >>> system: >>> NMI ISA c, EISA ff >>> NMI ... going to debugger >>> >> >> Hmm, that's interesting. Are you using hwpmc to do instruction sampling? > > hwpmc driver is in the kernel, but it was not used. > >>> bhyve seems to spin here: >>> vmm.ko`svm_vmrun+0x894 >>> vmm.ko`vm_run+0xbb7 >>> vmm.ko`vmmdev_ioctl+0x5a4 >>> kernel`devfs_ioctl_f+0x13b >>> kernel`kern_ioctl+0x1e1 >>> kernel`sys_ioctl+0x16a >>> kernel`amd64_syscall+0x3ca >>> kernel`0xffffffff8088997b >>> >>> (kgdb) list *svm_vmrun+0x894 >>> 0xffffffff813c9194 is in svm_vmrun >>> (/usr/src/sys/modules/vmm/../../amd64/vmm/amd/svm.c:1895). >>> 1890 >>> 1891 static __inline void >>> 1892 enable_gintr(void) >>> 1893 { >>> 1894 >>> 1895 __asm __volatile("stgi"); >>> 1896 } >>> 1897 >>> 1898 /* >>> 1899 * Start vcpu with specified RIP. >>> >> >> Yeah, that's not surprising because host interrupts are blocked when >> the cpu is executing in guest context. The 'enable_gintr()' enables >> interrupts so it gets blamed by the interrupt-based sampling. >> >> In this case it just means that the cpu was in guest context when a >> host-interrupt fired. > > I see. FWIW, that was captured with DTrace. > > -- > Andriy Gapon