From owner-freebsd-hackers@FreeBSD.ORG Tue Jun 17 16:59:17 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 76FB3D5A; Tue, 17 Jun 2014 16:59:17 +0000 (UTC) Received: from secure.freebsdsolutions.net (secure.freebsdsolutions.net [69.55.234.48]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3B413246A; Tue, 17 Jun 2014 16:59:16 +0000 (UTC) Received: from [10.10.1.198] (office.betterlinux.com [199.58.199.60]) (authenticated bits=0) by secure.freebsdsolutions.net (8.14.4/8.14.4) with ESMTP id s5HGx6rx056955 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 17 Jun 2014 12:59:07 -0400 (EDT) (envelope-from lists@jnielsen.net) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.2\)) Subject: Re: consistent VM hang during reboot From: John Nielsen In-Reply-To: <0238084D-FD0F-42A5-85F5-597A590E666C@jnielsen.net> Date: Tue, 17 Jun 2014 10:59:14 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <7D960087-1839-4295-AEE3-5D8F60D9D710@jnielsen.net> References: <201405081303.17079.jhb@freebsd.org> <2CCD4068-A9CB-442C-BB91-ADBF62FF22C6@jnielsen.net> <83DA2398-0004-49EC-8AC1-9AA64F33A194@jnielsen.net> <0238084D-FD0F-42A5-85F5-597A590E666C@jnielsen.net> To: "freebsd-hackers@freebsd.org" , "freebsd-virtualization@freebsd.org" X-Mailer: Apple Mail (2.1878.2) X-DCC-sonic.net-Metrics: ns1.jnielsen.net 1117; Body=2 Fuz1=2 Fuz2=2 X-Virus-Scanned: clamav-milter 0.97.8 at ns1.jnielsen.net X-Virus-Status: Clean X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Jun 2014 16:59:17 -0000 On Jun 13, 2014, at 4:23 PM, John Nielsen wrote: > On Wednesday, May 07, 2014 7:15:43 pm John Nielsen wrote: >> I am trying to solve a problem with amd64 FreeBSD virtual machines = running on a Linux+KVM hypervisor. To be honest I'm not sure if the = problem is in FreeBSD or the hypervisor, but I'm trying to rule out the = OS first. >>=20 >> The _second_ time FreeBSD boots in a virtual machine with more than = one core, the boot hangs just before the kernel would normally print = e.g. "SMP: AP CPU #1 Launched!" (The last line on the console is = "usbus0: 12Mbps Full Speed USB v1.0", but the problem persists even = without USB). The VM will boot fine a first time, but running either = "shutdown -r now" OR "reboot" will lead to a hung second boot. Stopping = and starting the host qemu-kvm process is the only way to continue. ... > Following up on the off chance anyone else is interested. I installed = -HEAD on a host that was having the problem ("v2" Xeon CPU) and ran a = FreeBSD 9 VM under bhyve. The problem did _not_ persist. That's not = entirely conclusive but it does point the finger at Qemu a bit more = strongly. I have filed a bug with them: > https://bugs.launchpad.net/qemu/+bug/1329956 With some help from the Qemu and KVM folks I've finally made some = headway. The salient difference between the working and non-working CPUs = above seems to be support for APIC virtualization. Loading the intel_kvm = module (on the Linux host) with "enable_apicv=3DN" works around the = reboot problem I've been having. Since this now looks like a Linux KVM bug I won't follow up here any = more, but I wanted to wrap up the thread for the archives. JN