From owner-freebsd-virtualization@FreeBSD.ORG Sun Mar 31 15:47:31 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E2429E71 for ; Sun, 31 Mar 2013 15:47:31 +0000 (UTC) (envelope-from mack@macktronics.com) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by mx1.freebsd.org (Postfix) with ESMTP id 9FB62155 for ; Sun, 31 Mar 2013 15:47:31 +0000 (UTC) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by coco.macktronics.com (Postfix) with ESMTP id EDD134AC40; Sun, 31 Mar 2013 10:47:24 -0500 (CDT) Date: Sun, 31 Mar 2013 10:47:24 -0500 (CDT) From: Dan Mack To: Neel Natu Subject: Re: bhyve tty / login problems / panic In-Reply-To: Message-ID: <20130331103902.V30526@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> <20130329172023.L15559@coco.macktronics.com> <20130330125954.G17063@coco.macktronics.com> <20130330141141.B17063@coco.macktronics.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Mar 2013 15:47:31 -0000 Here's the latest (note I didn't update my release.iso with the latest, at this point I'm still running a slightly earlier (cd /usr/src/release ; make release) copy from 248935 below. Also, the kernel that the VM is running I actually built while in uni-processor mode which shows how well behaved it is when running in non-SMP. You can see my motherboard type and such from the previous host boot log (Intel desktop board), let me know if you want any ACPI bios setting details; the firmware from Intel is about 1 month old so maybe it is related. Hope this helps, let me know where to upload my release.iso and/or a small disk image to boot if needed. Dan root@olive:/vms # bhyvectl --vm=coco --get-stats --cpu=0 ; bhyvectl --vm=coco --get-stats --cpu=1 vcpu0 vm exits due to external interrupt 29118 number of times hlt was intercepted 3410 number of times %cr access was intercepted 1 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 4 number of monitor trap exits 0 number of times pause was intercepted 7967499 vm exits due to interrupt window opening 11701 vm exits due to nmi window opening 0 number of times in/out was intercepted 480563 number of times cpuid was intercepted 122 vm exits due to nested page fault 1490965 number of vm exits for unknown reason 0 number of times astpending at exit 282 number of vm exits handled in userspace 8451625 number of NMIs delivered to vcpu 0 vcpu total runtime 94121692198 number of ticks vcpu was idle 5750 timer interrupts generated by vlapic 9073 vcpu migration across host cpus 52 number of times hlt was ignored 0 total number of vm exits 9983383 vcpu1 vm exits due to external interrupt 73725 number of times hlt was intercepted 4995 number of times %cr access was intercepted 2 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 4 number of monitor trap exits 0 number of times pause was intercepted 7707742 vm exits due to interrupt window opening 45746 vm exits due to nmi window opening 0 number of times in/out was intercepted 79601 number of times cpuid was intercepted 105 vm exits due to nested page fault 384793 number of vm exits for unknown reason 0 number of times astpending at exit 263 number of vm exits handled in userspace 7792338 number of NMIs delivered to vcpu 0 vcpu total runtime 87399262934 number of ticks vcpu was idle 3940 timer interrupts generated by vlapic 10039 vcpu migration across host cpus 65 number of times hlt was ignored 0 total number of vm exits 8296713 root@olive:/vms # uname -a FreeBSD olive.example.com 10.0-CURRENT FreeBSD 10.0-CURRENT #8 r248949: Sun Mar 31 09:42:40 CDT 2013 root@olive.example.com:/usr/obj/usr/src/sys/MACKGEN amd64 root@olive:/vms # top -bHI last pid: 1417; load averages: 1.96, 0.74, 0.35 up 0+00:26:00 10:38:13 23 processes: 3 running, 20 sleeping Mem: 21M Active, 13M Inact, 8166M Wired, 1920K Cache, 54G Free ARC: 832M Total, 310M MFU, 510M MRU, 8336K Anon, 2364K Header, 2125K Other Swap: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 1390 root 103 0 4120M 139M CPU7 7 1:48 100.00% bhyve{coco vcpu 0} 1390 root 102 0 4120M 139M CPU1 1 1:44 100.00% bhyve{coco vcpu 1} And here is the vm boot sequence and panic again which shows how the clock is going very fast (the crash only took about 2 minutes of wall time if that): FreeBSD/amd64 User boot, Revision 1.1 (root@olive.example.com, Sun Mar 31 09:32:01 CDT 2013) Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2518 data=0x980e8+0xef8b0 syms=[0x8+0xca980+0x8+0x12bb86] / ______ ____ _____ _____ | ____| | _ \ / ____| __ \ | |___ _ __ ___ ___ | |_) | (___ | | | | | ___| '__/ _ \/ _ \| _ < \___ \| | | | | | | | | __/ __/| |_) |____) | |__| | | | | | | | || | | | |_| |_| \___|\___||____/|_____/|_____/ ``` ` s` `.....---.......--.``` -/ +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. | | yo`:. :o `+- | 1. Boot Multi User [Enter] | y/ -/` -o/ | 2. Boot [S]ingle User | .- ::/sy+:. | 3. [Esc]ape to loader prompt | / `-- / | 4. Reboot | `: :` | | `: :` | Options: | / / | 5. Configure Boot [O]ptions... | .- -. | | -- -. | | `:` `:` | | .-- `--. | | .---.....----. +-----------------------------------------+ Booting... ^PCopyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #1 r248935: Sat Mar 30 13:56:05 CDT 2013 root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64 FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.83-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0x8fa3ab7f Features2=0x83bee217 AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant real memory = 5368709120 (5120 MB) avail memory = 3957530624 (3774 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 2 package(s) x 1 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 random device not loaded; using insecure entropy ioapic0: Changing APIC ID to 2 ioapic0 irqs 0-16 on motherboard kbd0 at kbdmux0 acpi0: on motherboard atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib0: no PRT entry for 0.31.INTA virtio_pci0: port 0x2000-0x201f mem 0xc0000000-0xc0001fff at device 1.0 on pci0 vtnet0: on virtio_pci0 virtio_pci0: host features: 0x18020 virtio_pci0: negotiated features: 0x18020 vtnet0: Ethernet address: 00:a0:98:ac:0d:df virtio_pci1: port 0x2040-0x207f mem 0xc0002000-0xc0003fff at device 2.0 on pci0 vtblk0: on virtio_pci1 virtio_pci1: host features: 0x10000004 virtio_pci1: negotiated features: 0x10000004 vtblk0: 8192MB (16777216 512 byte sectors) uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 uart2: console (9600,n,8,1) Timecounters tick every 10.000 msec g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/vtbd0p2 [rw]... WARNING: / was not properly dismounted WARNING: /: mount pending error: blocks 8 files 3 Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf. Setting hostid: 0x24bfd025. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: ** SU+J Recovering /dev/vtbd0p2 ** Reading 33554432 byte journal from inode 4. ** Building recovery table. ** Resolving unreferenced inode list. ** Processing journal entries. ** 89 journal records in 10240 bytes for 27.81% utilization ** Freed 9 inodes (0 dirs) 7 blocks, and 5 frags. ***** FILE SYSTEM MARKED CLEAN ***** g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT Mounting local file systems:. Writing entropy file:. Setting hostname: cocopuff.example.com. vtnet0: link state changed to UP Starting Network: lo0 vtnet0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 vtnet0: flags=8943 metric 0 mtu 1500 options=80028 ether 00:a0:98:ac:0d:df inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 tentative scopeid 0x1 nd6 options=29 media: Ethernet 1000baseT status: active Starting devd. add net default: gateway 172.22.22.252 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. syslogd: timed out waiting for child /etc/rc: WARNING: failed to start syslogd Clearing /tmp (X related). Updating motd:. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Thu May 9 19:33:49 CDT 2013 FreeBSD/amd64 (cocopuff.example.com) (console) login: root Login timed out after 300 seconds FreeBSD/amd64 (cocopuff.example.com) (console) login: panic: deadlkres: possible deadlock detected for 0xfffffe00a90a8000, blocked for 183132 ticks cpuid = 1 Uptime: 249d7h13m51s Automatic reboot in 15 seconds - press a key on the console to abort Rebooting... cpu_reset: Restarting BSP cpu_reset_proxy: Stopped CPU 1 Consoles: userboot