From owner-freebsd-virtualization@FreeBSD.ORG Sat Mar 30 19:17:45 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2C66515A for ; Sat, 30 Mar 2013 19:17:45 +0000 (UTC) (envelope-from mack@macktronics.com) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by mx1.freebsd.org (Postfix) with ESMTP id 7CF7E833 for ; Sat, 30 Mar 2013 19:17:44 +0000 (UTC) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by coco.macktronics.com (Postfix) with ESMTP id 96BF94AC40; Sat, 30 Mar 2013 14:17:43 -0500 (CDT) Date: Sat, 30 Mar 2013 14:17:43 -0500 (CDT) From: Dan Mack To: Neel Natu Subject: Re: bhyve tty / login problems / panic In-Reply-To: <20130330125954.G17063@coco.macktronics.com> Message-ID: <20130330141141.B17063@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> <20130329172023.L15559@coco.macktronics.com> <20130330125954.G17063@coco.macktronics.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Mar 2013 19:17:45 -0000 Here you go: - uni-processor output: root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 vcpu0 vm exits due to external interrupt 1330 number of times hlt was intercepted 676 number of times %cr access was intercepted 1 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 4 number of monitor trap exits 0 number of times pause was intercepted 428499 vm exits due to interrupt window opening 1231 vm exits due to nmi window opening 0 number of times in/out was intercepted 423985 number of times cpuid was intercepted 22 vm exits due to nested page fault 7523 number of vm exits for unknown reason 0 number of times astpending at exit 38 number of vm exits handled in userspace 853290 number of NMIs delivered to vcpu 0 vcpu total runtime 6954314530 number of ticks vcpu was idle 59572 vcpu migration across host cpus 29 number of times hlt was ignored 0 total number of vm exits 863271 And here is when running with vcpu==2: root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 vcpu0 vm exits due to external interrupt 19234 number of times hlt was intercepted 1965 number of times %cr access was intercepted 1 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 4 number of monitor trap exits 0 number of times pause was intercepted 3907709 vm exits due to interrupt window opening 8105 vm exits due to nmi window opening 0 number of times in/out was intercepted 457998 number of times cpuid was intercepted 50 vm exits due to nested page fault 940909 number of vm exits for unknown reason 0 number of times astpending at exit 102 number of vm exits handled in userspace 4367825 number of NMIs delivered to vcpu 0 vcpu total runtime 51834598990 number of ticks vcpu was idle 887 vcpu migration across host cpus 25 number of times hlt was ignored 0 total number of vm exits 5335975 root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 vcpu1 vm exits due to external interrupt 51249 number of times hlt was intercepted 3676 number of times %cr access was intercepted 2 number of times rdmsr was intercepted 0 number of times wrmsr was intercepted 4 number of monitor trap exits 0 number of times pause was intercepted 4667527 vm exits due to interrupt window opening 32453 vm exits due to nmi window opening 0 number of times in/out was intercepted 57278 number of times cpuid was intercepted 63 vm exits due to nested page fault 266091 number of vm exits for unknown reason 0 number of times astpending at exit 164 number of vm exits handled in userspace 4728481 number of NMIs delivered to vcpu 0 vcpu total runtime 53833466076 number of ticks vcpu was idle 965 vcpu migration across host cpus 16 number of times hlt was ignored 0 total number of vm exits 5078343 The panic comes eventually like this: login: panic: deadlkres: possible deadlock detected for 0xfffffe0090fb5490, blocked for 217798 ticks cpuid = 0 Uptime: 255d6h29m56s Automatic reboot in 15 seconds - press a key on the console to abort Hope this helps, and FYI, here's a dmesg from the host: Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #7 r248935: Sat Mar 30 13:19:43 CDT 2013 root@olive.example.com:/usr/obj/usr/src/sys/MACKGEN amd64 FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3200.19-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0xbfebfbff Features2=0x1fbee3bf AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 68719476736 (65536 MB) avail memory = 63123800064 (60199 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 8 cpu9 (AP): APIC ID: 9 cpu10 (AP): APIC ID: 10 cpu11 (AP): APIC ID: 11 ioapic1 irqs 24-47 on motherboard ioapic0 irqs 0-23 on motherboard lapic0: Forcing LINT1 to edge trigger kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) acpi0: reservation of f0000000, 18000000 (3) failed cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed03fff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 pcib2: at device 1.1 on pci0 pci2: on pcib2 xhci0: mem 0xe3400000-0xe3401fff irq 16 at device 0.0 on pci2 usbus0: waiting for BIOS to give up control xhci0: 32 byte context size. usbus0 on xhci0 pcib3: at device 2.0 on pci0 pci3: on pcib3 vgapci0: port 0x3000-0x307f mem 0xe2000000-0xe2ffffff,0xd0000000-0xdfffffff,0xe0000000-0xe1ffffff irq 16 at device 0.0 on pci3 pci3: at device 0.1 (no driver attached) pcib4: at device 3.0 on pci0 pci4: on pcib4 pci0: at device 5.0 (no driver attached) pci0: at device 5.2 (no driver attached) pcib5: at device 17.0 on pci0 pci5: on pcib5 pci0: at device 22.0 (no driver attached) em0: port 0x4040-0x405f mem 0xe3500000-0xe351ffff,0xe3521000-0xe3521fff irq 20 at device 25.0 on pci0 em0: Using an MSI interrupt em0: Ethernet address: 4c:72:b9:d1:b2:d2 ehci0: mem 0xe3601000-0xe36013ff irq 16 at device 26.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci0 pcib6: at device 28.0 on pci0 pci6: on pcib6 pcib7: at device 28.2 on pci0 pci7: on pcib7 pci7: at device 0.0 (no driver attached) pcib8: at device 28.3 on pci0 pci8: on pcib8 xhci1: mem 0xe3200000-0xe3201fff irq 19 at device 0.0 on pci8 usbus2: waiting for BIOS to give up control xhci1: 32 byte context size. usbus2 on xhci1 pcib9: at device 28.5 on pci0 pci9: on pcib9 em1: port 0x1000-0x101f mem 0xe3100000-0xe311ffff,0xe3120000-0xe3123fff irq 17 at device 0.0 on pci9 em1: Using MSIX interrupts with 3 vectors em1: Ethernet address: 4c:72:b9:d1:b2:d3 ehci1: mem 0xe3602000-0xe36023ff irq 23 at device 29.0 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci1 pcib10: at device 30.0 on pci0 pci10: on pcib10 isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0x4068-0x406f,0x4074-0x4077,0x4060-0x4067,0x4070-0x4073,0x4020-0x403f mem 0xe3523000-0xe35237ff irq 18 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 ahcich4: at channel 4 on ahci0 ahcich5: at channel 5 on ahci0 ahciem0: on ahci0 pci0: at device 31.3 (no driver attached) orm0: at iomem 0xd1000-0xd1fff,0xd2000-0xd2fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ppc0: cannot reserve I/O port range est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 est2: on cpu2 p4tcc2: on cpu2 est3: on cpu3 p4tcc3: on cpu3 est4: on cpu4 p4tcc4: on cpu4 est5: on cpu5 p4tcc5: on cpu5 est6: on cpu6 p4tcc6: on cpu6 est7: on cpu7 p4tcc7: on cpu7 est8: on cpu8 p4tcc8: on cpu8 est9: on cpu9 p4tcc9: on cpu9 est10: on cpu10 p4tcc10: on cpu10 est11: on cpu11 p4tcc11: on cpu11 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec usbus0: 5.0Gbps Super Speed USB v3.0 usbus1: 480Mbps High Speed USB v2.0 usbus2: 5.0Gbps Super Speed USB v3.0 usbus3: 480Mbps High Speed USB v2.0 ugen1.1: at usbus1 uhub0: on usbus1 ugen0.1: <0x1033> at usbus0 uhub1: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 ugen3.1: at usbus3 uhub2: on usbus3 ugen2.1: <0x1033> at usbus2 uhub3: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus2 ses0 at ahciem0 bus 0 scbus6 target 0 lun 0 ses0: SEMB S-E-S 2.00 device ses0: SEMB SES Device ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-9 SATA 3.x device ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 122104MB (250069680 512 byte sectors: 16H 63S/T 16383C) ada0: Previously was known as ad4 ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: ATA-9 SATA 3.x device ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C) ada1: Previously was known as ad6 ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 ada2: ATA-8 SATA 3.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada2: Previously was known as ad8 ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 ada3: ATA-8 SATA 3.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada3: Previously was known as ad10 ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 ada4: ATA-8 SATA 3.x device ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada4: Command Queueing enabled ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada4: Previously was known as ad12 ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 ada5: ATA-8 SATA 3.x device ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada5: Command Queueing enabled ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada5: Previously was known as ad14 lapic1: Forcing LINT1 to edge trigger SMP: AP CPU #1 Launched! lapic4: Forcing LINT1 to edge trigger SMP: AP CPU #4 Launched! lapic11: Forcing LINT1 to edge trigger SMP: AP CPU #11 Launched! lapic7: Forcing LINT1 to edge trigger SMP: AP CPU #7 Launched! lapic5: Forcing LINT1 to edge trigger SMP: AP CPU #5 Launched! lapic9: Forcing LINT1 to edge trigger SMP: AP CPU #9 Launched! lapic2: Forcing LINT1 to edge trigger SMP: AP CPU #2 Launched! lapic3: Forcing LINT1 to edge trigger SMP: AP CPU #3 Launched! lapic10: Forcing LINT1 to edge trigger SMP: AP CPU #10 Launched! lapic6: Forcing LINT1 to edge trigger SMP: AP CPU #6 Launched! lapic8: Forcing LINT1 to edge trigger SMP: AP CPU #8 Launched! uhub3: 4 ports with 4 removable, self powered uhub1: 4 ports with 4 removable, self powered Root mount waiting for: usbus3 usbus1 uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered Root mount waiting for: usbus3 usbus1 ugen1.2: at usbus1 uhub4: on usbus1 ugen3.2: at usbus3 uhub5: on usbus3 Root mount waiting for: usbus3 usbus1 uhub4: 6 ports with 6 removable, self powered uhub5: 8 ports with 8 removable, self powered ugen1.3: at usbus1 ukbd0: on usbus1 kbd2 at ukbd0 ugen3.3: at usbus3 Root mount waiting for: usbus3 ugen3.4: at usbus3 Trying to mount root from zfs:tron []... ums0: on usbus1 ums0: 5 buttons and [XYZ] coordinates ID=0 run0: <1.0> on usbus3 run0: MAC/BBP RT3070 (rev 0x0201), RF RT3020 (MIMO 1T1R), address 94:db:c9:e3:6e:c8 ubt0: on usbus3 WARNING: attempt to domain_add(bluetooth) after domainfinalize() WARNING: attempt to domain_add(netgraph) after domainfinalize() tap0: Ethernet address: 00:bd:8c:54:07:00 tap0: link state changed to UP bridge0: Ethernet address: 02:4c:89:ce:33:00 tap0: promiscuous mode enabled bridge0: link state changed to UP em1: link state changed to DOWN em1: promiscuous mode enabled em1: link state changed to UP tap0: link state changed to DOWN tap0: link state changed to UP tap0: link state changed to DOWN tap0: link state changed to UP > > Yep, no problem. I saw the checkin and started a rebuild a few minutes ago > :-) > > Thanks, > > Dan > > On Sat, 30 Mar 2013, Neel Natu wrote: > >> Hi Dan, >> >> On Fri, Mar 29, 2013 at 3:20 PM, Dan Mack wrote: >> >>> >>> >>> Here you go Neel: >>> >>> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 >>> vcpu0 >>> vm exits due to external interrupt 45051 >>> number of times hlt was intercepted 2277 >>> number of NMIs delivered to vcpu 0 >>> vcpu total runtime 105001652482 >>> number of ticks vcpu was idle 919 >>> vcpu migration across host cpus 13 >>> number of times hlt was ignored 0 >>> total number of vm exits 9240925 >>> root@olive:~ # >>> root@olive:~ # >>> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 >>> vcpu1 >>> vm exits due to external interrupt 149431 >>> number of times hlt was intercepted 4222 >>> number of NMIs delivered to vcpu 0 >>> vcpu total runtime 106876622528 >>> number of ticks vcpu was idle 869 >>> vcpu migration across host cpus 20 >>> number of times hlt was ignored 0 >>> total number of vm exits 8065909 >>> >>> >> That did not help a whole lot because we were not keeping track of all the >> reasons a vcpu could exit. >> >> I have fixed that in r248935: >> http://svnweb.freebsd.org/base?view=revision&revision=248935 >> >> Do you mind updating your vmm.ko with the change and getting the stats >> again? >> >> best >> Neel >> >> >>> Dan >>> >>> >>> >>> On Fri, 29 Mar 2013, Neel Natu wrote: >>> >>> Hi Dan, >>>> >>>> On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack wrote: >>>> >>>> >>>>> I ran the procsystime dtrace script on bhyve when it was chewing up all >>>>> the CPU in vCPU=2 mode, and this is what I see for about 10s of runtime: >>>>> >>>>> root@olive:/usr/share/dtrace/****toolkit # ./procsystime -n bhyve -aT >>>>> >>>>> Tracing... Hit Ctrl-C to end... >>>>> dtrace: 158536 dynamic variable drops with non-empty dirty list >>>>> dtrace: 207447 dynamic variable drops with non-empty dirty list >>>>> dtrace: 189205 dynamic variable drops with non-empty dirty list >>>>> dtrace: 164341 dynamic variable drops with non-empty dirty list >>>>> dtrace: 246307 dynamic variable drops with non-empty dirty list >>>>> dtrace: 187640 dynamic variable drops with non-empty dirty list >>>>> dtrace: 214771 dynamic variable drops with non-empty dirty list >>>>> dtrace: 221265 dynamic variable drops with non-empty dirty list >>>>> ^C >>>>> dtrace: 243468 dynamic variable drops with non-empty dirty list >>>>> >>>>> Elapsed Times for processes bhyve, >>>>> >>>>> SYSCALL TIME (ns) >>>>> _umtx_op 18349 >>>>> writev 135712 >>>>> preadv 16175267 >>>>> pwritev 22924378 >>>>> ioctl 4353897920 >>>>> TOTAL: 4393151626 >>>>> >>>>> CPU Times for processes bhyve, >>>>> >>>>> SYSCALL TIME (ns) >>>>> _umtx_op 8815 >>>>> writev 103145 >>>>> pwritev 10647023 >>>>> preadv 15159976 >>>>> ioctl 3943399889 >>>>> TOTAL: 3969318848 >>>>> >>>>> Syscall Counts for processes bhyve, >>>>> >>>>> SYSCALL COUNT >>>>> _umtx_op 17 >>>>> writev 43 >>>>> pwritev 639 >>>>> preadv 1183 >>>>> ioctl 652782 >>>>> TOTAL: 654664 >>>>> >>>>> Not sure if that helps or not. Hotkernel shows the cpu in the >>>>> kernel`acpi_cpu_c1 call most of the time, normal ? >>>>> >>>>> Dan >>>>> >>>>> root@olive:/usr/share/dtrace/****toolkit # ./hotkernel >>>>> >>>>> Sampling... Hit Ctrl-C to end. >>>>> ^C >>>>> FUNCTION COUNT PCNT >>>>> zfs.ko`space_map_sync 1 0.0% >>>>> kernel`syscall_thread_exit 1 0.0% >>>>> kernel`should_yield 1 0.0% >>>>> zfs.ko`lzjb_compress 1 0.0% >>>>> zfs.ko`dbuf_rele 1 0.0% >>>>> zfs.ko`dsl_pool_tempreserve_****space 1 0.0% >>>>> >>>>> kernel`_bus_dmamap_load_buffer 1 0.0% >>>>> kernel`cpu_idleclock 1 0.0% >>>>> kernel`_rw_runlock_cookie 1 0.0% >>>>> kernel`pmap_enter 1 0.0% >>>>> kernel`__lockmgr_args 1 0.0% >>>>> zfs.ko`vdev_geom_io_start 1 0.0% >>>>> kernel`ahci_ch_intr 1 0.0% >>>>> kernel`__rw_rlock 1 0.0% >>>>> kernel`VOP_UNLOCK_APV 1 0.0% >>>>> zfs.ko`arc_buf_thaw 1 0.0% >>>>> kernel`atomic_set_int 1 0.0% >>>>> kernel`g_io_schedule_down 1 0.0% >>>>> kernel`__mtx_lock_flags 1 0.0% >>>>> kernel`_callout_stop_safe 1 0.0% >>>>> kernel`sched_idletd 1 0.0% >>>>> kernel`_sx_slock 1 0.0% >>>>> vmm.ko`lapic_set_intr 1 0.0% >>>>> kernel`cpusetobj_ffs 1 0.0% >>>>> zfs.ko`dbuf_hold 1 0.0% >>>>> vmm.ko`vlapic_intr_accepted 1 0.0% >>>>> zfs.ko`sa_attr_op 1 0.0% >>>>> zfs.ko`dmu_buf_hold_array_by_****dnode 1 0.0% >>>>> >>>>> zfs.ko`taskq_run_safe 1 0.0% >>>>> kernel`vm_reserv_free_page 1 0.0% >>>>> kernel`_sx_sunlock 1 0.0% >>>>> kernel`vn_rlimit_fsize 1 0.0% >>>>> zfs.ko`vdev_queue_io_to_issue 1 0.0% >>>>> kernel`vn_write 1 0.0% >>>>> kernel`lock_init 1 0.0% >>>>> vmm.ko`vmx_resume 1 0.0% >>>>> zfs.ko`zio_gang_tree_free 1 0.0% >>>>> zfs.ko`dbuf_rele_and_unlock 1 0.0% >>>>> kernel`mnt_vnode_next_active 1 0.0% >>>>> kernel`vm_page_remove 1 0.0% >>>>> zfs.ko`vdev_queue_deadline_****compare 1 0.0% >>>>> >>>>> kernel`zone_free_item 1 0.0% >>>>> vmm.ko`vmx_launch 1 0.0% >>>>> kernel`m_pullup 1 0.0% >>>>> kernel`pmap_change_wiring 1 0.0% >>>>> kernel`sched_userret 1 0.0% >>>>> zfs.ko`dbuf_write_done 1 0.0% >>>>> zfs.ko`zio_execute 1 0.0% >>>>> kernel`PHYS_TO_VM_PAGE 2 0.0% >>>>> zfs.ko`avl_rotation 2 0.0% >>>>> kernel`ehci_device_isoc_fs_****enter 2 0.0% >>>>> >>>>> vmm.ko`vm_exitinfo 2 0.0% >>>>> zfs.ko`zfs_freebsd_write 2 0.0% >>>>> vmm.ko`vcpu_stats 2 0.0% >>>>> kernel`atomic_add_long 2 0.0% >>>>> vmm.ko`0xffffffff81 2 0.0% >>>>> kernel`vm_reserv_alloc_page 2 0.0% >>>>> kernel`pagezero 2 0.0% >>>>> kernel`bzero 2 0.0% >>>>> zfs.ko`fletcher_2_native 2 0.0% >>>>> vmm.ko`vmx_getreg 2 0.0% >>>>> vmm.ko`vm_guest_msrs 3 0.0% >>>>> vmm.ko`vmx_setreg 3 0.0% >>>>> vmm.ko`vm_get_register 3 0.0% >>>>> kernel`sx_try_xlock_ 3 0.0% >>>>> kernel`vm_radix_lookup_le 3 0.0% >>>>> vmm.ko`lapic_pending_intr 4 0.0% >>>>> kernel`syscall_thread_enter 4 0.0% >>>>> kernel`cpu_set_syscall_retval 4 0.0% >>>>> kernel`__mnt_vnode_next_all 4 0.0% >>>>> kernel`vfs_stdsync 4 0.0% >>>>> vmm.ko`vmm_fetch_instruction 5 0.0% >>>>> vmm.ko`vlapic_op_mem_read 6 0.0% >>>>> kernel`_sx_xunlock 7 0.0% >>>>> vmm.ko`vmm_emulate_instruction 7 0.0% >>>>> vmm.ko`vlapic_op_mem_write 7 0.0% >>>>> kernel`__mtx_lock_spin_flags 8 0.0% >>>>> vmm.ko`lapic_timer_tick 8 0.0% >>>>> zfs.ko`list_next 8 0.0% >>>>> zfs.ko`l2arc_feed_thread 8 0.0% >>>>> kernel`userret 9 0.0% >>>>> vmm.ko`vm_gpa2hpa 9 0.0% >>>>> vmm.ko`vmm_decode_instruction 11 0.0% >>>>> vmm.ko`ept_vmmmap_get 11 0.0% >>>>> vmm.ko`vm_lapic 11 0.0% >>>>> kernel`spinlock_enter 14 0.0% >>>>> kernel`__mtx_lock_sleep 15 0.0% >>>>> vmm.ko`vcpu_set_state 16 0.0% >>>>> vmm.ko`vm_nmi_pending 16 0.0% >>>>> kernel`malloc_type_zone_****allocated 16 0.0% >>>>> >>>>> kernel`devfs_ioctl_f 16 0.0% >>>>> kernel`pmap_kextract 17 0.0% >>>>> kernel`sys_ioctl 18 0.0% >>>>> kernel`malloc 21 0.0% >>>>> kernel`cpu_fetch_syscall_args 24 0.0% >>>>> kernel`uma_zfree_arg 28 0.0% >>>>> vmm.ko`vlapic_pending_intr 29 0.0% >>>>> kernel`trap 30 0.0% >>>>> kernel`hpet_get_timecount 31 0.0% >>>>> dtrace.ko`dtrace_trap 35 0.0% >>>>> vmm.ko`vlapic_update_ppr 39 0.0% >>>>> vmm.ko`vmmdev_ioctl 40 0.0% >>>>> kernel`copyout 43 0.0% >>>>> kernel`critical_enter 44 0.0% >>>>> kernel`copyin 51 0.1% >>>>> vmm.ko`vmcs_read 53 0.1% >>>>> kernel`fpusave 61 0.1% >>>>> kernel`uma_zalloc_arg 62 0.1% >>>>> kernel`devvn_refthread 69 0.1% >>>>> kernel`atomic_load_acq_int 70 0.1% >>>>> kernel`fpurestore 70 0.1% >>>>> vmm.ko`vlapic_timer_tick 75 0.1% >>>>> kernel`amd64_syscall 77 0.1% >>>>> kernel`bcopy 91 0.1% >>>>> kernel`fpudna 103 0.1% >>>>> kernel`free 103 0.1% >>>>> kernel`critical_exit 105 0.1% >>>>> kernel`dev_relthread 108 0.1% >>>>> kernel`spinlock_exit 119 0.1% >>>>> kernel`kern_ioctl 138 0.1% >>>>> kernel`fpuexit 242 0.3% >>>>> kernel`0xffffffff80 306 0.3% >>>>> vmm.ko`restore_host_msrs 398 0.4% >>>>> vmm.ko`vm_run 428 0.4% >>>>> vmm.ko`restore_guest_msrs 433 0.5% >>>>> kernel`cpu_idle 906 1.0% >>>>> vmm.ko`vmx_run 9266 9.7% >>>>> kernel`acpi_cpu_c1 81134 85.2% >>>>> >>>>> Hope this helps, let me know if you have any ideas on what to try next. >>>>> >>>>> >>>>> Could you try the following command to get vcpu stats? >>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=0 >>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=1 >>>> >>>> best >>>> Neel >>>> >>>> >>>> Dan >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, 29 Mar 2013, Dan Mack wrote: >>>>> >>>>> On Thu, 28 Mar 2013, Peter Grehan wrote: >>>>> >>>>>> >>>>>> Hi Dan, >>>>>> >>>>>>> >>>>>>> Any ideas? >>>>>>> >>>>>>>> >>>>>>>> >>>>>>> Just to confirm: your host system is an i7 3930k, which looks like a 6 >>>>>>> core system. Should be plenty beefy enough for a 2 vCPU guest. >>>>>>> >>>>>>> >>>>>> That's what I thought, 6 cores, 12 threads, 64 GB RAM. >>>>>> >>>>>> The issue you are seeing looks a bit like what happens when systems >>>>>> are >>>>>> >>>>>>> heavily oversubscribed and vCPUs can't get enough run time. >>>>>>> >>>>>>> >>>>>> I was running in on an idle system; only bhyve was running. >>>>>> >>>>>> >>>>>> How much spare RAM is on the host when you started the vm ? Also, is >>>>>> >>>>>>> there much running on the host when the guest is started ? >>>>>>> >>>>>>> >>>>>> 64GB >>>>>> >>>>>> Here's some more information (sorry for the formatting): >>>>>> >>>>>> Case 1: single vCPU via: >>>>>> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 >>>>>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>>>> >>>>>> Top shows it idle (note low load, 4GB memory given to guest, still 53GB >>>>>> free RAM, nothing else clocking CPU cycles): >>>>>> >>>>>> root@olive:~ # top -bH >>>>>> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 >>>>>> 14:11:51 >>>>>> 24 processes: 1 running, 23 sleeping >>>>>> >>>>>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free >>>>>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other >>>>>> Swap: >>>>>> >>>>>> >>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>>>>> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% >>>>>> bhyve{coco vcpu 0} >>>>>> 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd >>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>>> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% >>>>>> bhyve{coco mevent} >>>>>> >>>>>> >>>>>> So, case #1 all is normal. I was able to build a kernel with very good >>>>>> performance. >>>>>> >>>>>> The boot sequence looks like the following: >>>>>> >>>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>>> Launching virtual machine "coco" ... >>>>>> Consoles: userboot >>>>>> >>>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08] >>>>>> >>>>>> / >>>>>> ______ ____ _____ _____ >>>>>> | ____| | _ \ / ____| __ \ >>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>>> | | | | | | || | | | >>>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>>> ` >>>>>> s` `.....---.......--.``` >>>>>> -/ >>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>>> +. >>>>>> | | yo`:. :o >>>>>> `+- >>>>>> | 1. Boot Multi User [Enter] | y/ -/` >>>>>> -o/ >>>>>> | 2. Boot [S]ingle User | .- >>>>>> ::/sy+:. >>>>>> | 3. [Esc]ape to loader prompt | / `-- >>>>>> / >>>>>> | 4. Reboot | `: >>>>>> :` >>>>>> | | `: >>>>>> :` >>>>>> | Options: | / >>>>>> / >>>>>> | 5. Configure Boot [O]ptions... | .- >>>>>> -. >>>>>> | | -- >>>>>> -. >>>>>> | | `:` `:` >>>>>> | | .-- `--. >>>>>> | | .---.....----. >>>>>> +-----------------------------****------------+ >>>>>> >>>>>> >>>>>> >>>>>> Booting... >>>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >>>>>> 1994 >>>>>> The Regents of the University of California. All rights >>>>>> reserved. >>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>>> root@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64 >>>>>> >>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class >>>>>> CPU) >>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>>> Stepping = 7 >>>>>> Features=0x8fa3ab7f>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE> >>>>>> Features2=0x83bee217>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,** >>>>>> TSCDLT,**AESNI,HV> >>>>>> AMD Features=0x2c100800 >>>>>> >>>>>> AMD Features2=0x1 >>>>>> TSC: P-state invariant >>>>>> real memory = 5368709120 (5120 MB) >>>>>> avail memory = 3957534720 (3774 MB) >>>>>> Event timer "LAPIC" quality 400 >>>>>> ACPI APIC Table: >>>>>> random device not loaded; using insecure entropy >>>>>> ioapic0: Changing APIC ID to 1 >>>>>> ioapic0 irqs 0-16 on motherboard >>>>>> kbd0 at kbdmux0 >>>>>> acpi0: on motherboard >>>>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>>> pcib0: port 0xcf8-0xcff on acpi0 >>>>>> pci0: on pcib0 >>>>>> pcib0: no PRT entry for 0.31.INTA >>>>>> virtio_pci0: port 0x2000-0x201f mem >>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>>> vtnet0: on virtio_pci0 >>>>>> virtio_pci0: host features: 0x18020 >>>>>> virtio_pci0: negotiated features: 0x18020 >>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>>> virtio_pci1: port 0x2040-0x207f mem >>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>>> vtblk0: on virtio_pci1 >>>>>> virtio_pci1: host features: 0x10000004 >>>>>> virtio_pci1: negotiated features: 0x10000004 >>>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>>>> pci0 >>>>>> uart2: console (9600,n,8,1) >>>>>> Timecounters tick every 10.000 msec >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 >>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf. >>>>>> >>>>>> Setting hostid: 0x24bfd025. >>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>>> Starting file system checks: >>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% >>>>>> fragmentation) >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> Mounting local file systems:. >>>>>> Writing entropy file:. >>>>>> Setting hostname: cocopuff.example.com. >>>>>> vtnet0: link state changed to UP >>>>>> Starting Network: lo0 vtnet0. >>>>>> lo0: flags=8049 metric 0 mtu 16384 >>>>>> options=600003 >>>>>> >>>>>> inet6 ::1 prefixlen 128 >>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>>> inet 127.0.0.1 netmask 0xff000000 >>>>>> nd6 options=21 >>>>>> vtnet0: flags=8943>>>>> MULTICAST> >>>>>> metric 0 mtu 1500 >>>>>> options=80028 >>>>>> >>>>>> ether 00:a0:98:ac:0d:df >>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>>>> nd6 options=29 >>>>>> >>>>>> media: Ethernet 1000baseT >>>>>> status: active >>>>>> Starting devd. >>>>>> add net default: gateway 172.22.22.252 >>>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>>> add net ::0.0.0.0: gateway ::1 >>>>>> add net fe80::: gateway ::1 >>>>>> add net ff02::: gateway ::1 >>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>>> Creating and/or trimming log files. >>>>>> Starting syslogd. >>>>>> Clearing /tmp (X related). >>>>>> Updating motd:. >>>>>> Performing sanity check on sshd configuration. >>>>>> Starting sshd. >>>>>> Starting cron. >>>>>> Starting background file system checks in 60 seconds. >>>>>> >>>>>> Fri Mar 29 14:09:20 CDT 2013 >>>>>> >>>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>>> >>>>>> login: >>>>>> >>>>>> >>>>>> ====== Next, boot the same image with two (2) vCPU: >>>>>> >>>>>> Command run by vmrun.sh: >>>>>> >>>>>> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 >>>>>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>>>> >>>>>> Top shows it running out of control: >>>>>> >>>>>> root@olive:~ # top -bH >>>>>> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 >>>>>> 14:17:32 >>>>>> 25 processes: 3 running, 22 sleeping >>>>>> >>>>>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free >>>>>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M >>>>>> Other >>>>>> Swap: >>>>>> >>>>>> >>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>>>>> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% >>>>>> bhyve{coco vcpu 1} >>>>>> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% >>>>>> bhyve{coco vcpu 0} >>>>>> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd >>>>>> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd >>>>>> 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd >>>>>> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron >>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>>> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% >>>>>> bhyve{coco mevent} >>>>>> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh >>>>>> >>>>>> Even though, it's just sitting at the login prompt: >>>>>> >>>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>>> Launching virtual machine "coco" ... >>>>>> Consoles: userboot >>>>>> >>>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08] >>>>>> >>>>>> / >>>>>> ______ ____ _____ _____ >>>>>> | ____| | _ \ / ____| __ \ >>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>>> | | | | | | || | | | >>>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>>> ` >>>>>> s` `.....---.......--.``` >>>>>> -/ >>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>>> +. >>>>>> | | yo`:. :o >>>>>> `+- >>>>>> | 1. Boot Multi User [Enter] | y/ -/` >>>>>> -o/ >>>>>> | 2. Boot [S]ingle User | .- >>>>>> ::/sy+:. >>>>>> | 3. [Esc]ape to loader prompt | / `-- >>>>>> / >>>>>> | 4. Reboot | `: >>>>>> :` >>>>>> | | `: >>>>>> :` >>>>>> | Options: | / >>>>>> / >>>>>> | 5. Configure Boot [O]ptions... | .- >>>>>> -. >>>>>> | | -- >>>>>> -. >>>>>> | | `:` `:` >>>>>> | | .-- `--. >>>>>> | | .---.....----. >>>>>> +-----------------------------****------------+ >>>>>> >>>>>> >>>>>> >>>>>> Booting... >>>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >>>>>> 1994 >>>>>> The Regents of the University of California. All rights >>>>>> reserved. >>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>>> root@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64 >>>>>> >>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class >>>>>> CPU) >>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>>> Stepping = 7 >>>>>> Features=0x8fa3ab7f>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE> >>>>>> Features2=0x83bee217>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,** >>>>>> TSCDLT,**AESNI,HV> >>>>>> AMD Features=0x2c100800 >>>>>> >>>>>> AMD Features2=0x1 >>>>>> TSC: P-state invariant >>>>>> real memory = 5368709120 (5120 MB) >>>>>> avail memory = 3957534720 (3774 MB) >>>>>> Event timer "LAPIC" quality 400 >>>>>> ACPI APIC Table: >>>>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >>>>>> FreeBSD/SMP: 2 package(s) x 1 core(s) >>>>>> cpu0 (BSP): APIC ID: 0 >>>>>> cpu1 (AP): APIC ID: 1 >>>>>> random device not loaded; using insecure entropy >>>>>> ioapic0: Changing APIC ID to 2 >>>>>> ioapic0 irqs 0-16 on motherboard >>>>>> kbd0 at kbdmux0 >>>>>> acpi0: on motherboard >>>>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>>> pcib0: port 0xcf8-0xcff on acpi0 >>>>>> pci0: on pcib0 >>>>>> pcib0: no PRT entry for 0.31.INTA >>>>>> virtio_pci0: port 0x2000-0x201f mem >>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>>> vtnet0: on virtio_pci0 >>>>>> virtio_pci0: host features: 0x18020 >>>>>> virtio_pci0: negotiated features: 0x18020 >>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>>> virtio_pci1: port 0x2040-0x207f mem >>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>>> vtblk0: on virtio_pci1 >>>>>> virtio_pci1: host features: 0x10000004 >>>>>> virtio_pci1: negotiated features: 0x10000004 >>>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>>>> pci0 >>>>>> uart2: console (9600,n,8,1) >>>>>> Timecounters tick every 10.000 msec >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> SMP: AP CPU #1 Launched! >>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf. >>>>>> >>>>>> Setting hostid: 0x24bfd025. >>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>>> Starting file system checks: >>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% >>>>>> fragmentation) >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>> Mounting local file systems:. >>>>>> Writing entropy file:. >>>>>> Setting hostname: cocopuff.example.com. >>>>>> vtnet0: link state changed to UP >>>>>> Starting Network: lo0 vtnet0. >>>>>> lo0: flags=8049 metric 0 mtu 16384 >>>>>> options=600003 >>>>>> >>>>>> inet6 ::1 prefixlen 128 >>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>>> inet 127.0.0.1 netmask 0xff000000 >>>>>> nd6 options=21 >>>>>> vtnet0: flags=8943>>>>> MULTICAST> >>>>>> metric 0 mtu 1500 >>>>>> options=80028 >>>>>> >>>>>> ether 00:a0:98:ac:0d:df >>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>>>> nd6 options=29 >>>>>> >>>>>> media: Ethernet 1000baseT >>>>>> status: active >>>>>> Starting devd. >>>>>> add net default: gateway 172.22.22.252 >>>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>>> add net ::0.0.0.0: gateway ::1 >>>>>> add net fe80::: gateway ::1 >>>>>> add net ff02::: gateway ::1 >>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>>> Creating and/or trimming log files. >>>>>> Starting syslogd. >>>>>> syslogd: timed out waiting for child >>>>>> /etc/rc: WARNING: failed to start syslogd >>>>>> Clearing /tmp (X related). >>>>>> Updating motd:. >>>>>> Performing sanity check on sshd configuration. >>>>>> Starting sshd. >>>>>> Starting cron. >>>>>> Starting background file system checks in 60 seconds. >>>>>> >>>>>> Thu Apr 18 02:06:43 CDT 2013 >>>>>> >>>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>>> >>>>>> login: >>>>>> >>>>>> >>>>>> ========== end of boot sequence >>>>>> >>>>>> So, something is broken with SMP on the guest, I'd be happy to help >>>>>> just >>>>>> let me know. >>>>>> >>>>>> Dan >>>>>> >>>>>> ______________________________****_________________ >>>>>> freebsd-virtualization@**freeb**sd.org < >>>>>> freebsd-virtualization@**freebsd.org>mailing >>>>>> list >>>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-**** >>>>>> virtualization >>>>>> >>>>>>> >>>>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>>>> unsubscribe@freebsd.org >>>>> unsubscribe@freebsd.org >>>>>>> " >>>>>> >>>>>> ______________________________****_________________ >>>>>> >>>>> freebsd-virtualization@**freeb**sd.org < >>>>> freebsd-virtualization@**freebsd.org>mailing >>>>> list >>>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-****virtualization >>>>> >>>>>> >>>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>>> unsubscribe@freebsd.org >>>> unsubscribe@freebsd.org >>>>>> " >>>>> >>>>> >>>> >> > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscribe@freebsd.org" >