From owner-freebsd-virtualization@FreeBSD.ORG Sun Mar 31 04:03:49 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 59FB87A7 for ; Sun, 31 Mar 2013 04:03:49 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-ie0-x229.google.com (mail-ie0-x229.google.com [IPv6:2607:f8b0:4001:c03::229]) by mx1.freebsd.org (Postfix) with ESMTP id 1DC1E8A6 for ; Sun, 31 Mar 2013 04:03:49 +0000 (UTC) Received: by mail-ie0-f169.google.com with SMTP id qd14so1562089ieb.28 for ; Sat, 30 Mar 2013 21:03:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=iPtmAf7DhdubezAzbFR89VItpHQjebW4/0zFlbCUpuI=; b=ezB9im/A1AnWumUQhXXlkpITzfxB8fFYdsz7ogdUstVhUPt1CUpKQAPqucp4hlZIkG ASvtg+m4gikd3JO4V7kTb40YSggor92WXR/Yc8zDiMg/2VwxJksGLU8GYIci7/h3Nvo8 LjwiDX5OoppBCB8Bxt1ODR2Wz3IfteraPMJ+frCDRZcTn98kyan8CbI4P6yyolmIWJI1 4Cv4/r4Fgr0hWLcKzCj5SPkgluaTtL38FS5hHVml9HxPYmVmrjmReix5uuK0vG64zzaL ybwzQqkQo2u9Ig2NTll3kLA9Y1nS9ZYaSFA4z91+ycP54ONmGwE8CHX5g51Ys5cbcx99 mLkw== MIME-Version: 1.0 X-Received: by 10.42.145.137 with SMTP id f9mr3529427icv.52.1364702628690; Sat, 30 Mar 2013 21:03:48 -0700 (PDT) Received: by 10.42.23.132 with HTTP; Sat, 30 Mar 2013 21:03:48 -0700 (PDT) In-Reply-To: <20130330141141.B17063@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> <20130329172023.L15559@coco.macktronics.com> <20130330125954.G17063@coco.macktronics.com> <20130330141141.B17063@coco.macktronics.com> Date: Sat, 30 Mar 2013 21:03:48 -0700 Message-ID: Subject: Re: bhyve tty / login problems / panic From: Neel Natu To: Dan Mack Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 Mar 2013 04:03:49 -0000 Hi Dan, On Sat, Mar 30, 2013 at 12:17 PM, Dan Mack wrote: > Here you go: > > - uni-processor output: > > > root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 > vcpu0 > vm exits due to external interrupt 1330 > number of times hlt was intercepted 676 > number of times %cr access was intercepted 1 > number of times rdmsr was intercepted 0 > number of times wrmsr was intercepted 4 > number of monitor trap exits 0 > number of times pause was intercepted 428499 > vm exits due to interrupt window opening 1231 > vm exits due to nmi window opening 0 > number of times in/out was intercepted 423985 > number of times cpuid was intercepted 22 > vm exits due to nested page fault 7523 > number of vm exits for unknown reason 0 > number of times astpending at exit 38 > number of vm exits handled in userspace 853290 > > number of NMIs delivered to vcpu 0 > vcpu total runtime 6954314530 > number of ticks vcpu was idle 59572 > vcpu migration across host cpus 29 > > number of times hlt was ignored 0 > total number of vm exits 863271 > > And here is when running with vcpu==2: > > > root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 > vcpu0 > vm exits due to external interrupt 19234 > number of times hlt was intercepted 1965 > number of times %cr access was intercepted 1 > number of times rdmsr was intercepted 0 > number of times wrmsr was intercepted 4 > number of monitor trap exits 0 > number of times pause was intercepted 3907709 > vm exits due to interrupt window opening 8105 > vm exits due to nmi window opening 0 > number of times in/out was intercepted 457998 > number of times cpuid was intercepted 50 > vm exits due to nested page fault 940909 > number of vm exits for unknown reason 0 > number of times astpending at exit 102 > number of vm exits handled in userspace 4367825 > > number of NMIs delivered to vcpu 0 > vcpu total runtime 51834598990 > number of ticks vcpu was idle 887 > vcpu migration across host cpus 25 > > number of times hlt was ignored 0 > total number of vm exits 5335975 > > root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 > vcpu1 > vm exits due to external interrupt 51249 > number of times hlt was intercepted 3676 > number of times %cr access was intercepted 2 > number of times rdmsr was intercepted 0 > number of times wrmsr was intercepted 4 > number of monitor trap exits 0 > number of times pause was intercepted 4667527 > vm exits due to interrupt window opening 32453 > vm exits due to nmi window opening 0 > number of times in/out was intercepted 57278 > number of times cpuid was intercepted 63 > vm exits due to nested page fault 266091 > number of vm exits for unknown reason 0 > number of times astpending at exit 164 > number of vm exits handled in userspace 4728481 > > number of NMIs delivered to vcpu 0 > vcpu total runtime 53833466076 > number of ticks vcpu was idle 965 > vcpu migration across host cpus 16 > > number of times hlt was ignored 0 > total number of vm exits 5078343 > > The panic comes eventually like this: > > login: panic: deadlkres: possible deadlock detected for > 0xfffffe0090fb5490, blocked for 217798 ticks > > cpuid = 0 > Uptime: 255d6h29m56s > Automatic reboot in 15 seconds - press a key on the console to abort > > Thanks for capturing that. In the SMP case the aberrant source of VM exits is the nested page table faults - which means local apic accesses since these are being entirely handled in the kernel. They are an order of magnitude more than in the UP case - even after adjusting for the difference in the run times in the two cases. Also, the uptime message (255d6h29m56s) is indicates something funky happening with the time keeping. I have added one more stat to vmm.ko that counts the number of times a local apic timer interrupt is generated for each vcpu. Also, I would like to install your release.iso if you are able to upload it. I very much doubt, it but want to rule out those specific bits as the source of the problem. Thanks again for helping to debug this. best Neel > > Hope this helps, and FYI, here's a dmesg from the host: > > > Copyright (c) 1992-2013 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 10.0-CURRENT #7 r248935: Sat Mar 30 13:19:43 CDT 2013 > root@olive.example.com:/usr/**obj/usr/src/sys/MACKGEN amd64 > > FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 > CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3200.19-MHz K8-class CPU) > > Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d > Stepping = 7 > > Features=0xbfebfbff APIC,SEP,MTRR,PGE,MCA,CMOV,**PAT,PSE36,CLFLUSH,DTS,ACPI,** > MMX,FXSR,SSE,SSE2,SS,HTT,TM,**PBE> > > Features2=0x1fbee3bf VMX,EST,TM2,SSSE3,CX16,xTPR,**PDCM,PCID,DCA,SSE4.1,SSE4.2,** > x2APIC,POPCNT,TSCDLT,AESNI,**XSAVE,OSXSAVE,AVX> > AMD Features=0x2c100800 > AMD Features2=0x1 > TSC: P-state invariant, performance statistics > real memory = 68719476736 (65536 MB) > avail memory = 63123800064 (60199 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs > FreeBSD/SMP: 1 package(s) x 6 core(s) x 2 SMT threads > > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 2 > cpu3 (AP): APIC ID: 3 > cpu4 (AP): APIC ID: 4 > cpu5 (AP): APIC ID: 5 > cpu6 (AP): APIC ID: 6 > cpu7 (AP): APIC ID: 7 > cpu8 (AP): APIC ID: 8 > cpu9 (AP): APIC ID: 9 > cpu10 (AP): APIC ID: 10 > cpu11 (AP): APIC ID: 11 > ioapic1 irqs 24-47 on motherboard > ioapic0 irqs 0-23 on motherboard > lapic0: Forcing LINT1 to edge trigger > kbd1 at kbdmux0 > acpi0: on motherboard > acpi0: Power Button (fixed) > acpi0: reservation of f0000000, 18000000 (3) failed > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > cpu8: on acpi0 > cpu9: on acpi0 > cpu10: on acpi0 > cpu11: on acpi0 > atrtc0: port 0x70-0x71,0x74-0x77 irq 8 on acpi0 > > Event timer "RTC" frequency 32768 Hz quality 0 > attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > hpet0: iomem 0xfed00000-0xfed03fff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 950 > Event timer "HPET" frequency 14318180 Hz quality 350 > Event timer "HPET1" frequency 14318180 Hz quality 340 > Event timer "HPET2" frequency 14318180 Hz quality 340 > Event timer "HPET3" frequency 14318180 Hz quality 340 > Event timer "HPET4" frequency 14318180 Hz quality 340 > Event timer "HPET5" frequency 14318180 Hz quality 340 > Event timer "HPET6" frequency 14318180 Hz quality 340 > Event timer "HPET7" frequency 14318180 Hz quality 340 > > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > acpi_button0: on acpi0 > > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: at device 1.0 on pci0 > pci1: on pcib1 > pcib2: at device 1.1 on pci0 > pci2: on pcib2 > xhci0: mem 0xe3400000-0xe3401fff irq 16 > at device 0.0 on pci2 > usbus0: waiting for BIOS to give up control > xhci0: 32 byte context size. > usbus0 on xhci0 > pcib3: at device 2.0 on pci0 > pci3: on pcib3 > vgapci0: port 0x3000-0x307f mem > 0xe2000000-0xe2ffffff,**0xd0000000-0xdfffffff,**0xe0000000-0xe1ffffff irq > 16 at device 0.0 on pci3 > pci3: at device 0.1 (no driver attached) > pcib4: at device 3.0 on pci0 > pci4: on pcib4 > pci0: at device 5.0 (no driver attached) > pci0: at device 5.2 (no driver attached) > pcib5: at device 17.0 on pci0 > pci5: on pcib5 > pci0: at device 22.0 (no driver attached) > em0: port 0x4040-0x405f mem > 0xe3500000-0xe351ffff,**0xe3521000-0xe3521fff irq 20 at device 25.0 on > pci0 > em0: Using an MSI interrupt > em0: Ethernet address: 4c:72:b9:d1:b2:d2 > ehci0: mem 0xe3601000-0xe36013ff irq > 16 at device 26.0 on pci0 > usbus1: EHCI version 1.0 > usbus1 on ehci0 > pcib6: at device 28.0 on pci0 > pci6: on pcib6 > pcib7: at device 28.2 on pci0 > pci7: on pcib7 > pci7: at device 0.0 (no driver attached) > pcib8: at device 28.3 on pci0 > pci8: on pcib8 > xhci1: mem 0xe3200000-0xe3201fff irq 19 > at device 0.0 on pci8 > usbus2: waiting for BIOS to give up control > xhci1: 32 byte context size. > usbus2 on xhci1 > pcib9: at device 28.5 on pci0 > pci9: on pcib9 > em1: port 0x1000-0x101f mem > 0xe3100000-0xe311ffff,**0xe3120000-0xe3123fff irq 17 at device 0.0 on pci9 > em1: Using MSIX interrupts with 3 vectors > em1: Ethernet address: 4c:72:b9:d1:b2:d3 > ehci1: mem 0xe3602000-0xe36023ff irq > 23 at device 29.0 on pci0 > usbus3: EHCI version 1.0 > usbus3 on ehci1 > pcib10: at device 30.0 on pci0 > pci10: on pcib10 > isab0: at device 31.0 on pci0 > isa0: on isab0 > ahci0: port > 0x4068-0x406f,0x4074-0x4077,**0x4060-0x4067,0x4070-0x4073,**0x4020-0x403f > mem 0xe3523000-0xe35237ff irq 18 at device 31.2 on pci0 > ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported > ahcich0: at channel 0 on ahci0 > ahcich1: at channel 1 on ahci0 > ahcich2: at channel 2 on ahci0 > ahcich3: at channel 3 on ahci0 > ahcich4: at channel 4 on ahci0 > ahcich5: at channel 5 on ahci0 > ahciem0: on ahci0 > pci0: at device 31.3 (no driver attached) > orm0: at iomem 0xd1000-0xd1fff,0xd2000-**0xd2fff on isa0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > atkbdc0: at port 0x60,0x64 on isa0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > ppc0: cannot reserve I/O port range > est0: on cpu0 > p4tcc0: on cpu0 > est1: on cpu1 > p4tcc1: on cpu1 > est2: on cpu2 > p4tcc2: on cpu2 > est3: on cpu3 > p4tcc3: on cpu3 > est4: on cpu4 > p4tcc4: on cpu4 > est5: on cpu5 > p4tcc5: on cpu5 > est6: on cpu6 > p4tcc6: on cpu6 > est7: on cpu7 > p4tcc7: on cpu7 > est8: on cpu8 > p4tcc8: on cpu8 > est9: on cpu9 > p4tcc9: on cpu9 > est10: on cpu10 > p4tcc10: on cpu10 > est11: on cpu11 > p4tcc11: on cpu11 > ZFS filesystem version: 5 > ZFS storage pool version: features support (5000) > Timecounters tick every 1.000 msec > usbus0: 5.0Gbps Super Speed USB v3.0 > usbus1: 480Mbps High Speed USB v2.0 > usbus2: 5.0Gbps Super Speed USB v3.0 > usbus3: 480Mbps High Speed USB v2.0 > ugen1.1: at usbus1 > uhub0: on usbus1 > ugen0.1: <0x1033> at usbus0 > uhub1: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 > ugen3.1: at usbus3 > uhub2: on usbus3 > ugen2.1: <0x1033> at usbus2 > uhub3: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus2 > ses0 at ahciem0 bus 0 scbus6 target 0 lun 0 > ses0: SEMB S-E-S 2.00 device > ses0: SEMB SES Device > ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 > ada0: ATA-9 SATA 3.x device > ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) > ada0: Command Queueing enabled > ada0: 122104MB (250069680 512 byte sectors: 16H 63S/T 16383C) > ada0: Previously was known as ad4 > ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 > ada1: ATA-9 SATA 3.x device > ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) > ada1: Command Queueing enabled > ada1: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C) > ada1: Previously was known as ad6 > ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 > ada2: ATA-8 SATA 3.x device > ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada2: Command Queueing enabled > ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) > ada2: Previously was known as ad8 > ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 > ada3: ATA-8 SATA 3.x device > ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada3: Command Queueing enabled > ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) > ada3: Previously was known as ad10 > ada4 at ahcich4 bus 0 scbus4 target 0 lun 0 > ada4: ATA-8 SATA 3.x device > ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada4: Command Queueing enabled > ada4: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) > ada4: Previously was known as ad12 > ada5 at ahcich5 bus 0 scbus5 target 0 lun 0 > ada5: ATA-8 SATA 3.x device > ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) > ada5: Command Queueing enabled > ada5: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) > ada5: Previously was known as ad14 > lapic1: Forcing LINT1 to edge trigger > > SMP: AP CPU #1 Launched! > lapic4: Forcing LINT1 to edge trigger > SMP: AP CPU #4 Launched! > lapic11: Forcing LINT1 to edge trigger > SMP: AP CPU #11 Launched! > lapic7: Forcing LINT1 to edge trigger > SMP: AP CPU #7 Launched! > lapic5: Forcing LINT1 to edge trigger > SMP: AP CPU #5 Launched! > lapic9: Forcing LINT1 to edge trigger > SMP: AP CPU #9 Launched! > lapic2: Forcing LINT1 to edge trigger > SMP: AP CPU #2 Launched! > lapic3: Forcing LINT1 to edge trigger > SMP: AP CPU #3 Launched! > lapic10: Forcing LINT1 to edge trigger > SMP: AP CPU #10 Launched! > lapic6: Forcing LINT1 to edge trigger > SMP: AP CPU #6 Launched! > lapic8: Forcing LINT1 to edge trigger > SMP: AP CPU #8 Launched! > uhub3: 4 ports with 4 removable, self powered > uhub1: 4 ports with 4 removable, self powered > Root mount waiting for: usbus3 usbus1 > uhub0: 2 ports with 2 removable, self powered > uhub2: 2 ports with 2 removable, self powered > Root mount waiting for: usbus3 usbus1 > ugen1.2: at usbus1 > uhub4: on > usbus1 > ugen3.2: at usbus3 > uhub5: on > usbus3 > Root mount waiting for: usbus3 usbus1 > uhub4: 6 ports with 6 removable, self powered > uhub5: 8 ports with 8 removable, self powered > ugen1.3: at usbus1 > ukbd0: on > usbus1 > kbd2 at ukbd0 > ugen3.3: at usbus3 > Root mount waiting for: usbus3 > ugen3.4: at usbus3 > Trying to mount root from zfs:tron []... > ums0: on > usbus1 > ums0: 5 buttons and [XYZ] coordinates ID=0 > run0: <1.0> on usbus3 > run0: MAC/BBP RT3070 (rev 0x0201), RF RT3020 (MIMO 1T1R), address > 94:db:c9:e3:6e:c8 > ubt0: > on usbus3 > WARNING: attempt to domain_add(bluetooth) after domainfinalize() > WARNING: attempt to domain_add(netgraph) after domainfinalize() > tap0: Ethernet address: 00:bd:8c:54:07:00 > tap0: link state changed to UP > bridge0: Ethernet address: 02:4c:89:ce:33:00 > tap0: promiscuous mode enabled > bridge0: link state changed to UP > em1: link state changed to DOWN > em1: promiscuous mode enabled > em1: link state changed to UP > tap0: link state changed to DOWN > tap0: link state changed to UP > tap0: link state changed to DOWN > tap0: link state changed to UP > > > > >> Yep, no problem. I saw the checkin and started a rebuild a few minutes >> ago :-) >> >> Thanks, >> >> Dan >> >> On Sat, 30 Mar 2013, Neel Natu wrote: >> >> Hi Dan, >>> >>> On Fri, Mar 29, 2013 at 3:20 PM, Dan Mack wrote: >>> >>> >>>> >>>> Here you go Neel: >>>> >>>> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 >>>> vcpu0 >>>> vm exits due to external interrupt 45051 >>>> number of times hlt was intercepted 2277 >>>> number of NMIs delivered to vcpu 0 >>>> vcpu total runtime 105001652482 >>>> number of ticks vcpu was idle 919 >>>> vcpu migration across host cpus 13 >>>> number of times hlt was ignored 0 >>>> total number of vm exits 9240925 >>>> root@olive:~ # >>>> root@olive:~ # >>>> root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 >>>> vcpu1 >>>> vm exits due to external interrupt 149431 >>>> number of times hlt was intercepted 4222 >>>> number of NMIs delivered to vcpu 0 >>>> vcpu total runtime 106876622528 >>>> number of ticks vcpu was idle 869 >>>> vcpu migration across host cpus 20 >>>> number of times hlt was ignored 0 >>>> total number of vm exits 8065909 >>>> >>>> >>>> That did not help a whole lot because we were not keeping track of all >>> the >>> reasons a vcpu could exit. >>> >>> I have fixed that in r248935: >>> http://svnweb.freebsd.org/**base?view=revision&revision=**248935 >>> >>> Do you mind updating your vmm.ko with the change and getting the stats >>> again? >>> >>> best >>> Neel >>> >>> >>> Dan >>>> >>>> >>>> >>>> On Fri, 29 Mar 2013, Neel Natu wrote: >>>> >>>> Hi Dan, >>>> >>>>> >>>>> On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack >>>>> wrote: >>>>> >>>>> >>>>> I ran the procsystime dtrace script on bhyve when it was chewing up >>>>>> all >>>>>> the CPU in vCPU=2 mode, and this is what I see for about 10s of >>>>>> runtime: >>>>>> >>>>>> root@olive:/usr/share/dtrace/******toolkit # ./procsystime -n bhyve >>>>>> -aT >>>>>> >>>>>> Tracing... Hit Ctrl-C to end... >>>>>> dtrace: 158536 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 207447 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 189205 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 164341 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 246307 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 187640 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 214771 dynamic variable drops with non-empty dirty list >>>>>> dtrace: 221265 dynamic variable drops with non-empty dirty list >>>>>> ^C >>>>>> dtrace: 243468 dynamic variable drops with non-empty dirty list >>>>>> >>>>>> Elapsed Times for processes bhyve, >>>>>> >>>>>> SYSCALL TIME (ns) >>>>>> _umtx_op 18349 >>>>>> writev 135712 >>>>>> preadv 16175267 >>>>>> pwritev 22924378 >>>>>> ioctl 4353897920 >>>>>> TOTAL: 4393151626 >>>>>> >>>>>> CPU Times for processes bhyve, >>>>>> >>>>>> SYSCALL TIME (ns) >>>>>> _umtx_op 8815 >>>>>> writev 103145 >>>>>> pwritev 10647023 >>>>>> preadv 15159976 >>>>>> ioctl 3943399889 >>>>>> TOTAL: 3969318848 >>>>>> >>>>>> Syscall Counts for processes bhyve, >>>>>> >>>>>> SYSCALL COUNT >>>>>> _umtx_op 17 >>>>>> writev 43 >>>>>> pwritev 639 >>>>>> preadv 1183 >>>>>> ioctl 652782 >>>>>> TOTAL: 654664 >>>>>> >>>>>> Not sure if that helps or not. Hotkernel shows the cpu in the >>>>>> kernel`acpi_cpu_c1 call most of the time, normal ? >>>>>> >>>>>> Dan >>>>>> >>>>>> root@olive:/usr/share/dtrace/******toolkit # ./hotkernel >>>>>> >>>>>> Sampling... Hit Ctrl-C to end. >>>>>> ^C >>>>>> FUNCTION COUNT PCNT >>>>>> zfs.ko`space_map_sync 1 0.0% >>>>>> kernel`syscall_thread_exit 1 0.0% >>>>>> kernel`should_yield 1 0.0% >>>>>> zfs.ko`lzjb_compress 1 0.0% >>>>>> zfs.ko`dbuf_rele 1 0.0% >>>>>> zfs.ko`dsl_pool_tempreserve_******space 1 >>>>>> 0.0% >>>>>> >>>>>> kernel`_bus_dmamap_load_buffer 1 0.0% >>>>>> kernel`cpu_idleclock 1 0.0% >>>>>> kernel`_rw_runlock_cookie 1 0.0% >>>>>> kernel`pmap_enter 1 0.0% >>>>>> kernel`__lockmgr_args 1 0.0% >>>>>> zfs.ko`vdev_geom_io_start 1 0.0% >>>>>> kernel`ahci_ch_intr 1 0.0% >>>>>> kernel`__rw_rlock 1 0.0% >>>>>> kernel`VOP_UNLOCK_APV 1 0.0% >>>>>> zfs.ko`arc_buf_thaw 1 0.0% >>>>>> kernel`atomic_set_int 1 0.0% >>>>>> kernel`g_io_schedule_down 1 0.0% >>>>>> kernel`__mtx_lock_flags 1 0.0% >>>>>> kernel`_callout_stop_safe 1 0.0% >>>>>> kernel`sched_idletd 1 0.0% >>>>>> kernel`_sx_slock 1 0.0% >>>>>> vmm.ko`lapic_set_intr 1 0.0% >>>>>> kernel`cpusetobj_ffs 1 0.0% >>>>>> zfs.ko`dbuf_hold 1 0.0% >>>>>> vmm.ko`vlapic_intr_accepted 1 0.0% >>>>>> zfs.ko`sa_attr_op 1 0.0% >>>>>> zfs.ko`dmu_buf_hold_array_by_******dnode 1 >>>>>> 0.0% >>>>>> >>>>>> zfs.ko`taskq_run_safe 1 0.0% >>>>>> kernel`vm_reserv_free_page 1 0.0% >>>>>> kernel`_sx_sunlock 1 0.0% >>>>>> kernel`vn_rlimit_fsize 1 0.0% >>>>>> zfs.ko`vdev_queue_io_to_issue 1 0.0% >>>>>> kernel`vn_write 1 0.0% >>>>>> kernel`lock_init 1 0.0% >>>>>> vmm.ko`vmx_resume 1 0.0% >>>>>> zfs.ko`zio_gang_tree_free 1 0.0% >>>>>> zfs.ko`dbuf_rele_and_unlock 1 0.0% >>>>>> kernel`mnt_vnode_next_active 1 0.0% >>>>>> kernel`vm_page_remove 1 0.0% >>>>>> zfs.ko`vdev_queue_deadline_******compare 1 >>>>>> 0.0% >>>>>> >>>>>> kernel`zone_free_item 1 0.0% >>>>>> vmm.ko`vmx_launch 1 0.0% >>>>>> kernel`m_pullup 1 0.0% >>>>>> kernel`pmap_change_wiring 1 0.0% >>>>>> kernel`sched_userret 1 0.0% >>>>>> zfs.ko`dbuf_write_done 1 0.0% >>>>>> zfs.ko`zio_execute 1 0.0% >>>>>> kernel`PHYS_TO_VM_PAGE 2 0.0% >>>>>> zfs.ko`avl_rotation 2 0.0% >>>>>> kernel`ehci_device_isoc_fs_******enter 2 >>>>>> 0.0% >>>>>> >>>>>> vmm.ko`vm_exitinfo 2 0.0% >>>>>> zfs.ko`zfs_freebsd_write 2 0.0% >>>>>> vmm.ko`vcpu_stats 2 0.0% >>>>>> kernel`atomic_add_long 2 0.0% >>>>>> vmm.ko`0xffffffff81 2 0.0% >>>>>> kernel`vm_reserv_alloc_page 2 0.0% >>>>>> kernel`pagezero 2 0.0% >>>>>> kernel`bzero 2 0.0% >>>>>> zfs.ko`fletcher_2_native 2 0.0% >>>>>> vmm.ko`vmx_getreg 2 0.0% >>>>>> vmm.ko`vm_guest_msrs 3 0.0% >>>>>> vmm.ko`vmx_setreg 3 0.0% >>>>>> vmm.ko`vm_get_register 3 0.0% >>>>>> kernel`sx_try_xlock_ 3 0.0% >>>>>> kernel`vm_radix_lookup_le 3 0.0% >>>>>> vmm.ko`lapic_pending_intr 4 0.0% >>>>>> kernel`syscall_thread_enter 4 0.0% >>>>>> kernel`cpu_set_syscall_retval 4 0.0% >>>>>> kernel`__mnt_vnode_next_all 4 0.0% >>>>>> kernel`vfs_stdsync 4 0.0% >>>>>> vmm.ko`vmm_fetch_instruction 5 0.0% >>>>>> vmm.ko`vlapic_op_mem_read 6 0.0% >>>>>> kernel`_sx_xunlock 7 0.0% >>>>>> vmm.ko`vmm_emulate_instruction 7 0.0% >>>>>> vmm.ko`vlapic_op_mem_write 7 0.0% >>>>>> kernel`__mtx_lock_spin_flags 8 0.0% >>>>>> vmm.ko`lapic_timer_tick 8 0.0% >>>>>> zfs.ko`list_next 8 0.0% >>>>>> zfs.ko`l2arc_feed_thread 8 0.0% >>>>>> kernel`userret 9 0.0% >>>>>> vmm.ko`vm_gpa2hpa 9 0.0% >>>>>> vmm.ko`vmm_decode_instruction 11 0.0% >>>>>> vmm.ko`ept_vmmmap_get 11 0.0% >>>>>> vmm.ko`vm_lapic 11 0.0% >>>>>> kernel`spinlock_enter 14 0.0% >>>>>> kernel`__mtx_lock_sleep 15 0.0% >>>>>> vmm.ko`vcpu_set_state 16 0.0% >>>>>> vmm.ko`vm_nmi_pending 16 0.0% >>>>>> kernel`malloc_type_zone_******allocated 16 >>>>>> 0.0% >>>>>> >>>>>> kernel`devfs_ioctl_f 16 0.0% >>>>>> kernel`pmap_kextract 17 0.0% >>>>>> kernel`sys_ioctl 18 0.0% >>>>>> kernel`malloc 21 0.0% >>>>>> kernel`cpu_fetch_syscall_args 24 0.0% >>>>>> kernel`uma_zfree_arg 28 0.0% >>>>>> vmm.ko`vlapic_pending_intr 29 0.0% >>>>>> kernel`trap 30 0.0% >>>>>> kernel`hpet_get_timecount 31 0.0% >>>>>> dtrace.ko`dtrace_trap 35 0.0% >>>>>> vmm.ko`vlapic_update_ppr 39 0.0% >>>>>> vmm.ko`vmmdev_ioctl 40 0.0% >>>>>> kernel`copyout 43 0.0% >>>>>> kernel`critical_enter 44 0.0% >>>>>> kernel`copyin 51 0.1% >>>>>> vmm.ko`vmcs_read 53 0.1% >>>>>> kernel`fpusave 61 0.1% >>>>>> kernel`uma_zalloc_arg 62 0.1% >>>>>> kernel`devvn_refthread 69 0.1% >>>>>> kernel`atomic_load_acq_int 70 0.1% >>>>>> kernel`fpurestore 70 0.1% >>>>>> vmm.ko`vlapic_timer_tick 75 0.1% >>>>>> kernel`amd64_syscall 77 0.1% >>>>>> kernel`bcopy 91 0.1% >>>>>> kernel`fpudna 103 0.1% >>>>>> kernel`free 103 0.1% >>>>>> kernel`critical_exit 105 0.1% >>>>>> kernel`dev_relthread 108 0.1% >>>>>> kernel`spinlock_exit 119 0.1% >>>>>> kernel`kern_ioctl 138 0.1% >>>>>> kernel`fpuexit 242 0.3% >>>>>> kernel`0xffffffff80 306 0.3% >>>>>> vmm.ko`restore_host_msrs 398 0.4% >>>>>> vmm.ko`vm_run 428 0.4% >>>>>> vmm.ko`restore_guest_msrs 433 0.5% >>>>>> kernel`cpu_idle 906 1.0% >>>>>> vmm.ko`vmx_run 9266 9.7% >>>>>> kernel`acpi_cpu_c1 81134 85.2% >>>>>> >>>>>> Hope this helps, let me know if you have any ideas on what to try >>>>>> next. >>>>>> >>>>>> >>>>>> Could you try the following command to get vcpu stats? >>>>>> >>>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=0 >>>>> sudo bhyvectl --vm=vm1 --get-stats --cpu=1 >>>>> >>>>> best >>>>> Neel >>>>> >>>>> >>>>> Dan >>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Fri, 29 Mar 2013, Dan Mack wrote: >>>>>> >>>>>> On Thu, 28 Mar 2013, Peter Grehan wrote: >>>>>> >>>>>> >>>>>>> Hi Dan, >>>>>>> >>>>>>> >>>>>>>> Any ideas? >>>>>>>> >>>>>>>> >>>>>>>>> >>>>>>>>> Just to confirm: your host system is an i7 3930k, which looks >>>>>>>> like a 6 >>>>>>>> core system. Should be plenty beefy enough for a 2 vCPU guest. >>>>>>>> >>>>>>>> >>>>>>>> That's what I thought, 6 cores, 12 threads, 64 GB RAM. >>>>>>> >>>>>>> The issue you are seeing looks a bit like what happens when systems >>>>>>> are >>>>>>> >>>>>>> heavily oversubscribed and vCPUs can't get enough run time. >>>>>>>> >>>>>>>> >>>>>>>> I was running in on an idle system; only bhyve was running. >>>>>>> >>>>>>> >>>>>>> How much spare RAM is on the host when you started the vm ? Also, is >>>>>>> >>>>>>> there much running on the host when the guest is started ? >>>>>>>> >>>>>>>> >>>>>>>> 64GB >>>>>>> >>>>>>> Here's some more information (sorry for the formatting): >>>>>>> >>>>>>> Case 1: single vCPU via: >>>>>>> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 >>>>>>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>>>>> >>>>>>> Top shows it idle (note low load, 4GB memory given to guest, still >>>>>>> 53GB >>>>>>> free RAM, nothing else clocking CPU cycles): >>>>>>> >>>>>>> root@olive:~ # top -bH >>>>>>> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 >>>>>>> 14:11:51 >>>>>>> 24 processes: 1 running, 23 sleeping >>>>>>> >>>>>>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free >>>>>>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other >>>>>>> Swap: >>>>>>> >>>>>>> >>>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU >>>>>>> COMMAND >>>>>>> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% >>>>>>> bhyve{coco vcpu 0} >>>>>>> 925 root 20 0 14296K 2044K select 11 0:00 0.00% >>>>>>> syslogd >>>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>>>> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% >>>>>>> bhyve{coco mevent} >>>>>>> >>>>>>> >>>>>>> So, case #1 all is normal. I was able to build a kernel with very >>>>>>> good >>>>>>> performance. >>>>>>> >>>>>>> The boot sequence looks like the following: >>>>>>> >>>>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>>>> Launching virtual machine "coco" ... >>>>>>> Consoles: userboot >>>>>>> >>>>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+******0x12bb08] >>>>>>> >>>>>>> / >>>>>>> ______ ____ _____ _____ >>>>>>> | ____| | _ \ / ____| __ \ >>>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>>>> | | | | | | || | | | >>>>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>>>> ` >>>>>>> s` `.....---.......--.``` >>>>>>> -/ >>>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>>>> +. >>>>>>> | | yo`:. :o >>>>>>> `+- >>>>>>> | 1. Boot Multi User [Enter] | y/ -/` >>>>>>> -o/ >>>>>>> | 2. Boot [S]ingle User | .- >>>>>>> ::/sy+:. >>>>>>> | 3. [Esc]ape to loader prompt | / >>>>>>> `-- >>>>>>> / >>>>>>> | 4. Reboot | `: >>>>>>> :` >>>>>>> | | `: >>>>>>> :` >>>>>>> | Options: | / >>>>>>> / >>>>>>> | 5. Configure Boot [O]ptions... | .- >>>>>>> -. >>>>>>> | | -- -. >>>>>>> | | `:` >>>>>>> `:` >>>>>>> | | .-- `--. >>>>>>> | | .---.....----. >>>>>>> +-----------------------------******------------+ >>>>>>> >>>>>>> >>>>>>> >>>>>>> Booting... >>>>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >>>>>>> 1994 >>>>>>> The Regents of the University of California. All rights >>>>>>> reserved. >>>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>>>> root@cocopuff.example.com:/******sys/amd64/compile/BHYVE amd64 >>>>>>> >>>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class >>>>>>> CPU) >>>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>>>> Stepping = 7 >>>>>>> Features=0x8fa3ab7f>>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,******MMX,FXSR,SSE,SSE2,SS,PBE> >>>>>>> Features2=0x83bee217>>>>>> **** >>>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.******1,SSE4.2,x2APIC,POPCNT,** >>>>>>> TSCDLT,**AESNI,HV> >>>>>>> AMD Features=0x2c100800 >>>>>>> >>>>>>> AMD Features2=0x1 >>>>>>> TSC: P-state invariant >>>>>>> real memory = 5368709120 (5120 MB) >>>>>>> avail memory = 3957534720 (3774 MB) >>>>>>> Event timer "LAPIC" quality 400 >>>>>>> ACPI APIC Table: >>>>>>> random device not loaded; using insecure entropy >>>>>>> ioapic0: Changing APIC ID to 1 >>>>>>> ioapic0 irqs 0-16 on motherboard >>>>>>> kbd0 at kbdmux0 >>>>>>> acpi0: on motherboard >>>>>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>>>> pcib0: port 0xcf8-0xcff on acpi0 >>>>>>> pci0: on pcib0 >>>>>>> pcib0: no PRT entry for 0.31.INTA >>>>>>> virtio_pci0: port 0x2000-0x201f mem >>>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>>>> vtnet0: on virtio_pci0 >>>>>>> virtio_pci0: host features: 0x18020 >>>>>>> virtio_pci0: negotiated features: 0x18020 >>>>>>> >>>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>>>> virtio_pci1: port 0x2040-0x207f mem >>>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>>>> vtblk0: on virtio_pci1 >>>>>>> virtio_pci1: host features: 0x10000004 >>>>>>> virtio_pci1: negotiated features: 0x10000004 >>>>>>> >>>>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>>>>> pci0 >>>>>>> uart2: console (9600,n,8,1) >>>>>>> Timecounters tick every 10.000 msec >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 >>>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-******00a098ac0ddf. >>>>>>> >>>>>>> Setting hostid: 0x24bfd025. >>>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>>>> Starting file system checks: >>>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>>>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% >>>>>>> fragmentation) >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> Mounting local file systems:. >>>>>>> Writing entropy file:. >>>>>>> Setting hostname: cocopuff.example.com. >>>>>>> vtnet0: link state changed to UP >>>>>>> Starting Network: lo0 vtnet0. >>>>>>> lo0: flags=8049 metric 0 mtu >>>>>>> 16384 >>>>>>> options=600003 >>>>>>> >>>>>>> inet6 ::1 prefixlen 128 >>>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>>>> inet 127.0.0.1 netmask 0xff000000 >>>>>>> nd6 options=21 >>>>>>> vtnet0: flags=8943>>>>>> MULTICAST> >>>>>>> metric 0 mtu 1500 >>>>>>> options=80028 >>>>>>> >>>>>>> ether 00:a0:98:ac:0d:df >>>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast >>>>>>> 172.22.255.255 >>>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>>>>> nd6 options=29 >>>>>>> >>>>>>> media: Ethernet 1000baseT >>>>>>> status: active >>>>>>> Starting devd. >>>>>>> add net default: gateway 172.22.22.252 >>>>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>>>> add net ::0.0.0.0: gateway ::1 >>>>>>> add net fe80::: gateway ::1 >>>>>>> add net ff02::: gateway ::1 >>>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>>>> Creating and/or trimming log files. >>>>>>> Starting syslogd. >>>>>>> Clearing /tmp (X related). >>>>>>> Updating motd:. >>>>>>> Performing sanity check on sshd configuration. >>>>>>> Starting sshd. >>>>>>> Starting cron. >>>>>>> Starting background file system checks in 60 seconds. >>>>>>> >>>>>>> Fri Mar 29 14:09:20 CDT 2013 >>>>>>> >>>>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>>>> >>>>>>> login: >>>>>>> >>>>>>> >>>>>>> ====== Next, boot the same image with two (2) vCPU: >>>>>>> >>>>>>> Command run by vmrun.sh: >>>>>>> >>>>>>> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 >>>>>>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>>>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>>>>> >>>>>>> Top shows it running out of control: >>>>>>> >>>>>>> root@olive:~ # top -bH >>>>>>> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 >>>>>>> 14:17:32 >>>>>>> 25 processes: 3 running, 22 sleeping >>>>>>> >>>>>>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free >>>>>>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M >>>>>>> Other >>>>>>> Swap: >>>>>>> >>>>>>> >>>>>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU >>>>>>> COMMAND >>>>>>> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% >>>>>>> bhyve{coco vcpu 1} >>>>>>> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% >>>>>>> bhyve{coco vcpu 0} >>>>>>> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd >>>>>>> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd >>>>>>> 925 root 20 0 14296K 2044K select 9 0:00 0.00% >>>>>>> syslogd >>>>>>> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron >>>>>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>>>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>>>>> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% >>>>>>> bhyve{coco mevent} >>>>>>> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh >>>>>>> >>>>>>> Even though, it's just sitting at the login prompt: >>>>>>> >>>>>>> root@olive:/vms/coco # ./vmrun.sh coco >>>>>>> Launching virtual machine "coco" ... >>>>>>> Consoles: userboot >>>>>>> >>>>>>> FreeBSD/amd64 User boot, Revision 1.1 >>>>>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>>>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>>>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+******0x12bb08] >>>>>>> >>>>>>> / >>>>>>> ______ ____ _____ _____ >>>>>>> | ____| | _ \ / ____| __ \ >>>>>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>>>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>>>>> | | | | | __/ __/| |_) |____) | |__| | >>>>>>> | | | | | | || | | | >>>>>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>>>>> ` >>>>>>> s` `.....---.......--.``` >>>>>>> -/ >>>>>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>>>>> +. >>>>>>> | | yo`:. :o >>>>>>> `+- >>>>>>> | 1. Boot Multi User [Enter] | y/ -/` >>>>>>> -o/ >>>>>>> | 2. Boot [S]ingle User | .- >>>>>>> ::/sy+:. >>>>>>> | 3. [Esc]ape to loader prompt | / >>>>>>> `-- >>>>>>> / >>>>>>> | 4. Reboot | `: >>>>>>> :` >>>>>>> | | `: >>>>>>> :` >>>>>>> | Options: | / >>>>>>> / >>>>>>> | 5. Configure Boot [O]ptions... | .- >>>>>>> -. >>>>>>> | | -- -. >>>>>>> | | `:` >>>>>>> `:` >>>>>>> | | .-- `--. >>>>>>> | | .---.....----. >>>>>>> +-----------------------------******------------+ >>>>>>> >>>>>>> >>>>>>> >>>>>>> Booting... >>>>>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>>>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >>>>>>> 1994 >>>>>>> The Regents of the University of California. All rights >>>>>>> reserved. >>>>>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>>>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>>>>> root@cocopuff.example.com:/******sys/amd64/compile/BHYVE amd64 >>>>>>> >>>>>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>>>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class >>>>>>> CPU) >>>>>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>>>>> Stepping = 7 >>>>>>> Features=0x8fa3ab7f>>>>>> SEP,PGE,CMOV,PAT,PSE36,DTS,******MMX,FXSR,SSE,SSE2,SS,PBE> >>>>>>> Features2=0x83bee217>>>>>> **** >>>>>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.******1,SSE4.2,x2APIC,POPCNT,** >>>>>>> TSCDLT,**AESNI,HV> >>>>>>> AMD Features=0x2c100800 >>>>>>> >>>>>>> AMD Features2=0x1 >>>>>>> TSC: P-state invariant >>>>>>> real memory = 5368709120 (5120 MB) >>>>>>> avail memory = 3957534720 (3774 MB) >>>>>>> Event timer "LAPIC" quality 400 >>>>>>> ACPI APIC Table: >>>>>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >>>>>>> FreeBSD/SMP: 2 package(s) x 1 core(s) >>>>>>> cpu0 (BSP): APIC ID: 0 >>>>>>> cpu1 (AP): APIC ID: 1 >>>>>>> random device not loaded; using insecure entropy >>>>>>> ioapic0: Changing APIC ID to 2 >>>>>>> ioapic0 irqs 0-16 on motherboard >>>>>>> kbd0 at kbdmux0 >>>>>>> acpi0: on motherboard >>>>>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>>>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>>>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>>>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>>>>> pcib0: port 0xcf8-0xcff on acpi0 >>>>>>> pci0: on pcib0 >>>>>>> pcib0: no PRT entry for 0.31.INTA >>>>>>> virtio_pci0: port 0x2000-0x201f mem >>>>>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>>>>> vtnet0: on virtio_pci0 >>>>>>> virtio_pci0: host features: 0x18020 >>>>>>> virtio_pci0: negotiated features: 0x18020 >>>>>>> >>>>>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>>>>> virtio_pci1: port 0x2040-0x207f mem >>>>>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>>>>> vtblk0: on virtio_pci1 >>>>>>> virtio_pci1: host features: 0x10000004 >>>>>>> virtio_pci1: negotiated features: 0x10000004 >>>>>>> >>>>>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>>>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>>>>> pci0 >>>>>>> uart2: console (9600,n,8,1) >>>>>>> Timecounters tick every 10.000 msec >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> SMP: AP CPU #1 Launched! >>>>>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>>>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-******00a098ac0ddf. >>>>>>> >>>>>>> Setting hostid: 0x24bfd025. >>>>>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>>>>> Starting file system checks: >>>>>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>>>>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% >>>>>>> fragmentation) >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>>>>> Mounting local file systems:. >>>>>>> Writing entropy file:. >>>>>>> Setting hostname: cocopuff.example.com. >>>>>>> vtnet0: link state changed to UP >>>>>>> Starting Network: lo0 vtnet0. >>>>>>> lo0: flags=8049 metric 0 mtu >>>>>>> 16384 >>>>>>> options=600003 >>>>>>> >>>>>>> inet6 ::1 prefixlen 128 >>>>>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>>>>> inet 127.0.0.1 netmask 0xff000000 >>>>>>> nd6 options=21 >>>>>>> vtnet0: flags=8943>>>>>> MULTICAST> >>>>>>> metric 0 mtu 1500 >>>>>>> options=80028 >>>>>>> >>>>>>> ether 00:a0:98:ac:0d:df >>>>>>> inet 172.22.222.230 netmask 0xffff0000 broadcast >>>>>>> 172.22.255.255 >>>>>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>>>>> nd6 options=29 >>>>>>> >>>>>>> media: Ethernet 1000baseT >>>>>>> status: active >>>>>>> Starting devd. >>>>>>> add net default: gateway 172.22.22.252 >>>>>>> add net ::ffff:0.0.0.0: gateway ::1 >>>>>>> add net ::0.0.0.0: gateway ::1 >>>>>>> add net fe80::: gateway ::1 >>>>>>> add net ff02::: gateway ::1 >>>>>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>>>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>>>>> Creating and/or trimming log files. >>>>>>> Starting syslogd. >>>>>>> syslogd: timed out waiting for child >>>>>>> /etc/rc: WARNING: failed to start syslogd >>>>>>> Clearing /tmp (X related). >>>>>>> Updating motd:. >>>>>>> Performing sanity check on sshd configuration. >>>>>>> Starting sshd. >>>>>>> Starting cron. >>>>>>> Starting background file system checks in 60 seconds. >>>>>>> >>>>>>> Thu Apr 18 02:06:43 CDT 2013 >>>>>>> >>>>>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>>>>> >>>>>>> login: >>>>>>> >>>>>>> >>>>>>> ========== end of boot sequence >>>>>>> >>>>>>> So, something is broken with SMP on the guest, I'd be happy to help >>>>>>> just >>>>>>> let me know. >>>>>>> >>>>>>> Dan >>>>>>> >>>>>>> ______________________________******_________________ >>>>>>> freebsd-virtualization@****freeb**sd.org < >>>>>>> freebsd-virtualization@**freeb**sd.org < >>>>>>> freebsd-virtualization@**freebsd.org >>>>>>> >>mailing >>>>>>> list >>>>>>> http://lists.freebsd.org/******mailman/listinfo/freebsd-**** >>>>>>> virtualization>>>>>> listinfo/freebsd-****virtualization >>>>>>> > >>>>>>> >>>>>> virtualization >>>>>>> >>>>>> freebsd-virtualization >>>>>>> > >>>>>>> >>>>>>>> >>>>>>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>>>>> unsubscribe@freebsd.org >>>>>> unsubscribe@freebsd.org>>>>>> freebsd.org > >>>>>>> >>>>>>>> " >>>>>>>> >>>>>>> >>>>>>> ______________________________******_________________ >>>>>>> >>>>>>> freebsd-virtualization@****freeb**sd.org < >>>>>> freebsd-virtualization@**freeb**sd.org < >>>>>> freebsd-virtualization@**freebsd.org >>>>>> >>mailing >>>>>> list >>>>>> http://lists.freebsd.org/******mailman/listinfo/freebsd-****** >>>>>> virtualization >>>>>> >>>>> virtualization >>>>>> > >>>>>> >>>>> virtualization >>>>>> >>>>>> > >>>>>> >>>>>>> >>>>>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>>>> unsubscribe@freebsd.org >>>>> unsubscribe@freebsd.org >>>>> unsubscribe@freebsd.org >>>>>> > >>>>>> >>>>>>> " >>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>> ______________________________**_________________ >> freebsd-virtualization@**freebsd.org mailing list >> http://lists.freebsd.org/**mailman/listinfo/freebsd-**virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-** >> unsubscribe@freebsd.org " >> >>