From owner-freebsd-virtualization@FreeBSD.ORG Sat Mar 30 17:50:45 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6513A22B; Sat, 30 Mar 2013 17:50:45 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-ia0-x232.google.com (mail-ia0-x232.google.com [IPv6:2607:f8b0:4001:c02::232]) by mx1.freebsd.org (Postfix) with ESMTP id 251952A2; Sat, 30 Mar 2013 17:50:45 +0000 (UTC) Received: by mail-ia0-f178.google.com with SMTP id r13so1053452iar.37 for ; Sat, 30 Mar 2013 10:50:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=y6W3M+oTMCUncM0vzoZAvXPjjy43/Z+KaVELrcBuDng=; b=huWpBzfqW5QJI0u2iZMC3EOjBKaa5T56kqFOBcs1YZGItuIANraGoZ0VXgF07RR5g2 FRwoMb+eMlOBar0aIY5OPVCeom9qz3gCW9EhzsNKA3Nm0N2JUZD4tEhV6K3yzRkBkNaM EkemXqMIKSoQj2NZ8QvTXJr2uRXucJVqPyBlmk6ylZpGQWJCtmwCmuhORLhIRkpNMwbG fcyxbakras9vNBnxsebVf2tUem4wEqGZ9tXtW+tLAvyNyhVFNwZPed3PiuGVpTMPsQgM ukPpfw2nHGHnaM86rXVaysGYHDgWPnqn8IUUiJ76Pj1n56YkckW3/I2gxWPbAKojhvds AufQ== MIME-Version: 1.0 X-Received: by 10.42.145.137 with SMTP id f9mr3107594icv.52.1364665843965; Sat, 30 Mar 2013 10:50:43 -0700 (PDT) Received: by 10.42.23.132 with HTTP; Sat, 30 Mar 2013 10:50:43 -0700 (PDT) In-Reply-To: <20130329172023.L15559@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> <20130329140714.L12104@coco.macktronics.com> <20130329143127.W12104@coco.macktronics.com> <20130329172023.L15559@coco.macktronics.com> Date: Sat, 30 Mar 2013 10:50:43 -0700 Message-ID: Subject: Re: bhyve tty / login problems / panic From: Neel Natu To: Dan Mack Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-virtualization@freebsd.org" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 30 Mar 2013 17:50:45 -0000 Hi Dan, On Fri, Mar 29, 2013 at 3:20 PM, Dan Mack wrote: > > > Here you go Neel: > > root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=0 > vcpu0 > vm exits due to external interrupt 45051 > number of times hlt was intercepted 2277 > number of NMIs delivered to vcpu 0 > vcpu total runtime 105001652482 > number of ticks vcpu was idle 919 > vcpu migration across host cpus 13 > number of times hlt was ignored 0 > total number of vm exits 9240925 > root@olive:~ # > root@olive:~ # > root@olive:~ # bhyvectl --vm=coco --get-stats --cpu=1 > vcpu1 > vm exits due to external interrupt 149431 > number of times hlt was intercepted 4222 > number of NMIs delivered to vcpu 0 > vcpu total runtime 106876622528 > number of ticks vcpu was idle 869 > vcpu migration across host cpus 20 > number of times hlt was ignored 0 > total number of vm exits 8065909 > > That did not help a whole lot because we were not keeping track of all the reasons a vcpu could exit. I have fixed that in r248935: http://svnweb.freebsd.org/base?view=revision&revision=248935 Do you mind updating your vmm.ko with the change and getting the stats again? best Neel > Dan > > > > On Fri, 29 Mar 2013, Neel Natu wrote: > > Hi Dan, >> >> On Fri, Mar 29, 2013 at 12:38 PM, Dan Mack wrote: >> >> >>> I ran the procsystime dtrace script on bhyve when it was chewing up all >>> the CPU in vCPU=2 mode, and this is what I see for about 10s of runtime: >>> >>> root@olive:/usr/share/dtrace/****toolkit # ./procsystime -n bhyve -aT >>> >>> Tracing... Hit Ctrl-C to end... >>> dtrace: 158536 dynamic variable drops with non-empty dirty list >>> dtrace: 207447 dynamic variable drops with non-empty dirty list >>> dtrace: 189205 dynamic variable drops with non-empty dirty list >>> dtrace: 164341 dynamic variable drops with non-empty dirty list >>> dtrace: 246307 dynamic variable drops with non-empty dirty list >>> dtrace: 187640 dynamic variable drops with non-empty dirty list >>> dtrace: 214771 dynamic variable drops with non-empty dirty list >>> dtrace: 221265 dynamic variable drops with non-empty dirty list >>> ^C >>> dtrace: 243468 dynamic variable drops with non-empty dirty list >>> >>> Elapsed Times for processes bhyve, >>> >>> SYSCALL TIME (ns) >>> _umtx_op 18349 >>> writev 135712 >>> preadv 16175267 >>> pwritev 22924378 >>> ioctl 4353897920 >>> TOTAL: 4393151626 >>> >>> CPU Times for processes bhyve, >>> >>> SYSCALL TIME (ns) >>> _umtx_op 8815 >>> writev 103145 >>> pwritev 10647023 >>> preadv 15159976 >>> ioctl 3943399889 >>> TOTAL: 3969318848 >>> >>> Syscall Counts for processes bhyve, >>> >>> SYSCALL COUNT >>> _umtx_op 17 >>> writev 43 >>> pwritev 639 >>> preadv 1183 >>> ioctl 652782 >>> TOTAL: 654664 >>> >>> Not sure if that helps or not. Hotkernel shows the cpu in the >>> kernel`acpi_cpu_c1 call most of the time, normal ? >>> >>> Dan >>> >>> root@olive:/usr/share/dtrace/****toolkit # ./hotkernel >>> >>> Sampling... Hit Ctrl-C to end. >>> ^C >>> FUNCTION COUNT PCNT >>> zfs.ko`space_map_sync 1 0.0% >>> kernel`syscall_thread_exit 1 0.0% >>> kernel`should_yield 1 0.0% >>> zfs.ko`lzjb_compress 1 0.0% >>> zfs.ko`dbuf_rele 1 0.0% >>> zfs.ko`dsl_pool_tempreserve_****space 1 0.0% >>> >>> kernel`_bus_dmamap_load_buffer 1 0.0% >>> kernel`cpu_idleclock 1 0.0% >>> kernel`_rw_runlock_cookie 1 0.0% >>> kernel`pmap_enter 1 0.0% >>> kernel`__lockmgr_args 1 0.0% >>> zfs.ko`vdev_geom_io_start 1 0.0% >>> kernel`ahci_ch_intr 1 0.0% >>> kernel`__rw_rlock 1 0.0% >>> kernel`VOP_UNLOCK_APV 1 0.0% >>> zfs.ko`arc_buf_thaw 1 0.0% >>> kernel`atomic_set_int 1 0.0% >>> kernel`g_io_schedule_down 1 0.0% >>> kernel`__mtx_lock_flags 1 0.0% >>> kernel`_callout_stop_safe 1 0.0% >>> kernel`sched_idletd 1 0.0% >>> kernel`_sx_slock 1 0.0% >>> vmm.ko`lapic_set_intr 1 0.0% >>> kernel`cpusetobj_ffs 1 0.0% >>> zfs.ko`dbuf_hold 1 0.0% >>> vmm.ko`vlapic_intr_accepted 1 0.0% >>> zfs.ko`sa_attr_op 1 0.0% >>> zfs.ko`dmu_buf_hold_array_by_****dnode 1 0.0% >>> >>> zfs.ko`taskq_run_safe 1 0.0% >>> kernel`vm_reserv_free_page 1 0.0% >>> kernel`_sx_sunlock 1 0.0% >>> kernel`vn_rlimit_fsize 1 0.0% >>> zfs.ko`vdev_queue_io_to_issue 1 0.0% >>> kernel`vn_write 1 0.0% >>> kernel`lock_init 1 0.0% >>> vmm.ko`vmx_resume 1 0.0% >>> zfs.ko`zio_gang_tree_free 1 0.0% >>> zfs.ko`dbuf_rele_and_unlock 1 0.0% >>> kernel`mnt_vnode_next_active 1 0.0% >>> kernel`vm_page_remove 1 0.0% >>> zfs.ko`vdev_queue_deadline_****compare 1 0.0% >>> >>> kernel`zone_free_item 1 0.0% >>> vmm.ko`vmx_launch 1 0.0% >>> kernel`m_pullup 1 0.0% >>> kernel`pmap_change_wiring 1 0.0% >>> kernel`sched_userret 1 0.0% >>> zfs.ko`dbuf_write_done 1 0.0% >>> zfs.ko`zio_execute 1 0.0% >>> kernel`PHYS_TO_VM_PAGE 2 0.0% >>> zfs.ko`avl_rotation 2 0.0% >>> kernel`ehci_device_isoc_fs_****enter 2 0.0% >>> >>> vmm.ko`vm_exitinfo 2 0.0% >>> zfs.ko`zfs_freebsd_write 2 0.0% >>> vmm.ko`vcpu_stats 2 0.0% >>> kernel`atomic_add_long 2 0.0% >>> vmm.ko`0xffffffff81 2 0.0% >>> kernel`vm_reserv_alloc_page 2 0.0% >>> kernel`pagezero 2 0.0% >>> kernel`bzero 2 0.0% >>> zfs.ko`fletcher_2_native 2 0.0% >>> vmm.ko`vmx_getreg 2 0.0% >>> vmm.ko`vm_guest_msrs 3 0.0% >>> vmm.ko`vmx_setreg 3 0.0% >>> vmm.ko`vm_get_register 3 0.0% >>> kernel`sx_try_xlock_ 3 0.0% >>> kernel`vm_radix_lookup_le 3 0.0% >>> vmm.ko`lapic_pending_intr 4 0.0% >>> kernel`syscall_thread_enter 4 0.0% >>> kernel`cpu_set_syscall_retval 4 0.0% >>> kernel`__mnt_vnode_next_all 4 0.0% >>> kernel`vfs_stdsync 4 0.0% >>> vmm.ko`vmm_fetch_instruction 5 0.0% >>> vmm.ko`vlapic_op_mem_read 6 0.0% >>> kernel`_sx_xunlock 7 0.0% >>> vmm.ko`vmm_emulate_instruction 7 0.0% >>> vmm.ko`vlapic_op_mem_write 7 0.0% >>> kernel`__mtx_lock_spin_flags 8 0.0% >>> vmm.ko`lapic_timer_tick 8 0.0% >>> zfs.ko`list_next 8 0.0% >>> zfs.ko`l2arc_feed_thread 8 0.0% >>> kernel`userret 9 0.0% >>> vmm.ko`vm_gpa2hpa 9 0.0% >>> vmm.ko`vmm_decode_instruction 11 0.0% >>> vmm.ko`ept_vmmmap_get 11 0.0% >>> vmm.ko`vm_lapic 11 0.0% >>> kernel`spinlock_enter 14 0.0% >>> kernel`__mtx_lock_sleep 15 0.0% >>> vmm.ko`vcpu_set_state 16 0.0% >>> vmm.ko`vm_nmi_pending 16 0.0% >>> kernel`malloc_type_zone_****allocated 16 0.0% >>> >>> kernel`devfs_ioctl_f 16 0.0% >>> kernel`pmap_kextract 17 0.0% >>> kernel`sys_ioctl 18 0.0% >>> kernel`malloc 21 0.0% >>> kernel`cpu_fetch_syscall_args 24 0.0% >>> kernel`uma_zfree_arg 28 0.0% >>> vmm.ko`vlapic_pending_intr 29 0.0% >>> kernel`trap 30 0.0% >>> kernel`hpet_get_timecount 31 0.0% >>> dtrace.ko`dtrace_trap 35 0.0% >>> vmm.ko`vlapic_update_ppr 39 0.0% >>> vmm.ko`vmmdev_ioctl 40 0.0% >>> kernel`copyout 43 0.0% >>> kernel`critical_enter 44 0.0% >>> kernel`copyin 51 0.1% >>> vmm.ko`vmcs_read 53 0.1% >>> kernel`fpusave 61 0.1% >>> kernel`uma_zalloc_arg 62 0.1% >>> kernel`devvn_refthread 69 0.1% >>> kernel`atomic_load_acq_int 70 0.1% >>> kernel`fpurestore 70 0.1% >>> vmm.ko`vlapic_timer_tick 75 0.1% >>> kernel`amd64_syscall 77 0.1% >>> kernel`bcopy 91 0.1% >>> kernel`fpudna 103 0.1% >>> kernel`free 103 0.1% >>> kernel`critical_exit 105 0.1% >>> kernel`dev_relthread 108 0.1% >>> kernel`spinlock_exit 119 0.1% >>> kernel`kern_ioctl 138 0.1% >>> kernel`fpuexit 242 0.3% >>> kernel`0xffffffff80 306 0.3% >>> vmm.ko`restore_host_msrs 398 0.4% >>> vmm.ko`vm_run 428 0.4% >>> vmm.ko`restore_guest_msrs 433 0.5% >>> kernel`cpu_idle 906 1.0% >>> vmm.ko`vmx_run 9266 9.7% >>> kernel`acpi_cpu_c1 81134 85.2% >>> >>> Hope this helps, let me know if you have any ideas on what to try next. >>> >>> >>> Could you try the following command to get vcpu stats? >> sudo bhyvectl --vm=vm1 --get-stats --cpu=0 >> sudo bhyvectl --vm=vm1 --get-stats --cpu=1 >> >> best >> Neel >> >> >> Dan >>> >>> >>> >>> >>> On Fri, 29 Mar 2013, Dan Mack wrote: >>> >>> On Thu, 28 Mar 2013, Peter Grehan wrote: >>> >>>> >>>> Hi Dan, >>>> >>>>> >>>>> Any ideas? >>>>> >>>>>> >>>>>> >>>>> Just to confirm: your host system is an i7 3930k, which looks like a 6 >>>>> core system. Should be plenty beefy enough for a 2 vCPU guest. >>>>> >>>>> >>>> That's what I thought, 6 cores, 12 threads, 64 GB RAM. >>>> >>>> The issue you are seeing looks a bit like what happens when systems are >>>> >>>>> heavily oversubscribed and vCPUs can't get enough run time. >>>>> >>>>> >>>> I was running in on an idle system; only bhyve was running. >>>> >>>> >>>> How much spare RAM is on the host when you started the vm ? Also, is >>>> >>>>> there much running on the host when the guest is started ? >>>>> >>>>> >>>> 64GB >>>> >>>> Here's some more information (sorry for the formatting): >>>> >>>> Case 1: single vCPU via: >>>> root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 >>>> /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>> >>>> Top shows it idle (note low load, 4GB memory given to guest, still 53GB >>>> free RAM, nothing else clocking CPU cycles): >>>> >>>> root@olive:~ # top -bH >>>> last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 >>>> 14:11:51 >>>> 24 processes: 1 running, 23 sleeping >>>> >>>> Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free >>>> ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other >>>> Swap: >>>> >>>> >>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>>> 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% >>>> bhyve{coco vcpu 0} >>>> 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd >>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>> 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% >>>> bhyve{coco mevent} >>>> >>>> >>>> So, case #1 all is normal. I was able to build a kernel with very good >>>> performance. >>>> >>>> The boot sequence looks like the following: >>>> >>>> root@olive:/vms/coco # ./vmrun.sh coco >>>> Launching virtual machine "coco" ... >>>> Consoles: userboot >>>> >>>> FreeBSD/amd64 User boot, Revision 1.1 >>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08] >>>> >>>> / >>>> ______ ____ _____ _____ >>>> | ____| | _ \ / ____| __ \ >>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>> | | | | | __/ __/| |_) |____) | |__| | >>>> | | | | | | || | | | >>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>> ` >>>> s` `.....---.......--.``` >>>> -/ >>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>> +. >>>> | | yo`:. :o >>>> `+- >>>> | 1. Boot Multi User [Enter] | y/ -/` -o/ >>>> | 2. Boot [S]ingle User | .- >>>> ::/sy+:. >>>> | 3. [Esc]ape to loader prompt | / `-- >>>> / >>>> | 4. Reboot | `: >>>> :` >>>> | | `: >>>> :` >>>> | Options: | / >>>> / >>>> | 5. Configure Boot [O]ptions... | .- >>>> -. >>>> | | -- -. >>>> | | `:` `:` >>>> | | .-- `--. >>>> | | .---.....----. >>>> +-----------------------------****------------+ >>>> >>>> >>>> >>>> Booting... >>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >>>> The Regents of the University of California. All rights >>>> reserved. >>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>> root@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64 >>>> >>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU) >>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>> Stepping = 7 >>>> Features=0x8fa3ab7f>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE> >>>> Features2=0x83bee217>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,** >>>> TSCDLT,**AESNI,HV> >>>> AMD Features=0x2c100800 >>>> >>>> AMD Features2=0x1 >>>> TSC: P-state invariant >>>> real memory = 5368709120 (5120 MB) >>>> avail memory = 3957534720 (3774 MB) >>>> Event timer "LAPIC" quality 400 >>>> ACPI APIC Table: >>>> random device not loaded; using insecure entropy >>>> ioapic0: Changing APIC ID to 1 >>>> ioapic0 irqs 0-16 on motherboard >>>> kbd0 at kbdmux0 >>>> acpi0: on motherboard >>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>> pcib0: port 0xcf8-0xcff on acpi0 >>>> pci0: on pcib0 >>>> pcib0: no PRT entry for 0.31.INTA >>>> virtio_pci0: port 0x2000-0x201f mem >>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>> vtnet0: on virtio_pci0 >>>> virtio_pci0: host features: 0x18020 >>>> virtio_pci0: negotiated features: 0x18020 >>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>> virtio_pci1: port 0x2040-0x207f mem >>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>> vtblk0: on virtio_pci1 >>>> virtio_pci1: host features: 0x10000004 >>>> virtio_pci1: negotiated features: 0x10000004 >>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>> pci0 >>>> uart2: console (9600,n,8,1) >>>> Timecounters tick every 10.000 msec >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 >>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf. >>>> >>>> Setting hostid: 0x24bfd025. >>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>> Starting file system checks: >>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>> /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% >>>> fragmentation) >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> Mounting local file systems:. >>>> Writing entropy file:. >>>> Setting hostname: cocopuff.example.com. >>>> vtnet0: link state changed to UP >>>> Starting Network: lo0 vtnet0. >>>> lo0: flags=8049 metric 0 mtu 16384 >>>> options=600003 >>>> >>>> inet6 ::1 prefixlen 128 >>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>> inet 127.0.0.1 netmask 0xff000000 >>>> nd6 options=21 >>>> vtnet0: flags=8943>>> MULTICAST> >>>> metric 0 mtu 1500 >>>> options=80028 >>>> >>>> ether 00:a0:98:ac:0d:df >>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>> nd6 options=29 >>>> >>>> media: Ethernet 1000baseT >>>> status: active >>>> Starting devd. >>>> add net default: gateway 172.22.22.252 >>>> add net ::ffff:0.0.0.0: gateway ::1 >>>> add net ::0.0.0.0: gateway ::1 >>>> add net fe80::: gateway ::1 >>>> add net ff02::: gateway ::1 >>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>> Creating and/or trimming log files. >>>> Starting syslogd. >>>> Clearing /tmp (X related). >>>> Updating motd:. >>>> Performing sanity check on sshd configuration. >>>> Starting sshd. >>>> Starting cron. >>>> Starting background file system checks in 60 seconds. >>>> >>>> Fri Mar 29 14:09:20 CDT 2013 >>>> >>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>> >>>> login: >>>> >>>> >>>> ====== Next, boot the same image with two (2) vCPU: >>>> >>>> Command run by vmrun.sh: >>>> >>>> root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 >>>> /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s >>>> 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco >>>> >>>> Top shows it running out of control: >>>> >>>> root@olive:~ # top -bH >>>> last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 >>>> 14:17:32 >>>> 25 processes: 3 running, 22 sleeping >>>> >>>> Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free >>>> ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other >>>> Swap: >>>> >>>> >>>> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >>>> 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% >>>> bhyve{coco vcpu 1} >>>> 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% >>>> bhyve{coco vcpu 0} >>>> 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd >>>> 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd >>>> 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd >>>> 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron >>>> 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd >>>> 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh >>>> 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% >>>> bhyve{coco mevent} >>>> 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh >>>> >>>> Even though, it's just sitting at the login prompt: >>>> >>>> root@olive:/vms/coco # ./vmrun.sh coco >>>> Launching virtual machine "coco" ... >>>> Consoles: userboot >>>> >>>> FreeBSD/amd64 User boot, Revision 1.1 >>>> (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) >>>> Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 >>>> data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+****0x12bb08] >>>> >>>> / >>>> ______ ____ _____ _____ >>>> | ____| | _ \ / ____| __ \ >>>> | |___ _ __ ___ ___ | |_) | (___ | | | | >>>> | ___| '__/ _ \/ _ \| _ < \___ \| | | | >>>> | | | | | __/ __/| |_) |____) | |__| | >>>> | | | | | | || | | | >>>> |_| |_| \___|\___||____/|_____/|_____/ ``` >>>> ` >>>> s` `.....---.......--.``` >>>> -/ >>>> +------------Welcome to FreeBSD-----------+ +o .--` /y:` >>>> +. >>>> | | yo`:. :o >>>> `+- >>>> | 1. Boot Multi User [Enter] | y/ -/` -o/ >>>> | 2. Boot [S]ingle User | .- >>>> ::/sy+:. >>>> | 3. [Esc]ape to loader prompt | / `-- >>>> / >>>> | 4. Reboot | `: >>>> :` >>>> | | `: >>>> :` >>>> | Options: | / >>>> / >>>> | 5. Configure Boot [O]ptions... | .- >>>> -. >>>> | | -- -. >>>> | | `:` `:` >>>> | | .-- `--. >>>> | | .---.....----. >>>> +-----------------------------****------------+ >>>> >>>> >>>> >>>> Booting... >>>> Copyright (c) 1992-2013 The FreeBSD Project. >>>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >>>> The Regents of the University of California. All rights >>>> reserved. >>>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>>> FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 >>>> root@cocopuff.example.com:/****sys/amd64/compile/BHYVE amd64 >>>> >>>> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 >>>> CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU) >>>> Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d >>>> Stepping = 7 >>>> Features=0x8fa3ab7f>>> SEP,PGE,CMOV,PAT,PSE36,DTS,****MMX,FXSR,SSE,SSE2,SS,PBE> >>>> Features2=0x83bee217>>> CX16,xTPR,PDCM,PCID,DCA,SSE4.****1,SSE4.2,x2APIC,POPCNT,** >>>> TSCDLT,**AESNI,HV> >>>> AMD Features=0x2c100800 >>>> >>>> AMD Features2=0x1 >>>> TSC: P-state invariant >>>> real memory = 5368709120 (5120 MB) >>>> avail memory = 3957534720 (3774 MB) >>>> Event timer "LAPIC" quality 400 >>>> ACPI APIC Table: >>>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >>>> FreeBSD/SMP: 2 package(s) x 1 core(s) >>>> cpu0 (BSP): APIC ID: 0 >>>> cpu1 (AP): APIC ID: 1 >>>> random device not loaded; using insecure entropy >>>> ioapic0: Changing APIC ID to 2 >>>> ioapic0 irqs 0-16 on motherboard >>>> kbd0 at kbdmux0 >>>> acpi0: on motherboard >>>> atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 >>>> Event timer "RTC" frequency 32768 Hz quality 0 >>>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>>> acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>>> pcib0: port 0xcf8-0xcff on acpi0 >>>> pci0: on pcib0 >>>> pcib0: no PRT entry for 0.31.INTA >>>> virtio_pci0: port 0x2000-0x201f mem >>>> 0xc0000000-0xc0001fff at device 1.0 on pci0 >>>> vtnet0: on virtio_pci0 >>>> virtio_pci0: host features: 0x18020 >>>> virtio_pci0: negotiated features: 0x18020 >>>> vtnet0: Ethernet address: 00:a0:98:ac:0d:df >>>> virtio_pci1: port 0x2040-0x207f mem >>>> 0xc0002000-0xc0003fff at device 2.0 on pci0 >>>> vtblk0: on virtio_pci1 >>>> virtio_pci1: host features: 0x10000004 >>>> virtio_pci1: negotiated features: 0x10000004 >>>> vtblk0: 8192MB (16777216 512 byte sectors) >>>> uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on >>>> pci0 >>>> uart2: console (9600,n,8,1) >>>> Timecounters tick every 10.000 msec >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> SMP: AP CPU #1 Launched! >>>> Trying to mount root from ufs:/dev/vtbd0p2 [rw]... >>>> Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-****00a098ac0ddf. >>>> >>>> Setting hostid: 0x24bfd025. >>>> Entropy harvesting: interrupts ethernet point_to_point kickstart. >>>> Starting file system checks: >>>> /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS >>>> /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% >>>> fragmentation) >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT >>>> Mounting local file systems:. >>>> Writing entropy file:. >>>> Setting hostname: cocopuff.example.com. >>>> vtnet0: link state changed to UP >>>> Starting Network: lo0 vtnet0. >>>> lo0: flags=8049 metric 0 mtu 16384 >>>> options=600003 >>>> >>>> inet6 ::1 prefixlen 128 >>>> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 >>>> inet 127.0.0.1 netmask 0xff000000 >>>> nd6 options=21 >>>> vtnet0: flags=8943>>> MULTICAST> >>>> metric 0 mtu 1500 >>>> options=80028 >>>> >>>> ether 00:a0:98:ac:0d:df >>>> inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 >>>> inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 >>>> nd6 options=29 >>>> >>>> media: Ethernet 1000baseT >>>> status: active >>>> Starting devd. >>>> add net default: gateway 172.22.22.252 >>>> add net ::ffff:0.0.0.0: gateway ::1 >>>> add net ::0.0.0.0: gateway ::1 >>>> add net fe80::: gateway ::1 >>>> add net ff02::: gateway ::1 >>>> ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib >>>> 32-bit compatibility ldconfig path: /usr/lib32 >>>> Creating and/or trimming log files. >>>> Starting syslogd. >>>> syslogd: timed out waiting for child >>>> /etc/rc: WARNING: failed to start syslogd >>>> Clearing /tmp (X related). >>>> Updating motd:. >>>> Performing sanity check on sshd configuration. >>>> Starting sshd. >>>> Starting cron. >>>> Starting background file system checks in 60 seconds. >>>> >>>> Thu Apr 18 02:06:43 CDT 2013 >>>> >>>> FreeBSD/amd64 (cocopuff.example.com) (console) >>>> >>>> login: >>>> >>>> >>>> ========== end of boot sequence >>>> >>>> So, something is broken with SMP on the guest, I'd be happy to help just >>>> let me know. >>>> >>>> Dan >>>> >>>> ______________________________****_________________ >>>> freebsd-virtualization@**freeb**sd.org < >>>> freebsd-virtualization@**freebsd.org>mailing >>>> list >>>> http://lists.freebsd.org/****mailman/listinfo/freebsd-**** >>>> virtualization >>>> >>>> > >>>> To unsubscribe, send any mail to "freebsd-virtualization-** >>>> unsubscribe@freebsd.org >>> unsubscribe@freebsd.org >>>> >" >>>> >>>> ______________________________****_________________ >>>> >>> freebsd-virtualization@**freeb**sd.org < >>> freebsd-virtualization@**freebsd.org>mailing >>> list >>> http://lists.freebsd.org/****mailman/listinfo/freebsd-****virtualization >>> >>> > >>> To unsubscribe, send any mail to "freebsd-virtualization-** >>> unsubscribe@freebsd.org >> unsubscribe@freebsd.org >>> >" >>> >>> >>