From owner-freebsd-virtualization@FreeBSD.ORG Fri Mar 29 19:19:20 2013 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A7C8274; Fri, 29 Mar 2013 19:19:20 +0000 (UTC) (envelope-from mack@macktronics.com) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by mx1.freebsd.org (Postfix) with ESMTP id 5205D734; Fri, 29 Mar 2013 19:19:19 +0000 (UTC) Received: from coco.macktronics.com (coco.macktronics.com [209.181.253.65]) by coco.macktronics.com (Postfix) with ESMTP id 3D5844AC40; Fri, 29 Mar 2013 14:19:19 -0500 (CDT) Date: Fri, 29 Mar 2013 14:19:19 -0500 (CDT) From: Dan Mack To: Peter Grehan Subject: Re: bhyve tty / login problems / panic In-Reply-To: <515514E2.1080905@freebsd.org> Message-ID: <20130329140714.L12104@coco.macktronics.com> References: <20130328155008.B5464@coco.macktronics.com> <515514E2.1080905@freebsd.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Mar 2013 19:19:20 -0000 On Thu, 28 Mar 2013, Peter Grehan wrote: > Hi Dan, > >> Any ideas? > > Just to confirm: your host system is an i7 3930k, which looks like a 6 core > system. Should be plenty beefy enough for a 2 vCPU guest. That's what I thought, 6 cores, 12 threads, 64 GB RAM. > The issue you are seeing looks a bit like what happens when systems are > heavily oversubscribed and vCPUs can't get enough run time. I was running in on an idle system; only bhyve was running. > How much spare RAM is on the host when you started the vm ? Also, is there > much running on the host when the guest is started ? 64GB Here's some more information (sorry for the formatting): Case 1: single vCPU via: root 64112 1.3 0.0 4219232 25588 1 D+ 2:09PM 0:02.47 /usr/sbin/bhyve -c 1 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco Top shows it idle (note low load, 4GB memory given to guest, still 53GB free RAM, nothing else clocking CPU cycles): root@olive:~ # top -bH last pid: 64148; load averages: 0.03, 0.22, 0.22 up 0+16:29:31 14:11:51 24 processes: 1 running, 23 sleeping Mem: 32M Active, 50M Inact, 8498M Wired, 504K Cache, 53G Free ARC: 778M Total, 146M MFU, 390M MRU, 16K Anon, 16M Header, 226M Other Swap: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 64112 root 20 0 4120M 25980K vmidle 2 0:03 0.00% bhyve{coco vcpu 0} 925 root 20 0 14296K 2044K select 11 0:00 0.00% syslogd 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh 64112 root 52 0 4120M 25980K kqread 9 0:00 0.00% bhyve{coco mevent} So, case #1 all is normal. I was able to build a kernel with very good performance. The boot sequence looks like the following: root@olive:/vms/coco # ./vmrun.sh coco Launching virtual machine "coco" ... Consoles: userboot FreeBSD/amd64 User boot, Revision 1.1 (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08] / ______ ____ _____ _____ | ____| | _ \ / ____| __ \ | |___ _ __ ___ ___ | |_) | (___ | | | | | ___| '__/ _ \/ _ \| _ < \___ \| | | | | | | | | __/ __/| |_) |____) | |__| | | | | | | | || | | | |_| |_| \___|\___||____/|_____/|_____/ ``` ` s` `.....---.......--.``` -/ +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. | | yo`:. :o `+- | 1. Boot Multi User [Enter] | y/ -/` -o/ | 2. Boot [S]ingle User | .- ::/sy+:. | 3. [Esc]ape to loader prompt | / `-- / | 4. Reboot | `: :` | | `: :` | Options: | / / | 5. Configure Boot [O]ptions... | .- -. | | -- -. | | `:` `:` | | .-- `--. | | .---.....----. +-----------------------------------------+ Booting... Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64 FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.84-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0x8fa3ab7f Features2=0x83bee217 AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant real memory = 5368709120 (5120 MB) avail memory = 3957534720 (3774 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: random device not loaded; using insecure entropy ioapic0: Changing APIC ID to 1 ioapic0 irqs 0-16 on motherboard kbd0 at kbdmux0 acpi0: on motherboard atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib0: no PRT entry for 0.31.INTA virtio_pci0: port 0x2000-0x201f mem 0xc0000000-0xc0001fff at device 1.0 on pci0 vtnet0: on virtio_pci0 virtio_pci0: host features: 0x18020 virtio_pci0: negotiated features: 0x18020 vtnet0: Ethernet address: 00:a0:98:ac:0d:df virtio_pci1: port 0x2040-0x207f mem 0xc0002000-0xc0003fff at device 2.0 on pci0 vtblk0: on virtio_pci1 virtio_pci1: host features: 0x10000004 virtio_pci1: negotiated features: 0x10000004 vtblk0: 8192MB (16777216 512 byte sectors) uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 uart2: console (9600,n,8,1) Timecounters tick every 10.000 msec g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT Timecounter "TSC-low" frequency 1599917752 Hz quality 1000 Trying to mount root from ufs:/dev/vtbd0p2 [rw]... Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf. Setting hostid: 0x24bfd025. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/vtbd0p2: clean, 610778 free (14298 frags, 74560 blocks, 0.7% fragmentation) g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT Mounting local file systems:. Writing entropy file:. Setting hostname: cocopuff.example.com. vtnet0: link state changed to UP Starting Network: lo0 vtnet0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 vtnet0: flags=8943 metric 0 mtu 1500 options=80028 ether 00:a0:98:ac:0d:df inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet 1000baseT status: active Starting devd. add net default: gateway 172.22.22.252 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. Clearing /tmp (X related). Updating motd:. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Fri Mar 29 14:09:20 CDT 2013 FreeBSD/amd64 (cocopuff.example.com) (console) login: ====== Next, boot the same image with two (2) vCPU: Command run by vmrun.sh: root 64163 182.5 0.1 4219360 48968 1 R+ 2:16PM 1:10.42 /usr/sbin/bhyve -c 2 -m 4096 -AI -H -P -g 0 -s 0:0,hostbridge -s 1:0,virtio-net,tap0 -s 2:0,virtio-blk,./diskdev -S 31,uart,stdio coco Top shows it running out of control: root@olive:~ # top -bH last pid: 64191; load averages: 1.45, 0.52, 0.31 up 0+16:35:12 14:17:32 25 processes: 3 running, 22 sleeping Mem: 32M Active, 50M Inact, 9016M Wired, 504K Cache, 53G Free ARC: 1293M Total, 370M MFU, 671M MRU, 8208K Anon, 17M Header, 227M Other Swap: PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 64163 root 102 0 4120M 120M CPU2 2 1:01 100.00% bhyve{coco vcpu 1} 64163 root 102 0 4120M 120M CPU8 8 1:03 94.48% bhyve{coco vcpu 0} 15751 root 20 0 81452K 7092K select 5 0:00 0.00% sshd 807 root 20 0 13112K 4812K select 5 0:00 0.00% devd 925 root 20 0 14296K 2044K select 9 0:00 0.00% syslogd 1031 root 20 0 16380K 2400K nanslp 10 0:00 0.00% cron 2695 root 20 0 81452K 6952K select 2 0:00 0.00% sshd 2697 root 20 0 20224K 4632K ttyin 1 0:00 0.00% csh 64163 root 52 0 4120M 120M kqread 5 0:00 0.00% bhyve{coco mevent} 15753 root 20 0 20224K 4580K pause 1 0:00 0.00% csh Even though, it's just sitting at the login prompt: root@olive:/vms/coco # ./vmrun.sh coco Launching virtual machine "coco" ... Consoles: userboot FreeBSD/amd64 User boot, Revision 1.1 (root@olive.example.com, Thu Mar 28 21:20:49 CDT 2013) Loading /boot/defaults/loader.conf /boot/kernel/kernel text=0x5a2178 data=0x98088+0xef8b0 syms=[0x8+0xca8d8+0x8+0x12bb08] / ______ ____ _____ _____ | ____| | _ \ / ____| __ \ | |___ _ __ ___ ___ | |_) | (___ | | | | | ___| '__/ _ \/ _ \| _ < \___ \| | | | | | | | | __/ __/| |_) |____) | |__| | | | | | | | || | | | |_| |_| \___|\___||____/|_____/|_____/ ``` ` s` `.....---.......--.``` -/ +------------Welcome to FreeBSD-----------+ +o .--` /y:` +. | | yo`:. :o `+- | 1. Boot Multi User [Enter] | y/ -/` -o/ | 2. Boot [S]ingle User | .- ::/sy+:. | 3. [Esc]ape to loader prompt | / `-- / | 4. Reboot | `: :` | | `: :` | Options: | / / | 5. Configure Boot [O]ptions... | .- -. | | -- -. | | `:` `:` | | .-- `--. | | .---.....----. +-----------------------------------------+ Booting... Copyright (c) 1992-2013 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-CURRENT #0 r248859: Thu Mar 28 19:52:06 CDT 2013 root@cocopuff.example.com:/sys/amd64/compile/BHYVE amd64 FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221 CPU: Intel(R) Core(TM) i7-3930K CPU @ 3.20GHz (3199.82-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 0x6 Model = 0x2d Stepping = 7 Features=0x8fa3ab7f Features2=0x83bee217 AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant real memory = 5368709120 (5120 MB) avail memory = 3957534720 (3774 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 2 package(s) x 1 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 random device not loaded; using insecure entropy ioapic0: Changing APIC ID to 2 ioapic0 irqs 0-16 on motherboard kbd0 at kbdmux0 acpi0: on motherboard atrtc0: port 0x70-0x71,0x72-0x77 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib0: no PRT entry for 0.31.INTA virtio_pci0: port 0x2000-0x201f mem 0xc0000000-0xc0001fff at device 1.0 on pci0 vtnet0: on virtio_pci0 virtio_pci0: host features: 0x18020 virtio_pci0: negotiated features: 0x18020 vtnet0: Ethernet address: 00:a0:98:ac:0d:df virtio_pci1: port 0x2040-0x207f mem 0xc0002000-0xc0003fff at device 2.0 on pci0 vtblk0: on virtio_pci1 virtio_pci1: host features: 0x10000004 virtio_pci1: negotiated features: 0x10000004 vtblk0: 8192MB (16777216 512 byte sectors) uart2: <16550 or compatible> port 0x3f8-0x3ff irq 4 at device 31.0 on pci0 uart2: console (9600,n,8,1) Timecounters tick every 10.000 msec g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/vtbd0p2 [rw]... Setting hostuuid: d4e0b8a0-97c6-11e2-b4c8-00a098ac0ddf. Setting hostid: 0x24bfd025. Entropy harvesting: interrupts ethernet point_to_point kickstart. Starting file system checks: /dev/vtbd0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS /dev/vtbd0p2: clean, 610776 free (14296 frags, 74560 blocks, 0.7% fragmentation) g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT g_handleattr: vtbd0 bio_length 24 len 28 -> EFAULT Mounting local file systems:. Writing entropy file:. Setting hostname: cocopuff.example.com. vtnet0: link state changed to UP Starting Network: lo0 vtnet0. lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 vtnet0: flags=8943 metric 0 mtu 1500 options=80028 ether 00:a0:98:ac:0d:df inet 172.22.222.230 netmask 0xffff0000 broadcast 172.22.255.255 inet6 fe80::2a0:98ff:feac:ddf%vtnet0 prefixlen 64 scopeid 0x1 nd6 options=29 media: Ethernet 1000baseT status: active Starting devd. add net default: gateway 172.22.22.252 add net ::ffff:0.0.0.0: gateway ::1 add net ::0.0.0.0: gateway ::1 add net fe80::: gateway ::1 add net ff02::: gateway ::1 ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib 32-bit compatibility ldconfig path: /usr/lib32 Creating and/or trimming log files. Starting syslogd. syslogd: timed out waiting for child /etc/rc: WARNING: failed to start syslogd Clearing /tmp (X related). Updating motd:. Performing sanity check on sshd configuration. Starting sshd. Starting cron. Starting background file system checks in 60 seconds. Thu Apr 18 02:06:43 CDT 2013 FreeBSD/amd64 (cocopuff.example.com) (console) login: ========== end of boot sequence So, something is broken with SMP on the guest, I'd be happy to help just let me know. Dan