From owner-freebsd-virtualization@FreeBSD.ORG Sun May 31 05:43:56 2015 Return-Path: Delivered-To: virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 68C4AFA0 for ; Sun, 31 May 2015 05:43:56 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2A0301753 for ; Sun, 31 May 2015 05:43:53 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from jre-mbp.elischer.org (ppp121-45-250-63.lns20.per4.internode.on.net [121.45.250.63]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t4V5hg48087298 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO) for ; Sat, 30 May 2015 22:43:45 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <556A9F88.80406@freebsd.org> Date: Sun, 31 May 2015 13:43:36 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: virtualization@FreeBSD.org Subject: anyone used AliYun cloud? Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 May 2015 05:43:56 -0000 Aliyun cloud is alibaba's amazon cloud competitor. Also uses Xen. They offer Linux and windows of course but nothing in FreeBSD. Has anyone looked at it? It's coming up with $JOB as a question vs FreeBSD. Julian From owner-freebsd-virtualization@FreeBSD.ORG Sun May 31 10:06:44 2015 Return-Path: Delivered-To: virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E62CAA8F for ; Sun, 31 May 2015 10:06:44 +0000 (UTC) (envelope-from gavin.mu@qq.com) Received: from smtpbgau1.qq.com (smtpbgau1.qq.com [54.206.16.166]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14AEC1018 for ; Sun, 31 May 2015 10:06:43 +0000 (UTC) (envelope-from gavin.mu@qq.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s201307; t=1433066791; bh=NeZBDSqntZw0XsJIIortohHKfO19mdaZenPxbeudXn4=; h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Content-Transfer-Encoding:Message-Id:References:To; b=sKOymeGvyBmBrbKWAzdXJ9AKlHmOeGkm9xHuo09jZykquSVNVwH5td0Upy+yaQf6+ JIdzMDtglz6G2gGOSEkBy3suhR0UEZtBN0/qDM6kX7PpjAQwl2JvMpfY2TFEmws2AU bvZ1e2J2Gcz9cHffxg3EcRhzrZZqoaJfGc6GCHgs= X-QQ-mid: esmtp23t1433066789t750t07152 Received: from [192.168.1.200] (unknown [114.250.195.162]) by esmtp4.qq.com (ESMTP) with id ; Sun, 31 May 2015 18:06:28 +0800 (CST) X-QQ-SSF: C100000000000080F5100200000000Z X-QQ-FEAT: lfr97vDGpNwddi8tka4z+YAb/V3uHrh1l6AOSlfILCWMowWd2e8PsJY8wz5fG 5uxlIuajg1ck2kZ9k3gWWX561yrvnRcAi3B6gz854LPxuRMnj1wSNdyKaAL9XyicUhqu/gH wGlqw6AxvsejEcpCCrsjvcoNq5ov5drfz6GiZ0TdKEamX+AqPHaInrt33Cd7rsI5kq/95XE evVZE2sMbb7nIyqHlhiaiJPXxjRM00sI= X-QQ-GoodBg: 0 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: anyone used AliYun cloud? From: Gavin Mu X-Mailer: iPad Mail (12F69) In-Reply-To: <556A9F88.80406@freebsd.org> Date: Sun, 31 May 2015 18:06:28 +0800 Cc: "virtualization@FreeBSD.org" Content-Transfer-Encoding: quoted-printable Message-Id: <48CDFA47-08B8-47A5-82C7-E3A8171042CE@qq.com> References: <556A9F88.80406@freebsd.org> To: Julian Elischer X-QQ-SENDSIZE: 520 X-QQ-Bgrelay: 1 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 May 2015 10:06:45 -0000 They use guestfs to do VM provisioning, and FreeBSD support is not ready to p= ublic yet. If FreeBSD must be used, one workaround is to keep the first MBR partition a= s Linux ext2, and write some scripts to read configurations from Linux. Regards, Gavin Mu > On May 31, 2015, at 13:43, Julian Elischer wrote: >=20 > Aliyun cloud is alibaba's amazon cloud competitor. Also uses Xen. > They offer Linux and windows of course but nothing in FreeBSD. > Has anyone looked at it? It's coming up with $JOB as a question vs FreeBS= D. >=20 > Julian >=20 >=20 > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freeb= sd.org" From owner-freebsd-virtualization@FreeBSD.ORG Sun May 31 15:41:23 2015 Return-Path: Delivered-To: virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 88814DA6 for ; Sun, 31 May 2015 15:41:23 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 60A3E1B7B for ; Sun, 31 May 2015 15:41:22 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from jre-mbp.elischer.org (ppp121-45-250-63.lns20.per4.internode.on.net [121.45.250.63]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id t4VFfDIJ090335 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sun, 31 May 2015 08:41:18 -0700 (PDT) (envelope-from julian@freebsd.org) Message-ID: <556B2B93.4050305@freebsd.org> Date: Sun, 31 May 2015 23:41:07 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Gavin Mu CC: "virtualization@FreeBSD.org" Subject: Re: anyone used AliYun cloud? References: <556A9F88.80406@freebsd.org> <48CDFA47-08B8-47A5-82C7-E3A8171042CE@qq.com> In-Reply-To: <48CDFA47-08B8-47A5-82C7-E3A8171042CE@qq.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 31 May 2015 15:41:23 -0000 On 5/31/15 6:06 PM, Gavin Mu wrote: > They use guestfs to do VM provisioning, and FreeBSD support is not ready to public yet. > > If FreeBSD must be used, one workaround is to keep the first MBR partition as Linux ext2, and write some scripts to read configurations from Linux. thanks.. Firstly, How do you know this? (I don't disbelieve it, I just want to know where you learned this information so I can look there too for more information). Secondly, is your comment "is not ready to public yet" talking about guestfs or aliyun? and can you expand on "yet"? Julian > > Regards, > Gavin Mu > >> On May 31, 2015, at 13:43, Julian Elischer wrote: >> >> Aliyun cloud is alibaba's amazon cloud competitor. Also uses Xen. >> They offer Linux and windows of course but nothing in FreeBSD. >> Has anyone looked at it? It's coming up with $JOB as a question vs FreeBSD. >> >> Julian >> >> >> _______________________________________________ >> freebsd-virtualization@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > > > > From owner-freebsd-virtualization@FreeBSD.ORG Mon Jun 1 00:56:20 2015 Return-Path: Delivered-To: virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D4DE560D for ; Mon, 1 Jun 2015 00:56:20 +0000 (UTC) (envelope-from gavin.mu@qq.com) Received: from smtpbgbr1.qq.com (smtpbgbr1.qq.com [54.207.19.206]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DE6CE18DB for ; Mon, 1 Jun 2015 00:56:19 +0000 (UTC) (envelope-from gavin.mu@qq.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=qq.com; s=s201307; t=1433120167; bh=dPjMWiudlFB5S4l8CjN/vvDvuHIMnR0M8KwMPYbC9NE=; h=Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Content-Transfer-Encoding:Message-Id:References:To; b=xSI4CC5A9WvxXxRAUFHqYGU+yV8IFTnjpOJrllN1xPMQBfEzqVkTsjnAkdDrAgg0F hIM8E1YABPr/ThGfhGdpYSLRiwszILKEsW6Td+7t6KscKiSDuA3S4JP3PnxnICDPug 7Qha50AYeQhm03cCt3+r5DaPTlqoLJ1QMrXWY8OA= X-QQ-mid: esmtp20t1433120165t321t31112 Received: from [10.30.189.249] (unknown [114.242.250.50]) by esmtp4.qq.com (ESMTP) with id ; Mon, 01 Jun 2015 08:56:02 +0800 (CST) X-QQ-SSF: C100000000000090F5100F00000000Z X-QQ-FEAT: wobzSXj0Mt7PSocpu8SzWmgMZJpNhqJEBNK3BObV9wRD+gtGfVLZwDQNXg5aF itm0JRrFMfTr16BUUfuZdJpzjPmvkTiLTe6sbw8/uXLC1OQvEUe7IoVh6H+yOsg4EwgrsEm q9MJfMb5dvIUpLSyfMnYBok4Bi7Y9lI2DcfRhBygC5SpwEnEmfbQB1RQUD9zYp6svbe5x7c 0P+eq/NZrbVbDwEdEQNK1NgiS7VlLI7A= X-QQ-GoodBg: 0 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: anyone used AliYun cloud? From: Gavin Mu X-Mailer: iPad Mail (12F69) In-Reply-To: <556B2B93.4050305@freebsd.org> Date: Mon, 1 Jun 2015 08:55:56 +0800 Cc: "virtualization@FreeBSD.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <556A9F88.80406@freebsd.org> <48CDFA47-08B8-47A5-82C7-E3A8171042CE@qq.com> <556B2B93.4050305@freebsd.org> To: Julian Elischer X-QQ-SENDSIZE: 520 X-QQ-Bgrelay: 1 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Jun 2015 00:56:21 -0000 This is a little long story... We have one product based on FreeBSD, and som= eones thought it is difficult to deploy on some popular clouds, especially i= n China, such as QingCloud and Aliyun, all provide Linux and Windows images b= ut not FreeBSD. First I tried on QingCloud, who uses KVM, and FreeBSD can run, but it failed= to be saved as an image. =46rom the communication with QingCloud engineers,= I knew they use guestfs and can not recognize UFS filesystem. Several days ago, I tried Aliyun, who use Xen. FreeBSD can run also, and can= also be saved as an image, but VM provisioning failed. Their engineer told m= e that the file system has been changed and can not work. Later I tried to make a dual OS image, with 2GB Ubuntu on ext2 as the first M= BR primary partition, and the second one as FreeBSD, VM provisioning can wor= k now. I believe the ext2 partition can be reduced, but it must be very careful to n= ot break VM provisioning. Both QingCloud and Aliyun use the technology to write the initial configurat= ions to guest file system during VM provisioning, the difference is QingClou= d do checks when saving images, and Aliyun not. I am planning to try QingClo= ud again with the dual OS solution. We have also built an internal channel with Aliyun guys recently, but they d= id not tell the release date for FreeBSD support (business reason maybe). Regards, Gavin Mu > On May 31, 2015, at 23:41, Julian Elischer wrote: >=20 >> On 5/31/15 6:06 PM, Gavin Mu wrote: >> They use guestfs to do VM provisioning, and FreeBSD support is not ready t= o public yet. >>=20 >> If FreeBSD must be used, one workaround is to keep the first MBR partitio= n as Linux ext2, and write some scripts to read configurations from Linux. >=20 > thanks.. > Firstly, How do you know this? (I don't disbelieve it, I just want to know= where you learned this information so I can look there too for more informa= tion). > Secondly, is your comment "is not ready to public yet" talking about gues= tfs or aliyun? and can you expand on "yet"? >=20 > Julian >=20 >>=20 >> Regards, >> Gavin Mu >>=20 >>> On May 31, 2015, at 13:43, Julian Elischer wrote: >>>=20 >>> Aliyun cloud is alibaba's amazon cloud competitor. Also uses Xen. >>> They offer Linux and windows of course but nothing in FreeBSD. >>> Has anyone looked at it? It's coming up with $JOB as a question vs Free= BSD. >>>=20 >>> Julian >>>=20 >>>=20 >>> _______________________________________________ >>> freebsd-virtualization@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >>> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fre= ebsd.org" >>=20 >>=20 >>=20 >>=20 >=20 From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 11:15:48 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C20A4F2F for ; Tue, 2 Jun 2015 11:15:48 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 16980155B for ; Tue, 2 Jun 2015 11:15:44 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA22970 for ; Tue, 02 Jun 2015 14:15:35 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1YzkAQ-0002Q4-V6 for freebsd-virtualization@freebsd.org; Tue, 02 Jun 2015 14:15:34 +0300 Message-ID: <556D9005.4020802@FreeBSD.org> Date: Tue, 02 Jun 2015 14:14:13 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-virtualization@FreeBSD.org Subject: bhyve: corrupting zfs pools? Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 11:15:48 -0000 I am doing a simple experiment. I get FreeBSD image from here: ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img.xz Then I run in bhyve with two additional "disks" created with truncate -s 4g: $ bhyveload -m 1G -d ~/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img test $ bhyve -A -HP -s 0:0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,/home/avg/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img -s 3:1,virtio-blk,/tmp/l2arc-test/hdd1,sectorsize=512/4096 -s 3:2,virtio-blk,/tmp/l2arc-test/hdd2,sectorsize=512/4096 -l com1,stdio -l com2,/dev/nmdm0A -c 2 -m 1g test Note sectorsize=512/4096 options. Not sure if it's them that cause the trouble. Then, in the VM: $ zpool create l2arc-test mirror /dev/vtbd1 /dev/vtbd2 $ zfs create -p l2arc-test/ROOT/initial $ tar -c --one-file-system -f - / | tar -x -C /l2arc-test/ROOT/initial -f - Afterwards, zpool status -v reports no problem. But then I run zpool scrub and get the following in the end: $ zpool status -v pool: l2arc-test state: ONLINE status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://illumos.org/msg/ZFS-8000-8A scan: scrub repaired 356K in 0h0m with 9 errors on Tue Jun 2 13:58:17 2015 config: NAME STATE READ WRITE CKSUM l2arc-test ONLINE 0 0 9 mirror-0 ONLINE 0 0 18 vtbd1 ONLINE 0 0 25 vtbd2 ONLINE 0 0 23 errors: Permanent errors have been detected in the following files: /l2arc-test/ROOT/initial/usr/bin/svnlitesync /l2arc-test/ROOT/initial/usr/freebsd-dist/kernel.txz /l2arc-test/ROOT/initial/usr/freebsd-dist/src.txz /l2arc-test/ROOT/initial/usr/lib/clang/3.6.1/lib/freebsd/libclang_rt.asan-x86_64.a The same issue is reproducible with ahci-hd. My host system is a recent amd64 CURRENT as well. The hardware platform is AMD. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 11:21:01 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 89F7FFB5 for ; Tue, 2 Jun 2015 11:21:01 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id B8D6E16C4 for ; Tue, 2 Jun 2015 11:21:00 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA23028 for ; Tue, 02 Jun 2015 14:20:59 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1YzkFe-0002QN-Vi for freebsd-virtualization@freebsd.org; Tue, 02 Jun 2015 14:20:59 +0300 Message-ID: <556D9163.1080704@FreeBSD.org> Date: Tue, 02 Jun 2015 14:20:03 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-virtualization@FreeBSD.org Subject: bhyve: bhyveload, bhyve, bhyvectl --destroy Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 11:21:01 -0000 I am very new to bhyve, so sorry if I am asking something silly or obvious. I am using bhyve to speed up my testing and it seems that each time I need to restart a VM I need to go through the cycle of destroying it with bhyvectl --destroy, then re-loading a kernel with bhyveload and then actually booting the VM with bhyve. It seems that I have to do this even if I don't change th kernel between reboots. My first naive impression was that the point of bhyveload was to load the kernel once. Seems it ain't so? -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 14:59:23 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 35DB6345 for ; Tue, 2 Jun 2015 14:59:23 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id 128371D29 for ; Tue, 2 Jun 2015 14:59:22 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [192.168.1.2] (unknown [192.168.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id EAD479500 for ; Tue, 2 Jun 2015 14:59:21 +0000 (UTC) Message-ID: <556DC4D1.7070205@freebsd.org> Date: Tue, 02 Jun 2015 10:59:29 -0400 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-virtualization@freebsd.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy References: <556D9163.1080704@FreeBSD.org> In-Reply-To: <556D9163.1080704@FreeBSD.org> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="RF8P9t3xU2KVagw2sj5at2JSMh2WMuVlX" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 14:59:23 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --RF8P9t3xU2KVagw2sj5at2JSMh2WMuVlX Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2015-06-02 07:20, Andriy Gapon wrote: >=20 > I am very new to bhyve, so sorry if I am asking something silly or obvi= ous. > I am using bhyve to speed up my testing and it seems that each time I n= eed to > restart a VM I need to go through the cycle of destroying it with bhyve= ctl > --destroy, then re-loading a kernel with bhyveload and then actually bo= oting the > VM with bhyve. It seems that I have to do this even if I don't change = th kernel > between reboots. My first naive impression was that the point of bhyve= load was > to load the kernel once. Seems it ain't so? >=20 Are you running -CURRENT, 10-STABLE, or 10.1? I think there was talk recently of removing the requirement to destroy the VM before running bhyveload again. --=20 Allan Jude --RF8P9t3xU2KVagw2sj5at2JSMh2WMuVlX Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJVbcTVAAoJEJrBFpNRJZKfPl4P/RHy41iajqAQcB5sxuIUh+zE laP4G5zaKFYjj/YzsWPM6bU1K2h7oT3teOUJaxs7mR8NB5Z5I9V0EjXMeGRVJVKw yr0fO/oKTar9vDMoIgVHsAsp3pJ2UQST77wPzYo/7amVOyV+4eWrqAkiAksXjTUQ MU08cwl4AwJq/owneONu7eboy7sr3NHc686TlW0NLXkXsUXvnneD9DQwy2WgJh2i +PyTlklkG378/2DrKQlyXHMyc7SIovlmHmi9WpNADfi5OpHJQnfY6R2+p5ArnSEt jSjTM7DMu+l2Eqp4eObThtTYw6rAgYjqhMrFpOrhOgCiu1I7KYHGTjQ98P5JLeqQ qmkE1ELmRqkP4/3Qd+TTsT04Uvxg1llct0gTTrzYa7Y58e3VCeQbX3aWxQZb9Hay TSWtdqTlKPY0GHK7a1P8dG80ogRU4BcCgbtyIB/g67P/Bfb0EGv799NU21IiWTUC UY63HSheQnU2mWP8tQKmB1xsJKday39IQWaW4T+PJCHKsj2kKIJjVKpSsOYrnSsV NGMhFkeD21zwLuKqa8Pk1hZohelyVAodpxxJBA+xMesBpJPufbQVCITGoR34dG+U gGayPAhMuJIO9wvdHDiit8IvI/uxheCdbSPhPEn87rCFddmSowc6m8zvFafQaWIM +caDhBZhKi/uRvbzicKm =U0V5 -----END PGP SIGNATURE----- --RF8P9t3xU2KVagw2sj5at2JSMh2WMuVlX-- From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 14:59:38 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B452E395 for ; Tue, 2 Jun 2015 14:59:38 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id 71CE71D33 for ; Tue, 2 Jun 2015 14:59:38 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id 3BD0A124F8 for ; Wed, 3 Jun 2015 00:51:36 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 3442F280996 for ; Wed, 3 Jun 2015 00:51:36 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au X-Virus-Scanned: amavisd-new at iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id IyNDiRG8QqBt for ; Wed, 3 Jun 2015 00:51:36 +1000 (AEST) Received: from Peters-MacBook-Pro.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 7838E280995; Wed, 3 Jun 2015 00:51:34 +1000 (AEST) Message-ID: <556DC2FE.1000308@freebsd.org> Date: Tue, 02 Jun 2015 07:51:42 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Andriy Gapon CC: freebsd-virtualization@FreeBSD.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy References: <556D9163.1080704@FreeBSD.org> In-Reply-To: <556D9163.1080704@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 14:59:38 -0000 Hi Andriy, > I am very new to bhyve, so sorry if I am asking something silly or obvious. > I am using bhyve to speed up my testing and it seems that each time I need to > restart a VM I need to go through the cycle of destroying it with bhyvectl > --destroy, then re-loading a kernel with bhyveload and then actually booting the > VM with bhyve. It seems that I have to do this even if I don't change th kernel > between reboots. My first naive impression was that the point of bhyveload was > to load the kernel once. Seems it ain't so? bhyveload does the job of what BIOS/boot0/1/2/loader would do on real h/w, so it has to be executed each time on restart. One optimization to the cycle you mentioned is that bhyvectl --destroy only has to be done when the VM is no longer needed i.e you can loop with bhyveload/bhyve. later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 15:05:56 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5E7F5D9C for ; Tue, 2 Jun 2015 15:05:56 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id 1D76D1FAC for ; Tue, 2 Jun 2015 15:05:55 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id 72A4F1249E for ; Wed, 3 Jun 2015 01:05:54 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 6AD29280996 for ; Wed, 3 Jun 2015 01:05:54 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au X-Virus-Scanned: amavisd-new at iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Nvrv-D_PsP-m for ; Wed, 3 Jun 2015 01:05:54 +1000 (AEST) Received: from Peters-MacBook-Pro.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 8C0C2280992; Wed, 3 Jun 2015 01:05:52 +1000 (AEST) Message-ID: <556DC658.2030601@freebsd.org> Date: Tue, 02 Jun 2015 08:06:00 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Allan Jude CC: freebsd-virtualization@freebsd.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy References: <556D9163.1080704@FreeBSD.org> <556DC4D1.7070205@freebsd.org> In-Reply-To: <556DC4D1.7070205@freebsd.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 15:05:56 -0000 > Are you running -CURRENT, 10-STABLE, or 10.1? > > I think there was talk recently of removing the requirement to destroy > the VM before running bhyveload again. That's been the case in 10.1/STABLE and CURRENT for quite a while (the original change was r267216). later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 16:17:26 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 03CF24FB; Tue, 2 Jun 2015 16:17:26 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 1D98416B9; Tue, 2 Jun 2015 16:17:24 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA26294; Tue, 02 Jun 2015 19:17:22 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1YzosU-0002c2-G0; Tue, 02 Jun 2015 19:17:22 +0300 Message-ID: <556DD6EE.3050104@FreeBSD.org> Date: Tue, 02 Jun 2015 19:16:46 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Peter Grehan CC: freebsd-virtualization@FreeBSD.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy References: <556D9163.1080704@FreeBSD.org> <556DC2FE.1000308@freebsd.org> In-Reply-To: <556DC2FE.1000308@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 16:17:26 -0000 On 02/06/2015 17:51, Peter Grehan wrote: > Hi Andriy, > >> I am very new to bhyve, so sorry if I am asking something silly or obvious. >> I am using bhyve to speed up my testing and it seems that each time I need to >> restart a VM I need to go through the cycle of destroying it with bhyvectl >> --destroy, then re-loading a kernel with bhyveload and then actually booting the >> VM with bhyve. It seems that I have to do this even if I don't change th kernel >> between reboots. My first naive impression was that the point of bhyveload was >> to load the kernel once. Seems it ain't so? > > bhyveload does the job of what BIOS/boot0/1/2/loader would do on real h/w, so > it has to be executed each time on restart. > > One optimization to the cycle you mentioned is that bhyvectl --destroy only has > to be done when the VM is no longer needed i.e you can loop with bhyveload/bhyve. I see now. Thank you very much! BTW, and probably you are already aware of this, the documentation could use some work :-) I noticed at least the following: - bhyvectl is not documented - bhyve(8) and bhyveload(8) refer to vmm(4), which does not exist - bhyveload(8) does not mention that -d can be used multiple times - bhyve(8) has at least one incorrect reference to bhyveload(*4*) - bhyve(8) mentions vmnet - does that exist? But these are very minor things. bhyve rules :) -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 16:19:01 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 65DCC54D for ; Tue, 2 Jun 2015 16:19:01 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id 43F9116C0 for ; Tue, 2 Jun 2015 16:19:00 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [192.168.1.10] (unknown [192.168.1.10]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id E9D8B95D3 for ; Tue, 2 Jun 2015 16:18:59 +0000 (UTC) Message-ID: <556DD771.9080402@freebsd.org> Date: Tue, 02 Jun 2015 12:18:57 -0400 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-virtualization@freebsd.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy References: <556D9163.1080704@FreeBSD.org> <556DC2FE.1000308@freebsd.org> <556DD6EE.3050104@FreeBSD.org> In-Reply-To: <556DD6EE.3050104@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 16:19:01 -0000 On 2015-06-02 12:16, Andriy Gapon wrote: > On 02/06/2015 17:51, Peter Grehan wrote: >> Hi Andriy, >> >>> I am very new to bhyve, so sorry if I am asking something silly or obvious. >>> I am using bhyve to speed up my testing and it seems that each time I need to >>> restart a VM I need to go through the cycle of destroying it with bhyvectl >>> --destroy, then re-loading a kernel with bhyveload and then actually booting the >>> VM with bhyve. It seems that I have to do this even if I don't change th kernel >>> between reboots. My first naive impression was that the point of bhyveload was >>> to load the kernel once. Seems it ain't so? >> >> bhyveload does the job of what BIOS/boot0/1/2/loader would do on real h/w, so >> it has to be executed each time on restart. >> >> One optimization to the cycle you mentioned is that bhyvectl --destroy only has >> to be done when the VM is no longer needed i.e you can loop with bhyveload/bhyve. > > I see now. Thank you very much! > > BTW, and probably you are already aware of this, the documentation could use > some work :-) I noticed at least the following: > - bhyvectl is not documented > - bhyve(8) and bhyveload(8) refer to vmm(4), which does not exist > - bhyveload(8) does not mention that -d can be used multiple times > - bhyve(8) has at least one incorrect reference to bhyveload(*4*) > - bhyve(8) mentions vmnet - does that exist? > > But these are very minor things. bhyve rules :) > vmnet does exist, it is a type of tap, I'll get it hooked up as an alias to the tap man page. -- Allan Jude From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 16:46:08 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0B23CBB9 for ; Tue, 2 Jun 2015 16:46:08 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 378791E62 for ; Tue, 2 Jun 2015 16:46:07 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA26583 for ; Tue, 02 Jun 2015 19:46:05 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1YzpKH-0002cs-Hz for freebsd-virtualization@freebsd.org; Tue, 02 Jun 2015 19:46:05 +0300 Message-ID: <556DDDA9.6090005@FreeBSD.org> Date: Tue, 02 Jun 2015 19:45:29 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-virtualization@FreeBSD.org Subject: Re: bhyve: corrupting zfs pools? References: <556D9005.4020802@FreeBSD.org> In-Reply-To: <556D9005.4020802@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 16:46:08 -0000 On 02/06/2015 14:14, Andriy Gapon wrote: > > I am doing a simple experiment. > > I get FreeBSD image from here: > ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img.xz > > Then I run in bhyve with two additional "disks" created with truncate -s 4g: > $ bhyveload -m 1G -d > ~/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img test > $ bhyve -A -HP -s 0:0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -s > 3:0,virtio-blk,/home/avg/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img > -s 3:1,virtio-blk,/tmp/l2arc-test/hdd1,sectorsize=512/4096 -s > 3:2,virtio-blk,/tmp/l2arc-test/hdd2,sectorsize=512/4096 -l com1,stdio -l > com2,/dev/nmdm0A -c 2 -m 1g test > > Note sectorsize=512/4096 options. Not sure if it's them that cause the trouble. > > Then, in the VM: > $ zpool create l2arc-test mirror /dev/vtbd1 /dev/vtbd2 > $ zfs create -p l2arc-test/ROOT/initial > $ tar -c --one-file-system -f - / | tar -x -C /l2arc-test/ROOT/initial -f - > > Afterwards, zpool status -v reports no problem. > But then I run zpool scrub and get the following in the end: > $ zpool status -v > pool: l2arc-test > state: ONLINE > status: One or more devices has experienced an error resulting in data > corruption. Applications may be affected. > action: Restore the file in question if possible. Otherwise restore the > entire pool from backup. > see: http://illumos.org/msg/ZFS-8000-8A > scan: scrub repaired 356K in 0h0m with 9 errors on Tue Jun 2 13:58:17 2015 > config: > > NAME STATE READ WRITE CKSUM > l2arc-test ONLINE 0 0 9 > mirror-0 ONLINE 0 0 18 > vtbd1 ONLINE 0 0 25 > vtbd2 ONLINE 0 0 23 > > errors: Permanent errors have been detected in the following files: > > /l2arc-test/ROOT/initial/usr/bin/svnlitesync > /l2arc-test/ROOT/initial/usr/freebsd-dist/kernel.txz > /l2arc-test/ROOT/initial/usr/freebsd-dist/src.txz > > /l2arc-test/ROOT/initial/usr/lib/clang/3.6.1/lib/freebsd/libclang_rt.asan-x86_64.a > > > The same issue is reproducible with ahci-hd. > > My host system is a recent amd64 CURRENT as well. The hardware platform is AMD. > I used the following monstrous command line to reproduce the test in qemu: $ qemu-system-x86_64 -smp 2 -m 1024 -drive file=/tmp/livecd2/R2.img,format=raw,if=none,id=bootd -device virtio-blk-pci,drive=bootd -drive file=/tmp/l2arc-test/hdd1,if=none,id=hdd1,format=raw -device virtio-blk-pci,drive=hdd1,logical_block_size=4096 -drive file=/tmp/l2arc-test/hdd2,id=hdd2,if=none,format=raw -device virtio-blk-pci,drive=hdd2,logical_block_size=4096 -drive file=/tmp/l2arc-test/ssd,id=ssd,if=none,format=raw -device virtio-blk-pci,drive=ssd,logical_block_size=4096 ... And several other variations of logical_block_size and physical_block_size. The tests a re very slow, but there are no checksum errors. So, I suspect guest memory corruption caused by bhyve. Perhaps the problem is indeed specific to AMD-V. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 16:49:51 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 45835CA6; Tue, 2 Jun 2015 16:49:51 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 153D11EA2; Tue, 2 Jun 2015 16:49:50 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t52Gnn2q090185 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 2 Jun 2015 09:49:49 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t52GnnRL090184; Tue, 2 Jun 2015 09:49:49 -0700 (PDT) (envelope-from jmg) Date: Tue, 2 Jun 2015 09:49:49 -0700 From: John-Mark Gurney To: Andriy Gapon Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy Message-ID: <20150602164949.GZ50817@funkthat.com> References: <556D9163.1080704@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <556D9163.1080704@FreeBSD.org> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Tue, 02 Jun 2015 09:49:49 -0700 (PDT) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 16:49:51 -0000 Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 14:20 +0300: > I am very new to bhyve, so sorry if I am asking something silly or obvious. > I am using bhyve to speed up my testing and it seems that each time I need to > restart a VM I need to go through the cycle of destroying it with bhyvectl > --destroy, then re-loading a kernel with bhyveload and then actually booting the > VM with bhyve. It seems that I have to do this even if I don't change th kernel > between reboots. My first naive impression was that the point of bhyveload was > to load the kernel once. Seems it ain't so? Hmm... I'm not seeing that here... I just scp a new kernel into the vm, install it, and run shutdown -r now which drops bhyve back to loader, and loads the new kernel... I've been doing this quite successfully over the last few months... I am running a month old HEAD though... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 18:16:07 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DC9C8287 for ; Tue, 2 Jun 2015 18:16:07 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 30B42157C for ; Tue, 2 Jun 2015 18:16:06 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id VAA27507; Tue, 02 Jun 2015 21:16:03 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1YzqjL-0002gY-LP; Tue, 02 Jun 2015 21:16:03 +0300 Message-ID: <556DF2AB.8070507@FreeBSD.org> Date: Tue, 02 Jun 2015 21:15:07 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: John-Mark Gurney CC: freebsd-virtualization@FreeBSD.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy References: <556D9163.1080704@FreeBSD.org> <20150602164949.GZ50817@funkthat.com> In-Reply-To: <20150602164949.GZ50817@funkthat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 18:16:07 -0000 On 02/06/2015 19:49, John-Mark Gurney wrote: > Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 14:20 +0300: >> I am very new to bhyve, so sorry if I am asking something silly or obvious. >> I am using bhyve to speed up my testing and it seems that each time I need to >> restart a VM I need to go through the cycle of destroying it with bhyvectl >> --destroy, then re-loading a kernel with bhyveload and then actually booting the >> VM with bhyve. It seems that I have to do this even if I don't change th kernel >> between reboots. My first naive impression was that the point of bhyveload was >> to load the kernel once. Seems it ain't so? > > Hmm... I'm not seeing that here... I just scp a new kernel into the > vm, install it, and run shutdown -r now which drops bhyve back to > loader, and loads the new kernel... I've been doing this quite > successfully over the last few months... > > I am running a month old HEAD though... > I guess you are running bhyve through the shell script vmrun.sh? I am doing everything by hand. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 21:40:54 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BB392BB3; Tue, 2 Jun 2015 21:40:54 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wg0-x22e.google.com (mail-wg0-x22e.google.com [IPv6:2a00:1450:400c:c00::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 572BF16B7; Tue, 2 Jun 2015 21:40:54 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wgme6 with SMTP id e6so150890321wgm.2; Tue, 02 Jun 2015 14:40:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=AqszvN3bekAfkFZa2VCt1U89CykEgQ3bB5eBFadirAY=; b=uHptRQnwildZm0PTN7cK9gJNZjOlI/RIXLTjJtUeVsm1EVQFIz6nVv1/c3CVTYieJr YOlyloVqAun+erylm2ZpOLoY5Q1bhM5MQrMqayF/72vPGoowfRxpLysiZYLCWZXMYejk pzO9s2qZQPlP0VW6RZzopguyQi0z33h9GVvSdOJCf7aPhgBCS7XcxnM8uuZ4AkPgMYo4 bAgQqFdG/ogrgs2QZ0Okt/ft0JBUNStHcBkjGwTx2xVtxTO+R5J0I3PynEs4ugcUmOgR MTIKiAwVScPl+NleEYdlVnF2HYRtlSimzux8XCTTsllFfW6a1ewzrLV26wdMY9dGX1w5 qExw== MIME-Version: 1.0 X-Received: by 10.194.62.201 with SMTP id a9mr53841138wjs.63.1433281251768; Tue, 02 Jun 2015 14:40:51 -0700 (PDT) Received: by 10.27.52.18 with HTTP; Tue, 2 Jun 2015 14:40:51 -0700 (PDT) In-Reply-To: <556DDDA9.6090005@FreeBSD.org> References: <556D9005.4020802@FreeBSD.org> <556DDDA9.6090005@FreeBSD.org> Date: Tue, 2 Jun 2015 14:40:51 -0700 Message-ID: Subject: Re: bhyve: corrupting zfs pools? From: Neel Natu To: Andriy Gapon Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 21:40:54 -0000 Hi Andriy, On Tue, Jun 2, 2015 at 9:45 AM, Andriy Gapon wrote: > On 02/06/2015 14:14, Andriy Gapon wrote: >> >> I am doing a simple experiment. >> >> I get FreeBSD image from here: >> ftp://ftp.freebsd.org/pub/FreeBSD/snapshots/ISO-IMAGES/11.0/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img.xz >> >> Then I run in bhyve with two additional "disks" created with truncate -s 4g: >> $ bhyveload -m 1G -d >> ~/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img test >> $ bhyve -A -HP -s 0:0,hostbridge -s 1,lpc -s 2:0,virtio-net,tap0 -s >> 3:0,virtio-blk,/home/avg/tmp/FreeBSD-11.0-CURRENT-amd64-r283577-20150526-memstick.img >> -s 3:1,virtio-blk,/tmp/l2arc-test/hdd1,sectorsize=512/4096 -s >> 3:2,virtio-blk,/tmp/l2arc-test/hdd2,sectorsize=512/4096 -l com1,stdio -l >> com2,/dev/nmdm0A -c 2 -m 1g test >> >> Note sectorsize=512/4096 options. Not sure if it's them that cause the trouble. >> >> Then, in the VM: >> $ zpool create l2arc-test mirror /dev/vtbd1 /dev/vtbd2 >> $ zfs create -p l2arc-test/ROOT/initial >> $ tar -c --one-file-system -f - / | tar -x -C /l2arc-test/ROOT/initial -f - >> >> Afterwards, zpool status -v reports no problem. >> But then I run zpool scrub and get the following in the end: >> $ zpool status -v >> pool: l2arc-test >> state: ONLINE >> status: One or more devices has experienced an error resulting in data >> corruption. Applications may be affected. >> action: Restore the file in question if possible. Otherwise restore the >> entire pool from backup. >> see: http://illumos.org/msg/ZFS-8000-8A >> scan: scrub repaired 356K in 0h0m with 9 errors on Tue Jun 2 13:58:17 2015 >> config: >> >> NAME STATE READ WRITE CKSUM >> l2arc-test ONLINE 0 0 9 >> mirror-0 ONLINE 0 0 18 >> vtbd1 ONLINE 0 0 25 >> vtbd2 ONLINE 0 0 23 >> >> errors: Permanent errors have been detected in the following files: >> >> /l2arc-test/ROOT/initial/usr/bin/svnlitesync >> /l2arc-test/ROOT/initial/usr/freebsd-dist/kernel.txz >> /l2arc-test/ROOT/initial/usr/freebsd-dist/src.txz >> >> /l2arc-test/ROOT/initial/usr/lib/clang/3.6.1/lib/freebsd/libclang_rt.asan-x86_64.a >> >> >> The same issue is reproducible with ahci-hd. >> >> My host system is a recent amd64 CURRENT as well. The hardware platform is AMD. >> > > I used the following monstrous command line to reproduce the test in qemu: > $ qemu-system-x86_64 -smp 2 -m 1024 -drive > file=/tmp/livecd2/R2.img,format=raw,if=none,id=bootd -device > virtio-blk-pci,drive=bootd -drive > file=/tmp/l2arc-test/hdd1,if=none,id=hdd1,format=raw -device > virtio-blk-pci,drive=hdd1,logical_block_size=4096 -drive > file=/tmp/l2arc-test/hdd2,id=hdd2,if=none,format=raw -device > virtio-blk-pci,drive=hdd2,logical_block_size=4096 -drive > file=/tmp/l2arc-test/ssd,id=ssd,if=none,format=raw -device > virtio-blk-pci,drive=ssd,logical_block_size=4096 ... > > And several other variations of logical_block_size and physical_block_size. > The tests a re very slow, but there are no checksum errors. > > So, I suspect guest memory corruption caused by bhyve. Perhaps the problem is > indeed specific to AMD-V. > Perhaps, but I wasn't able to repro this. I tried your recipe to repro on two systems running -current at r283917. - Intel Sandybridge server: Xeon E52650 with 8 cores/16 threads and 64GB memory - hdd1/hdd2 on UFS - hdd1/hdd2 on ZFS - AMD Opteron server: Opteron 6230 with 8 cores and 16GB memory - hdd1/hdd2 on UFS Can you provide some more details about your setup? I can then try to repro on a system similar to your setup. - uname -a - sysctl hw.model - sysctl hw.ncpu - sysctl hw.physmem - sysctl hw.vmm - host filesystem underlying hdd1 and hdd2 best Neel > -- > Andriy Gapon > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 23:04:18 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0DE23CA; Tue, 2 Jun 2015 23:04:18 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C33ED1B08; Tue, 2 Jun 2015 23:04:17 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t52N4FJL099383 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 2 Jun 2015 16:04:15 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t52N4Fcc099382; Tue, 2 Jun 2015 16:04:15 -0700 (PDT) (envelope-from jmg) Date: Tue, 2 Jun 2015 16:04:15 -0700 From: John-Mark Gurney To: Andriy Gapon Cc: freebsd-virtualization@FreeBSD.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy Message-ID: <20150602230415.GC50817@funkthat.com> References: <556D9163.1080704@FreeBSD.org> <20150602164949.GZ50817@funkthat.com> <556DF2AB.8070507@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <556DF2AB.8070507@FreeBSD.org> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Tue, 02 Jun 2015 16:04:16 -0700 (PDT) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 23:04:18 -0000 Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300: > On 02/06/2015 19:49, John-Mark Gurney wrote: > > Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 14:20 +0300: > >> I am very new to bhyve, so sorry if I am asking something silly or obvious. > >> I am using bhyve to speed up my testing and it seems that each time I need to > >> restart a VM I need to go through the cycle of destroying it with bhyvectl > >> --destroy, then re-loading a kernel with bhyveload and then actually booting the > >> VM with bhyve. It seems that I have to do this even if I don't change th kernel > >> between reboots. My first naive impression was that the point of bhyveload was > >> to load the kernel once. Seems it ain't so? > > > > Hmm... I'm not seeing that here... I just scp a new kernel into the > > vm, install it, and run shutdown -r now which drops bhyve back to > > loader, and loads the new kernel... I've been doing this quite > > successfully over the last few months... > > > > I am running a month old HEAD though... > > I guess you are running bhyve through the shell script vmrun.sh? > I am doing everything by hand. Correct: sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0 It's nice.. shutdown -r now and shutdown -p now both work exactly as you'd expect them to... :) -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-virtualization@FreeBSD.ORG Tue Jun 2 23:09:55 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5517D236 for ; Tue, 2 Jun 2015 23:09:55 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id 339D91B28 for ; Tue, 2 Jun 2015 23:09:54 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [192.168.1.10] (unknown [192.168.1.10]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id A7ADB98EE for ; Tue, 2 Jun 2015 23:09:53 +0000 (UTC) Message-ID: <556E37BF.8090901@freebsd.org> Date: Tue, 02 Jun 2015 19:09:51 -0400 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-virtualization@freebsd.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy References: <556D9163.1080704@FreeBSD.org> <20150602164949.GZ50817@funkthat.com> <556DF2AB.8070507@FreeBSD.org> <20150602230415.GC50817@funkthat.com> In-Reply-To: <20150602230415.GC50817@funkthat.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jun 2015 23:09:55 -0000 On 2015-06-02 19:04, John-Mark Gurney wrote: > Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300: >> On 02/06/2015 19:49, John-Mark Gurney wrote: >>> Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 14:20 +0300: >>>> I am very new to bhyve, so sorry if I am asking something silly or obvious. >>>> I am using bhyve to speed up my testing and it seems that each time I need to >>>> restart a VM I need to go through the cycle of destroying it with bhyvectl >>>> --destroy, then re-loading a kernel with bhyveload and then actually booting the >>>> VM with bhyve. It seems that I have to do this even if I don't change th kernel >>>> between reboots. My first naive impression was that the point of bhyveload was >>>> to load the kernel once. Seems it ain't so? >>> >>> Hmm... I'm not seeing that here... I just scp a new kernel into the >>> vm, install it, and run shutdown -r now which drops bhyve back to >>> loader, and loads the new kernel... I've been doing this quite >>> successfully over the last few months... >>> >>> I am running a month old HEAD though... >> >> I guess you are running bhyve through the shell script vmrun.sh? >> I am doing everything by hand. > > Correct: > sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0 > > It's nice.. shutdown -r now and shutdown -p now both work exactly as > you'd expect them to... :) > yes, vmrun.sh puts bhyve in a while loop. -- Allan Jude From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 3 07:11:49 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 17BEA822 for ; Wed, 3 Jun 2015 07:11:49 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 5DD961B09 for ; Wed, 3 Jun 2015 07:11:47 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA06034; Wed, 03 Jun 2015 10:11:45 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z02q0-0003BE-LM; Wed, 03 Jun 2015 10:11:44 +0300 Message-ID: <556EA85F.1010907@FreeBSD.org> Date: Wed, 03 Jun 2015 10:10:23 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: John-Mark Gurney CC: freebsd-virtualization@FreeBSD.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy References: <556D9163.1080704@FreeBSD.org> <20150602164949.GZ50817@funkthat.com> <556DF2AB.8070507@FreeBSD.org> <20150602230415.GC50817@funkthat.com> In-Reply-To: <20150602230415.GC50817@funkthat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 07:11:49 -0000 On 03/06/2015 02:04, John-Mark Gurney wrote: > Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300: >> I guess you are running bhyve through the shell script vmrun.sh? >> I am doing everything by hand. > > Correct: > sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0 > > It's nice.. shutdown -r now and shutdown -p now both work exactly as > you'd expect them to... :) > Yes, it's quite convenient. The only drawback is that currently it can't pass multiple -d options to bhyveload and I need that from time to time. Probably not hard to change that. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 3 10:02:46 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 870B75F8 for ; Wed, 3 Jun 2015 10:02:46 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id CB9691431 for ; Wed, 3 Jun 2015 10:02:45 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA08343; Wed, 03 Jun 2015 13:02:43 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z05VS-0003HZ-UJ; Wed, 03 Jun 2015 13:02:42 +0300 Message-ID: <556ED071.5030009@FreeBSD.org> Date: Wed, 03 Jun 2015 13:01:21 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Neel Natu CC: "freebsd-virtualization@freebsd.org" Subject: Re: bhyve: corrupting zfs pools? References: <556D9005.4020802@FreeBSD.org> <556DDDA9.6090005@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 10:02:46 -0000 On 03/06/2015 00:40, Neel Natu wrote: > Perhaps, but I wasn't able to repro this. I tried your recipe to repro > on two systems running -current at r283917. > > - Intel Sandybridge server: Xeon E52650 with 8 cores/16 threads and 64GB memory > - hdd1/hdd2 on UFS > - hdd1/hdd2 on ZFS > > - AMD Opteron server: Opteron 6230 with 8 cores and 16GB memory > - hdd1/hdd2 on UFS > > Can you provide some more details about your setup? I can then try to > repro on a system similar to your setup. > > - uname -a FreeBSD trant 11.0-CURRENT FreeBSD 11.0-CURRENT #134 r283188+ab4f83f(devel): Fri May 22 15:55:27 EEST 2015 avg@trant:/usr/obj/usr/src/sys/TRANT amd64 This is not a pure FreeBSD, there are some local changes, but none to vmm or VM. > - sysctl hw.model hw.model: AMD Athlon(tm) II X2 250 Processor > - sysctl hw.ncpu hw.ncpu: 2 > - sysctl hw.physmem hw.physmem: 8029335552 > - sysctl hw.vmm hw.vmm.npt.pmap_flags: 507 hw.vmm.svm.num_asids: 64 hw.vmm.svm.disable_npf_assist: 0 hw.vmm.svm.features: 15 hw.vmm.svm.vmcb_clean: 959 hw.vmm.vmx.vpid_alloc_failed: 0 hw.vmm.vmx.posted_interrupt_vector: -1 hw.vmm.vmx.cap.posted_interrupts: 0 hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 hw.vmm.vmx.cap.invpcid: 0 hw.vmm.vmx.cap.monitor_trap: 0 hw.vmm.vmx.cap.unrestricted_guest: 0 hw.vmm.vmx.cap.pause_exit: 0 hw.vmm.vmx.cap.halt_exit: 0 hw.vmm.vmx.initialized: 0 hw.vmm.vmx.cr4_zeros_mask: 0 hw.vmm.vmx.cr4_ones_mask: 0 hw.vmm.vmx.cr0_zeros_mask: 0 hw.vmm.vmx.cr0_ones_mask: 0 hw.vmm.ept.pmap_flags: 0 hw.vmm.vrtc.flag_broken_time: 1 hw.vmm.ppt.devices: 0 hw.vmm.iommu.initialized: 0 hw.vmm.bhyve_xcpuids: 4 hw.vmm.topology.cpuid_leaf_b: 1 hw.vmm.topology.cores_per_package: 1 hw.vmm.topology.threads_per_core: 1 hw.vmm.create: beavis hw.vmm.destroy: beavis hw.vmm.force_iommu: 0 hw.vmm.trace_guest_exceptions: 0 hw.vmm.ipinum: 251 hw.vmm.halt_detection: 1 > - host filesystem underlying hdd1 and hdd2 It's ZFS. Thank you! -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 3 10:03:52 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8F607795 for ; Wed, 3 Jun 2015 10:03:52 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id BD8FC1443 for ; Wed, 3 Jun 2015 10:03:50 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id NAA08358 for ; Wed, 03 Jun 2015 13:03:49 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z05WX-0003Hc-Dk for freebsd-virtualization@freebsd.org; Wed, 03 Jun 2015 13:03:49 +0300 Message-ID: <556ED0CD.30705@FreeBSD.org> Date: Wed, 03 Jun 2015 13:02:53 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: "freebsd-virtualization@freebsd.org" Subject: bhyve and illumos Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 10:03:52 -0000 Has anyone tried to boot an illumos-based OS in bhyve? Any success stories? Thanks! -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 3 15:26:19 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4A27B2A2 for ; Wed, 3 Jun 2015 15:26:19 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id 09BDB1436 for ; Wed, 3 Jun 2015 15:26:18 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id BF7171268B for ; Thu, 4 Jun 2015 01:26:10 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id B4AA4280999 for ; Thu, 4 Jun 2015 01:26:10 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id YuuVPiO3dBHb for ; Thu, 4 Jun 2015 01:26:10 +1000 (AEST) Received: from Peters-MacBook-Pro.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id D2044280996; Thu, 4 Jun 2015 01:26:08 +1000 (AEST) Message-ID: <556F1C90.6040405@freebsd.org> Date: Wed, 03 Jun 2015 08:26:08 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Andriy Gapon CC: "freebsd-virtualization@freebsd.org" Subject: Re: bhyve and illumos References: <556ED0CD.30705@FreeBSD.org> In-Reply-To: <556ED0CD.30705@FreeBSD.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 15:26:19 -0000 Hi Andriy, > Has anyone tried to boot an illumos-based OS in bhyve? > Any success stories? The Illumos kernel makes some BIOS calls so it can't be booted with grub-bhyve/multiboot - a triple-fault will result on the first BIOS call. It will need the UEFI/CSM firwmare work that's currently being worked on. later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 3 15:53:07 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 4062FC5F; Wed, 3 Jun 2015 15:53:07 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client CN "gold.funkthat.com", Issuer "gold.funkthat.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id EF8851B7B; Wed, 3 Jun 2015 15:53:06 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.14.5/8.14.5) with ESMTP id t53Fr56V010582 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 3 Jun 2015 08:53:05 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.14.5/8.14.5/Submit) id t53Fr5TZ010581; Wed, 3 Jun 2015 08:53:05 -0700 (PDT) (envelope-from jmg) Date: Wed, 3 Jun 2015 08:53:05 -0700 From: John-Mark Gurney To: Andriy Gapon Cc: freebsd-virtualization@FreeBSD.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy Message-ID: <20150603155305.GG50817@funkthat.com> References: <556D9163.1080704@FreeBSD.org> <20150602164949.GZ50817@funkthat.com> <556DF2AB.8070507@FreeBSD.org> <20150602230415.GC50817@funkthat.com> <556EA85F.1010907@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <556EA85F.1010907@FreeBSD.org> X-Operating-System: FreeBSD 9.1-PRERELEASE amd64 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.5.21 (2010-09-15) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (gold.funkthat.com [127.0.0.1]); Wed, 03 Jun 2015 08:53:05 -0700 (PDT) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 15:53:07 -0000 Andriy Gapon wrote this message on Wed, Jun 03, 2015 at 10:10 +0300: > On 03/06/2015 02:04, John-Mark Gurney wrote: > > Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300: > >> I guess you are running bhyve through the shell script vmrun.sh? > >> I am doing everything by hand. > > > > Correct: > > sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0 > > > > It's nice.. shutdown -r now and shutdown -p now both work exactly as > > you'd expect them to... :) > > Yes, it's quite convenient. The only drawback is that currently it can't pass > multiple -d options to bhyveload and I need that from time to time. Hmm... Looking at the script, it looks like you can (at least on HEAD): d) eval "disk_dev${disk_total}=\"${OPTARG}\"" disk_total=$(($disk_total + 1)) > Probably not hard to change that. It'd also be nice to allow you to switch the default from virtio-blk to ahci when you're image doesn't support it.. Hmm.. It wouldn't be hard to try to use environment vars for the defaults either... Maybe it's time to move vmrun.sh to bhyverun, install it and give it a proper man page? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 3 16:19:37 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0DD0CAD2 for ; Wed, 3 Jun 2015 16:19:37 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 511BE10D4 for ; Wed, 3 Jun 2015 16:19:35 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA13004; Wed, 03 Jun 2015 19:19:33 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z0BO9-0003WE-62; Wed, 03 Jun 2015 19:19:33 +0300 Message-ID: <556F28C4.8050100@FreeBSD.org> Date: Wed, 03 Jun 2015 19:18:12 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: John-Mark Gurney CC: freebsd-virtualization@FreeBSD.org Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy References: <556D9163.1080704@FreeBSD.org> <20150602164949.GZ50817@funkthat.com> <556DF2AB.8070507@FreeBSD.org> <20150602230415.GC50817@funkthat.com> <556EA85F.1010907@FreeBSD.org> <20150603155305.GG50817@funkthat.com> In-Reply-To: <20150603155305.GG50817@funkthat.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 16:19:37 -0000 On 03/06/2015 18:53, John-Mark Gurney wrote: > Andriy Gapon wrote this message on Wed, Jun 03, 2015 at 10:10 +0300: >> On 03/06/2015 02:04, John-Mark Gurney wrote: >>> Andriy Gapon wrote this message on Tue, Jun 02, 2015 at 21:15 +0300: >>>> I guess you are running bhyve through the shell script vmrun.sh? >>>> I am doing everything by hand. >>> >>> Correct: >>> sh /usr/share/examples/bhyve/vmrun.sh -g 6444 -d mach10s.img -t tap0 >>> >>> It's nice.. shutdown -r now and shutdown -p now both work exactly as >>> you'd expect them to... :) >> >> Yes, it's quite convenient. The only drawback is that currently it can't pass >> multiple -d options to bhyveload and I need that from time to time. > > Hmm... Looking at the script, it looks like you can (at least on HEAD): > d) > eval "disk_dev${disk_total}=\"${OPTARG}\"" > disk_total=$(($disk_total + 1)) This was only half of the equation, please see this: https://reviews.freebsd.org/D2723 >> Probably not hard to change that. > > It'd also be nice to allow you to switch the default from virtio-blk > to ahci when you're image doesn't support it.. > > Hmm.. It wouldn't be hard to try to use environment vars for the defaults > either... > > Maybe it's time to move vmrun.sh to bhyverun, install it and give > it a proper man page? JIMO, bhyveload(8) could be just merged into bhyve(8) and the latter should behave like vmrun.sh does. bhyve(8) should retain an option to run a preloaded kernel, so that things like bhyve-grub keep working. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 3 17:12:19 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 729C1972 for ; Wed, 3 Jun 2015 17:12:19 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-ob0-x22c.google.com (mail-ob0-x22c.google.com [IPv6:2607:f8b0:4003:c01::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3C0081FCA for ; Wed, 3 Jun 2015 17:12:19 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by obew15 with SMTP id w15so13222382obe.1 for ; Wed, 03 Jun 2015 10:12:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=kDxLi7fyKX6oMHp/yn2AcMDqiDsfd3zpByWTow9DL3Y=; b=yrzEihRj7fd1jGYZpjbDfEZ0Yds85VCyruLERj7RwyBee2w8r27UAw8ZwRq8tErbSQ uFZWSE76hN0r22H07qp4B/nohTAlEJsvMKPkN1Bel6pqx95WRaA3nG9BUTzJT67m/N6A w0Ie2jPqlLXW5X6Kml0HUBlhTnSs8idNuK2cyTM7cPYErHHr/BdxSWIymhZhwUscTceE X7qMzSngL+CCIQFhdac7mlqpBS3E3p4+I8rER2EIrnha999FdcwuIoICOleib9e41ZSV gT/mdACU4RRr84spTQei26/ju66Ceg8yptwuh+2n83X9E6xLsTthrVvmxzBlbGK8APVR HBUg== MIME-Version: 1.0 X-Received: by 10.60.132.208 with SMTP id ow16mr28351079oeb.66.1433351538254; Wed, 03 Jun 2015 10:12:18 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Wed, 3 Jun 2015 10:12:18 -0700 (PDT) Date: Wed, 3 Jun 2015 20:12:18 +0300 Message-ID: Subject: Where are the VCPU threads created? From: Stefan Andritoiu To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 17:12:19 -0000 I see that in the main() function of bhyverun.c it adds CPU0 by calling fbsdrun_addcpu(ctx, 0, 0, rip). But I can't find where the other VCPUs are added. Or if the 'guest_ncpus' variable is ever used (except for error checks) From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 3 17:17:20 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 81ECBA70 for ; Wed, 3 Jun 2015 17:17:20 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by mx1.freebsd.org (Postfix) with ESMTP id 3E444102D for ; Wed, 3 Jun 2015 17:17:19 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id ADC791271D for ; Thu, 4 Jun 2015 03:17:17 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 9AEE828099B for ; Thu, 4 Jun 2015 03:17:17 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Cg-yrv6hJ3ck for ; Thu, 4 Jun 2015 03:17:17 +1000 (AEST) Received: from Peters-MacBook-Pro.local (c-50-184-135-69.hsd1.ca.comcast.net [50.184.135.69]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id BEB0A280999; Thu, 4 Jun 2015 03:17:14 +1000 (AEST) Message-ID: <556F3699.7000609@freebsd.org> Date: Wed, 03 Jun 2015 10:17:13 -0700 From: Peter Grehan User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Stefan Andritoiu CC: freebsd-virtualization@freebsd.org Subject: Re: Where are the VCPU threads created? References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 17:17:20 -0000 Hi Stefan, > I see that in the main() function of bhyverun.c it adds CPU0 by > calling fbsdrun_addcpu(ctx, 0, 0, rip). But I can't find where the > other VCPUs are added. Or if the 'guest_ncpus' variable is ever used > (except for error checks) Have a look at bhyve/spinup_ap.c:spinup_ap() which is called when a the vmm kernel module detects a SIPI message being sent to the local APIC and forces a vm-exit to userspace. later, Peter. From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 3 17:38:43 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1A0D45D9; Wed, 3 Jun 2015 17:38:43 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: from mail-yh0-x229.google.com (mail-yh0-x229.google.com [IPv6:2607:f8b0:4002:c01::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C9AAD15CF; Wed, 3 Jun 2015 17:38:42 +0000 (UTC) (envelope-from crodr001@gmail.com) Received: by yhda23 with SMTP id a23so3813693yhd.2; Wed, 03 Jun 2015 10:38:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Wy/B5IFlWwenP5zPm7VRSAHV1JGvbih1hKtPVxaulkA=; b=EHipf5MptbSskym/x3TQenyJfAu9QFJqVMOAvSvY9oz0+Ocu8PWVdxYKkA1VcMNLM5 gwneVPXV1zgaClgQnNPFt5HezSexYGKrBbMEtxJGL9wS2hUleKWxJtUGapjoP6lRoEYO h/RxsPX5Ug2LzX2tGgvjsFj7o0qDFZ97mE+u+u9dV21UZC5F5QnQxaZ/KCzPqUsfkEhp Nm3grFwEL3TeMO5tZEUFft7631BEkfsiCGUhNKuLxNjChGXGcoPYv8dEkoIzmB1e2m9g 5VcncI1kmav8A2S05BGh6YsKUzvcm0vFj37euGQaUwmY4i1h++POnqgXLtjv21bWgXVR DqnA== MIME-Version: 1.0 X-Received: by 10.170.125.208 with SMTP id r199mr14319475ykb.15.1433353121823; Wed, 03 Jun 2015 10:38:41 -0700 (PDT) Sender: crodr001@gmail.com Received: by 10.13.233.70 with HTTP; Wed, 3 Jun 2015 10:38:41 -0700 (PDT) In-Reply-To: <556F28C4.8050100@FreeBSD.org> References: <556D9163.1080704@FreeBSD.org> <20150602164949.GZ50817@funkthat.com> <556DF2AB.8070507@FreeBSD.org> <20150602230415.GC50817@funkthat.com> <556EA85F.1010907@FreeBSD.org> <20150603155305.GG50817@funkthat.com> <556F28C4.8050100@FreeBSD.org> Date: Wed, 3 Jun 2015 10:38:41 -0700 X-Google-Sender-Auth: ir3GjImXNYQbFmX7skdF6JYMkHo Message-ID: Subject: Re: bhyve: bhyveload, bhyve, bhyvectl --destroy From: Craig Rodrigues To: Andriy Gapon Cc: John-Mark Gurney , "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 17:38:43 -0000 On Wed, Jun 3, 2015 at 9:18 AM, Andriy Gapon wrote: > On 03/06/2015 18:53, John-Mark Gurney wrote: > > JIMO, bhyveload(8) could be just merged into bhyve(8) and the latter should > behave like vmrun.sh does. bhyve(8) should retain an option to run a > preloaded > kernel, so that things like bhyve-grub keep working. > I understand why bhyveload(8) and bhyve(8) exists. However, from a usability perspective, the fact that I need to invoke two commands to start a VM is unfortunate. >From a end-user usability standpoint, this is no good. vmrun.sh is not bad for simple usage, but its limitations become apparent quite quickly. For anyone doing serious stuff with bhyve, you end up having to write your own replacement for vmrun.sh. I've done it, and I am sure others have as well. What I would like to see is a single command that I can type with all the command-line options to start a bhyve VM, instead of multiple commands that need to be stitched together via a script. For example, if bhyve(8) could be enhanced with a flag "--loader" (or choose another flag name that you prefer): (1) If "--loader" is not specified, then by default, bhyve uses bhyveload (2) If "--loader" is specified, then valid options are bhyveload and grub-bhyve (3) bhyve(8) will internally fork the loader specified in (1) or (2) (4) If (3) fails, then stop. If (3) succeeds, then bhyve(8) will continue to start the VM I would also like to see: -> command-line option parsing for bhyveload and grub-bhyve changed, so that they silently ignore command-line options that they don't grok. That would make it easier to implement (3), since bhyve(8) could pass its full command-line arguments to the forked loader. I don't have time to work on this, but it would be nice if this could be implemented. -- Craig From owner-freebsd-virtualization@FreeBSD.ORG Wed Jun 3 22:41:24 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 393B5E52; Wed, 3 Jun 2015 22:41:24 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 015FD11AB; Wed, 3 Jun 2015 22:41:24 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by obbea3 with SMTP id ea3so19841248obb.0; Wed, 03 Jun 2015 15:41:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=BYkc+8YYXHrfpeD5AALbzLdDYdYWSM3lUidee1gQGxk=; b=nqwf7uXEwOpgYEbKqL4yUbyTdFHVQUESyxQ2nSb1yPE/Nd3Z+STL3uLZ1qlXcZXvd5 OgM6db2Nbv+L4ItH/LrsTckvJvC91mR4fmLCT8R62kVbhlN1EJbw8E9yV2BKemWoDI9z Mq3DM7NTlhER8xzvcpG+VcIjGqAK74jGZsfAOgpVUqCZlyRW4P8qdE9RDHsPtmGditRl iC6Q80/lHG089pCZf8xC6P0DeRWl7RF3XZFeiukYdQiG2Ml7qP73Ij5qCdfIjeHOCAvj 0k2bBAOZB+e3+2jfIMjHLALVMD7RXoa07sJ5LdcLwet9CDjV7xvaiV/WGsgvfUALxmc3 7XUg== MIME-Version: 1.0 X-Received: by 10.60.132.208 with SMTP id ow16mr29488343oeb.66.1433371283034; Wed, 03 Jun 2015 15:41:23 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Wed, 3 Jun 2015 15:41:22 -0700 (PDT) In-Reply-To: <556F3699.7000609@freebsd.org> References: <556F3699.7000609@freebsd.org> Date: Thu, 4 Jun 2015 01:41:22 +0300 Message-ID: Subject: Re: Where are the VCPU threads created? From: Stefan Andritoiu To: Peter Grehan Cc: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jun 2015 22:41:24 -0000 On Wed, Jun 3, 2015 at 8:17 PM, Peter Grehan wrote: >> I see that in the main() function of bhyverun.c it adds CPU0 by >> calling fbsdrun_addcpu(ctx, 0, 0, rip). But I can't find where the >> other VCPUs are added. Or if the 'guest_ncpus' variable is ever used >> (except for error checks) > > > Have a look at bhyve/spinup_ap.c:spinup_ap() which is called when a the vmm > kernel module detects a SIPI message being sent to the local APIC and forces > a vm-exit to userspace. Hi Peter, Thank you, that answered my question. Another thing: Are these threads (the ones that run the VCPUs) ever destroyed and recreated? In the timespan between the moment they are created (when the guest starts) and when the guest is shut down, is there any chance of the threads ending/being killed and others created to take their place? Or do you end up with the same threads? I am trying to add some information to the VCPU thread at the moment it is created by bhyve, and want that information to be available for the entirety of the guest's lifespan. From owner-freebsd-virtualization@FreeBSD.ORG Thu Jun 4 04:37:06 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 121D9648; Thu, 4 Jun 2015 04:37:06 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wg0-x232.google.com (mail-wg0-x232.google.com [IPv6:2a00:1450:400c:c00::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9F410123C; Thu, 4 Jun 2015 04:37:05 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wgez8 with SMTP id z8so24121766wge.0; Wed, 03 Jun 2015 21:37:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=pCr3PjEExDUQp/gYK8FdIGaMIEPl6ezwYocd9FTb8p4=; b=vHmG6egGm8o+QDwVzlnv8jP66dC6GnYTdb6mfbREnNYEthokozapffVjeV2D3wO9la HR822z4S7btQf3EUTWejQywmdPFSkK8fNymRAVyqlz8w8pa0TNR25H8AR/w2DpKGrB/m VXsLQYWe5bl1M3oRB8tsLpFTmedJFkvPZj/mwnKTcwZN+F1KYiVQe4+vqNml4Bmbqx/w 2b2CGG91COJnNrOTcld148Rb1MxQNG7QFYTIRLRuln6dhJ9dMErfNnBaa9jEZHBty1DE dYuTimHZVbwlQYsTG9vnKcNSoFqef100urNK39yGSayTkV3w75L+ODQWslLAT9D/gGy9 5Q3w== MIME-Version: 1.0 X-Received: by 10.180.98.103 with SMTP id eh7mr3636115wib.75.1433392624114; Wed, 03 Jun 2015 21:37:04 -0700 (PDT) Received: by 10.27.52.18 with HTTP; Wed, 3 Jun 2015 21:37:04 -0700 (PDT) In-Reply-To: <556ED071.5030009@FreeBSD.org> References: <556D9005.4020802@FreeBSD.org> <556DDDA9.6090005@FreeBSD.org> <556ED071.5030009@FreeBSD.org> Date: Wed, 3 Jun 2015 21:37:04 -0700 Message-ID: Subject: Re: bhyve: corrupting zfs pools? From: Neel Natu To: Andriy Gapon Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jun 2015 04:37:06 -0000 Hi Andriy, On Wed, Jun 3, 2015 at 3:01 AM, Andriy Gapon wrote: > On 03/06/2015 00:40, Neel Natu wrote: >> Perhaps, but I wasn't able to repro this. I tried your recipe to repro >> on two systems running -current at r283917. >> >> - Intel Sandybridge server: Xeon E52650 with 8 cores/16 threads and 64GB memory >> - hdd1/hdd2 on UFS >> - hdd1/hdd2 on ZFS >> >> - AMD Opteron server: Opteron 6230 with 8 cores and 16GB memory >> - hdd1/hdd2 on UFS >> >> Can you provide some more details about your setup? I can then try to >> repro on a system similar to your setup. >> >> - uname -a > > FreeBSD trant 11.0-CURRENT FreeBSD 11.0-CURRENT #134 r283188+ab4f83f(devel): Fri > May 22 15:55:27 EEST 2015 avg@trant:/usr/obj/usr/src/sys/TRANT amd64 > > This is not a pure FreeBSD, there are some local changes, but none to vmm or VM. > >> - sysctl hw.model > > hw.model: AMD Athlon(tm) II X2 250 Processor > >> - sysctl hw.ncpu > > hw.ncpu: 2 > >> - sysctl hw.physmem > > hw.physmem: 8029335552 > >> - sysctl hw.vmm > > hw.vmm.npt.pmap_flags: 507 > hw.vmm.svm.num_asids: 64 > hw.vmm.svm.disable_npf_assist: 0 > hw.vmm.svm.features: 15 > hw.vmm.svm.vmcb_clean: 959 > hw.vmm.vmx.vpid_alloc_failed: 0 > hw.vmm.vmx.posted_interrupt_vector: -1 > hw.vmm.vmx.cap.posted_interrupts: 0 > hw.vmm.vmx.cap.virtual_interrupt_delivery: 0 > hw.vmm.vmx.cap.invpcid: 0 > hw.vmm.vmx.cap.monitor_trap: 0 > hw.vmm.vmx.cap.unrestricted_guest: 0 > hw.vmm.vmx.cap.pause_exit: 0 > hw.vmm.vmx.cap.halt_exit: 0 > hw.vmm.vmx.initialized: 0 > hw.vmm.vmx.cr4_zeros_mask: 0 > hw.vmm.vmx.cr4_ones_mask: 0 > hw.vmm.vmx.cr0_zeros_mask: 0 > hw.vmm.vmx.cr0_ones_mask: 0 > hw.vmm.ept.pmap_flags: 0 > hw.vmm.vrtc.flag_broken_time: 1 > hw.vmm.ppt.devices: 0 > hw.vmm.iommu.initialized: 0 > hw.vmm.bhyve_xcpuids: 4 > hw.vmm.topology.cpuid_leaf_b: 1 > hw.vmm.topology.cores_per_package: 1 > hw.vmm.topology.threads_per_core: 1 > hw.vmm.create: beavis > hw.vmm.destroy: beavis > hw.vmm.force_iommu: 0 > hw.vmm.trace_guest_exceptions: 0 > hw.vmm.ipinum: 251 > hw.vmm.halt_detection: 1 > >> - host filesystem underlying hdd1 and hdd2 > > It's ZFS. > Ok, there are some differences in our systems. The interesting ones are number of ASIDs (64 versus 65536), flush-by-asid capability, vmcb-clean capability and the number of cores. I was able to mimic all of these on my Opteron but still wasn't able to reproduce the issue. I am going to get a Sempron tomorrow which belongs to the same processor family as the Athlon II so hoping that it is easier to repro. BTW does this happen consistently on your system? best Neel > Thank you! > > -- > Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Thu Jun 4 06:06:06 2015 Return-Path: Delivered-To: freebsd-virtualization@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2F765CB9 for ; Thu, 4 Jun 2015 06:06:06 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 73077179E for ; Thu, 4 Jun 2015 06:06:05 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id JAA25010; Thu, 04 Jun 2015 09:06:02 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Z0OHy-00045V-2c; Thu, 04 Jun 2015 09:06:02 +0300 Message-ID: <556FEA92.4050704@FreeBSD.org> Date: Thu, 04 Jun 2015 09:05:06 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: Neel Natu CC: "freebsd-virtualization@freebsd.org" Subject: Re: bhyve: corrupting zfs pools? References: <556D9005.4020802@FreeBSD.org> <556DDDA9.6090005@FreeBSD.org> <556ED071.5030009@FreeBSD.org> In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Jun 2015 06:06:06 -0000 On 04/06/2015 07:37, Neel Natu wrote: > Ok, there are some differences in our systems. The interesting ones > are number of ASIDs (64 versus 65536), flush-by-asid capability, > vmcb-clean capability and the number of cores. > > I was able to mimic all of these on my Opteron but still wasn't able > to reproduce the issue. I am going to get a Sempron tomorrow which > belongs to the same processor family as the Athlon II so hoping that > it is easier to repro. > > BTW does this happen consistently on your system? Yes, it happened every time I tried that scenario. Thank you very much! If there is anything else that I can do on my side to help the investigation just let me know. -- Andriy Gapon From owner-freebsd-virtualization@FreeBSD.ORG Fri Jun 5 17:23:41 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 7F7A795A for ; Fri, 5 Jun 2015 17:23:41 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-ob0-x229.google.com (mail-ob0-x229.google.com [IPv6:2607:f8b0:4003:c01::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 485591AA1 for ; Fri, 5 Jun 2015 17:23:41 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by obbgp2 with SMTP id gp2so36928561obb.2 for ; Fri, 05 Jun 2015 10:23:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=pixQyo/b5Y3Yp/hhACU2Gbe72zur/pIUkEX2uQ80/b0=; b=x4V5cID0EOYw21C677bN+2BeFNWZpuDkigEwYIi+w3dV+XbjHdbu3ki7HdWc5Jj3pu tePp4UA3qaiwWgfL1uhtzwZfiiln3eiwqqh8iKmrbgHztiIimGY/mupyVcpVgXuwAmCA 9suMSI4p3whhPMFkN4ZKWmIphG4952tA5rx37xakFMKxvnLG2Pcxmz76zbqiie78BiMq P99yihUu5JgICuqOhOwiifVsMt9jgWFy6o7gvcUiAt/hNnO0gQ6BBsnTn5cOMPFLuf9B mbfhgTi71CGxI/7oZRcYkG3ox0r54JtiEc5NhK6+S9YM4AEuv9XHB6ele/iq9wdGAasq OIWw== MIME-Version: 1.0 X-Received: by 10.202.50.198 with SMTP id y189mr3662410oiy.21.1433525020457; Fri, 05 Jun 2015 10:23:40 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Fri, 5 Jun 2015 10:23:40 -0700 (PDT) Date: Fri, 5 Jun 2015 20:23:40 +0300 Message-ID: Subject: Setting custom field in struct thread from bhyverun code From: Stefan Andritoiu To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jun 2015 17:23:41 -0000 I've added a new field to struct thread and want to set it's value when the VCPU thread is created. I need this new field in the ULE scheduler. My plan was this: in the fbsdrun_start_thread() function make a custom-ioctl call {ioctl(ctx->fd, VM_SET_CUSTOM, ..)} to set it. But I have no idea how to get the thread I need, to set it's value. How can I get the thread? Or is there a better way of setting, a VCPU-thread's field, at the moment the thread is created, to use it in the scheduler? From owner-freebsd-virtualization@FreeBSD.ORG Fri Jun 5 18:51:52 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F12AB919 for ; Fri, 5 Jun 2015 18:51:52 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: from mail-wi0-x231.google.com (mail-wi0-x231.google.com [IPv6:2a00:1450:400c:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8914510A7 for ; Fri, 5 Jun 2015 18:51:52 +0000 (UTC) (envelope-from neelnatu@gmail.com) Received: by wibdq8 with SMTP id dq8so27955081wib.1 for ; Fri, 05 Jun 2015 11:51:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=B9Ryr/R765fpaOwQjBvTV9Cq/N0UYJDOSuD4snPaZqc=; b=uMJlbvndSc2lsHY5obY0/pwqhukYiR5FxShcZPpyyeea1WTYoRFIYfKwgIXxjfEe2K tMyYkfVpLYpeeemTo6vrMeXw9wDrjoRurLffQnh7fR5IdQ43HT6/8iXVv1y3+0FCG9Nq 7SudSBbLVHMpljGIEl1nIjG3CEXsikwo57EImQsjC/CclPLwJipOnMjj3+5zGmLc9Zf0 t0xnkR6rQMZD02Kx2oLbT0vqETFOy4uM0SIt8zs96+FuUt/TKVlXrqYH4kIkNezFoye0 a7EL9M5GU0yIiW9R9TPnRCEuEh4fSIxKwJUvnC8noP/o7+kWedP5abiZuo4vioxfGpoN pflQ== MIME-Version: 1.0 X-Received: by 10.194.62.201 with SMTP id a9mr8723224wjs.63.1433530311009; Fri, 05 Jun 2015 11:51:51 -0700 (PDT) Received: by 10.27.52.18 with HTTP; Fri, 5 Jun 2015 11:51:50 -0700 (PDT) In-Reply-To: References: Date: Fri, 5 Jun 2015 11:51:50 -0700 Message-ID: Subject: Re: Setting custom field in struct thread from bhyverun code From: Neel Natu To: Stefan Andritoiu Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jun 2015 18:51:53 -0000 Hi Stefan, On Fri, Jun 5, 2015 at 10:23 AM, Stefan Andritoiu wrote: > I've added a new field to struct thread and want to set it's value > when the VCPU thread is created. I need this new field in the ULE > scheduler. > My plan was this: in the fbsdrun_start_thread() function make a > custom-ioctl call {ioctl(ctx->fd, VM_SET_CUSTOM, ..)} to set it. But I > have no idea how to get the thread I need, to set it's value. > How can I get the thread? Or is there a better way of setting, a > VCPU-thread's field, at the moment the thread is created, to use it in > the scheduler? The thread context in which the ioctl is executed is passed as the last argument to the ioctl handler. For example: vmmdev_ioctl(struct cdev *cdev, u_long cmd, caddr_t data, int fflag, struct thread *td) So, if you issue the ioctl() in vcpu context then 'td' points to the vcpu thread. best Neel > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" From owner-freebsd-virtualization@FreeBSD.ORG Fri Jun 5 22:59:30 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 788439CA for ; Fri, 5 Jun 2015 22:59:30 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-oi0-x235.google.com (mail-oi0-x235.google.com [IPv6:2607:f8b0:4003:c06::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 41EF71A6D for ; Fri, 5 Jun 2015 22:59:30 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by oihb142 with SMTP id b142so63252444oih.3 for ; Fri, 05 Jun 2015 15:59:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=/g7Dznp3aCHJn0dPTYqfZg2RhV5DtT+7t0mmN9nZuZc=; b=C7V/fVb5/c4dPL7Zw63YNq2bSboijuK1tLgWib/16Fo9OO3tutUDpC8y/YtDRNDa5d v/EcKgHk1e+5muytIeA5HTxL6KruEWC97wCAhQXpz0e98fL1AnaAxhPcvEF5//oiUkkl 3aVIdOahGi4xGSCzIwZgEeI0MxQ/QnI3q6stWXtK68tt5MrkLYvqXArjtERyBouvIknl i2EwBut8jJGBqv6BZRNAvjq91aCQmLklnx4PK1pOLb7KR9csQDRWeyw8BXOw9MaXrc+l a2qI6moXPoC5Q2D0w9ocNLXRdvrieCTXn6RhLx9grTtzsSuULnNUCkZphz1rDk3OcQXJ M2mQ== MIME-Version: 1.0 X-Received: by 10.202.129.70 with SMTP id c67mr4712945oid.42.1433545169470; Fri, 05 Jun 2015 15:59:29 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Fri, 5 Jun 2015 15:59:29 -0700 (PDT) Date: Sat, 6 Jun 2015 01:59:29 +0300 Message-ID: Subject: Lock Holder Preemption on bhyve From: Stefan Andritoiu To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jun 2015 22:59:30 -0000 Hello everyone, My name is Stefan Andritoiu and I'm currently studying Computer Science in my 4th year at the University POLITEHNICA of Bucharest. I'm fairly new to the FreeBSD operating system, having only a background in Linux. For the past few months I've been investigating the problem of Lock Holder Preemption on bhyve, how other hypervisors deal with this problem and a possible implementation of these solutions on bhyve. I am currently working on implementing Gang Scheduling on FreeBSD to test if it is a viable solution. I also plan to continue my work, by implementing and testing other techniques of avoiding overhead caused by lock holder preemption, and comparing their results. best, Stefan From owner-freebsd-virtualization@FreeBSD.ORG Fri Jun 5 23:03:32 2015 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F0B84A6E for ; Fri, 5 Jun 2015 23:03:31 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: from mail-oi0-x22b.google.com (unknown [IPv6:2607:f8b0:4003:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B0C211C98 for ; Fri, 5 Jun 2015 23:03:31 +0000 (UTC) (envelope-from stefan.andritoiu@gmail.com) Received: by oiww2 with SMTP id w2so63457457oiw.0 for ; Fri, 05 Jun 2015 16:03:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=c6TP8KRYcbNW++cdwIFS3SyKFUVIRgCFhnRzuYncA/Q=; b=wxowSLealvCyAIJ9gK6f7Jg8B+tvle2OwihxQlRw3ZcPRY/5hcs2WVM7v5TGcvh/km Zz4t19d2vUZ7vQP8zhiZkVxviB64oCjokw5euDJrNWyuHB+RFNYnDs6JcF9UkShcfTZ8 nuk+fRPLhVCkGGbkwPA3qvwdFBlfAyPZdNMtU8zu2oV1+5C1u8uBihA5Wja0MzZ11Uix NYbgECRojODgL2FucxAZzEEL9aSvoxxdTMtVCU2GV5Dw1XG9n2E0BwmR3hle4mS7uiCv y5lAsJWSkJ5CQDwcr1eF1MQlnkwafZxWnx4CpeTq5PcIlnTxD2Kixs5aE1vC/x4xPcSh DTgQ== MIME-Version: 1.0 X-Received: by 10.60.155.97 with SMTP id vv1mr5028114oeb.15.1433545410656; Fri, 05 Jun 2015 16:03:30 -0700 (PDT) Received: by 10.60.82.168 with HTTP; Fri, 5 Jun 2015 16:03:30 -0700 (PDT) Date: Sat, 6 Jun 2015 02:03:30 +0300 Message-ID: Subject: Implementation of gang-scheduling for bhyve VCPUs From: Stefan Andritoiu To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Jun 2015 23:03:32 -0000 Hello, I have broken the implementation of gang-scheduling in three steps: 1. Marking threads that should be gang scheduled - add a custom field "int gang" to the FreeBSD thread structure, defined in proc.h - create a custom ioctl request code VM_SET_GANG, that will set the 'gang' field of the VCPU thread, to the gang it is part of - in bhyverun.c, in the fbsdrun_start_thread() function that is invoked when a new pthread is created (a new VCPU is added), call the ioctl function with VM_SET_GANG request code 2.When a thread that must be gang scheduled is selected from a CPU's thread queue, inform other CPUs so they can schedule a thread belonging to that gang - add two new fields to struct tdq: bool gang_leader /* set if CPU is gang leader */ int scheduled_gang /* set if current CPU needs to search it's queue for a thread belonging to scheduled_gang */ - in sched_ule.c, in the sched_choose() function, after a thread is selected, check if it is part of a gang (has the gang field set) - if so, mark the CPU it is on as gang-leader (set gang_leader in tdq), and use function smp_rendezvous() to send IPI to all other CPUs to run schedule_gang() /* custom function defined in sched_ule.c*/ - in function schedule_gang(), set field scheduled_gang, and inform the processor it needs to reschedule (not yet sure how) 3. Scheduling rest of threads - in function tdq_choose(), check first if the tdq's scheduled_gang field is set. If so, reset scheduled_gang field to 0 and return the first thread in the tdq belonging to the gang. If sched_gang==0 or no thread belonging to the gang can be found continue with normal thread selection. This is the implementation of gang-scheduling on FreeBSD the way I have sketched it. If anyone has any thoughts/suggestions, what will work, what will not work, what could have been done better/simpler/easier, anything at all, I would really appreciate and welcome it. Thank you, Stefan