From owner-freebsd-virtualization@freebsd.org Sun Oct 11 17:44:04 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B5437A112D6 for ; Sun, 11 Oct 2015 17:44:04 +0000 (UTC) (envelope-from alfred@freebsd.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 9EF2912E5 for ; Sun, 11 Oct 2015 17:44:04 +0000 (UTC) (envelope-from alfred@freebsd.org) Received: from Alfreds-MacBook-Pro-2.local (unknown [IPv6:2601:645:8004:7515:4d2f:952f:e666:2ea2]) by elvis.mu.org (Postfix) with ESMTPSA id 5196B345A93F for ; Sun, 11 Oct 2015 10:43:57 -0700 (PDT) To: freebsd-virtualization@freebsd.org From: Alfred Perlstein Subject: Program for dynamically making taps/bridge topologies. Organization: FreeBSD Message-ID: <561A9FDC.40808@freebsd.org> Date: Sun, 11 Oct 2015 10:43:56 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2015 17:44:04 -0000 Hello, Here at Norse we are using bhyve to run our appliance in a test environment. We have a need for a test network to be made on demand, so this weekend I took some time and wrote a tool that allows you to specify a topology of bridges and taps that you can then use to create your virtual network. In our case our test servers have a bridge0 that is static and bridges to our private test network. However we also need a bridge to be dynamically that we will generate traffic over in isolation. We then need taps created and "assigned" or labeled properly for our QA suite to attach to our VMs properly. We then also need the ability to query the topology and emit the correct taps to assign to each vm. Finally we need the ability to tear down the virtual network once it's no longer needed. This tool provides all three functions. The tool is available here: https://github.com/splbio/netmanager Comments, pull requests and questions are welcome. -Alfred From owner-freebsd-virtualization@freebsd.org Sun Oct 11 21:00:44 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1921B9B118C for ; Sun, 11 Oct 2015 21:00:44 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E7B031CD0 for ; Sun, 11 Oct 2015 21:00:43 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id t9BL0hAm075723 for ; Sun, 11 Oct 2015 21:00:43 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201510112100.t9BL0hAm075723@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-virtualization@FreeBSD.org Subject: Problem reports for freebsd-virtualization@FreeBSD.org that need special attention X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 Date: Sun, 11 Oct 2015 21:00:43 +0000 Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2015 21:00:44 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- In Progress | 202321 | [bhyve,patch] More verbose error reporting in bhy New | 202322 | [bhyve,patch] add option to have bhyve write its 2 problems total for which you should take action. From owner-freebsd-virtualization@freebsd.org Mon Oct 12 05:59:59 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D0351A0E508 for ; Mon, 12 Oct 2015 05:59:59 +0000 (UTC) (envelope-from nicolascpa88@outlook.com) Received: from SNT004-OMC4S22.hotmail.com (snt004-omc4s22.hotmail.com [65.55.90.225]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "*.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A4CA617FD for ; Mon, 12 Oct 2015 05:59:59 +0000 (UTC) (envelope-from nicolascpa88@outlook.com) Received: from SNT150-W48 ([65.55.90.199]) by SNT004-OMC4S22.hotmail.com over TLS secured channel with Microsoft SMTPSVC(7.5.7601.23008); Sun, 11 Oct 2015 22:58:53 -0700 X-TMN: [IKPPOhkZl306RkSsaleijUxC+Klhx+p3] X-Originating-Email: [nicolascpa88@outlook.com] Message-ID: From: =?iso-8859-1?B?Tmljb2zhcyBD6XNhciBQcmVzdGEgQXVicull?= To: "freebsd-virtualization@freebsd.org" Subject: PCI Passthrough on FreeBSD host AMD64 IOMMU AMD-Vi Date: Mon, 12 Oct 2015 05:58:53 +0000 Importance: Normal MIME-Version: 1.0 X-OriginalArrivalTime: 12 Oct 2015 05:58:53.0616 (UTC) FILETIME=[13138B00:01D104B3] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2015 05:59:59 -0000 Hello=2C want to use FreeBSD as a VM Host with PCI Passthrough=20 is bhyve able to do PCI Passthrough on AMD64 host?VirtualBox's PCI Passthro= ugh works on FreeBSD? i've used PCI Passthrough on GNU/Linux with QEMU+KVM+VFIO=2C KVM exist on F= reeBSD? = From owner-freebsd-virtualization@freebsd.org Mon Oct 12 15:51:12 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EDC5DA113CD for ; Mon, 12 Oct 2015 15:51:12 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from smtp.digiware.nl (smtp.digiware.nl [31.223.170.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 498D519FF; Mon, 12 Oct 2015 15:51:12 +0000 (UTC) (envelope-from wjw@digiware.nl) Received: from rack1.digiware.nl (unknown [127.0.0.1]) by smtp.digiware.nl (Postfix) with ESMTP id 01411153431; Mon, 12 Oct 2015 17:51:03 +0200 (CEST) X-Virus-Scanned: amavisd-new at digiware.nl Received: from smtp.digiware.nl ([127.0.0.1]) by rack1.digiware.nl (rack1.digiware.nl [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TBT2yH1X0CN2; Mon, 12 Oct 2015 17:50:53 +0200 (CEST) Received: from [192.168.101.176] (vpn.ecoracks.nl [31.223.170.173]) by smtp.digiware.nl (Postfix) with ESMTPA id 44C4115340D; Mon, 12 Oct 2015 17:50:53 +0200 (CEST) Subject: Re: Program for dynamically making taps/bridge topologies. To: Alfred Perlstein , freebsd-virtualization@freebsd.org References: <561A9FDC.40808@freebsd.org> From: Willem Jan Withagen Message-ID: <561BD6DF.5080404@digiware.nl> Date: Mon, 12 Oct 2015 17:50:55 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <561A9FDC.40808@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2015 15:51:13 -0000 On 11-10-2015 19:43, Alfred Perlstein wrote: > Hello, > > Here at Norse we are using bhyve to run our appliance in a test > environment. > > We have a need for a test network to be made on demand, so this weekend > I took some time and wrote a tool that allows you to specify a topology > of bridges and taps that you can then use to create your virtual network. > > In our case our test servers have a bridge0 that is static and bridges > to our private test network. However we also need a bridge to be > dynamically that we will generate traffic over in isolation. We then > need taps created and "assigned" or labeled properly for our QA suite to > attach to our VMs properly. > > We then also need the ability to query the topology and emit the correct > taps to assign to each vm. > > Finally we need the ability to tear down the virtual network once it's > no longer needed. > > This tool provides all three functions. > > The tool is available here: > https://github.com/splbio/netmanager > > Comments, pull requests and questions are welcome. Sounds like cool stuff.... This should fit nicely into what is now fashionably called SDN... Now I got to figure out a way to glue this to my home-grown vmrun.sh stuff. And the ultimate step could be to see if it matches the things that openstack trieds to do for networking. Looking at it right away. But seems that it is getting harder and harder not to want to understand/write python. --WjW From owner-freebsd-virtualization@freebsd.org Mon Oct 12 16:40:03 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 68A46A11E89 for ; Mon, 12 Oct 2015 16:40:03 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 2870C1024 for ; Mon, 12 Oct 2015 16:40:02 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id 033C5115CB for ; Tue, 13 Oct 2015 02:39:56 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id E9AAE2811A5 for ; Tue, 13 Oct 2015 02:39:55 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KC1dYefFmEPg for ; Tue, 13 Oct 2015 02:39:55 +1000 (AEST) Received: from Peters-MacBook-Pro.local (c-50-184-135-69.hsd1.ca.comcast.net [50.184.135.69]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 008F4280F84; Tue, 13 Oct 2015 02:39:52 +1000 (AEST) Subject: Re: PCI Passthrough on FreeBSD host AMD64 IOMMU AMD-Vi To: =?UTF-8?B?Tmljb2zDoXMgQ8Opc2FyIFByZXN0YSBBdWJyw6ll?= , "freebsd-virtualization@freebsd.org" References: From: Peter Grehan Message-ID: <561BE254.8010908@freebsd.org> Date: Mon, 12 Oct 2015 09:39:48 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2015 16:40:03 -0000 > Hello, want to use FreeBSD as a VM Host with PCI Passthrough > is bhyve able to do PCI Passthrough on AMD64 host? bhyve is able to do passthrough on Intel systems that have VT-d hardware. https://wiki.freebsd.org/bhyve/pci_passthru later, Peter. From owner-freebsd-virtualization@freebsd.org Mon Oct 12 16:43:24 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0AA48A11016 for ; Mon, 12 Oct 2015 16:43:24 +0000 (UTC) (envelope-from nathan.bosley@gmail.com) Received: from mail-yk0-x236.google.com (mail-yk0-x236.google.com [IPv6:2607:f8b0:4002:c07::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BDE8112C8 for ; Mon, 12 Oct 2015 16:43:23 +0000 (UTC) (envelope-from nathan.bosley@gmail.com) Received: by ykey125 with SMTP id y125so17394547yke.3 for ; Mon, 12 Oct 2015 09:43:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=K/3jjUflXDZvXUDuxufI9N7UzPUy29D6nRQGQt3pllI=; b=vi6+mmdtws1cHx5N8mWhIArGnGnnlqZYGwgczvDy+vYlrXC+mmx7NoBdIi9iRIbwlr KmaMNy8jQpsbwixr5iSAY7UWM32SlkHdHD98JubZauyIpXlY+Wff+sxEkMTR6ghrzd6g dtZZvW9XVs39/8CLOI2vaWueeNpAFtloLP42EbcpZ+6wIbLVHNkLE5OfupHKWKx/2oBU w3wv5F52TrNAR3anvh+9SbCGsOLhIIgmGHVV3qXvqDInzAkgFjcyJiG9hf2bsJMUKye8 i2Ca9c95QseQiU9T0m5+9O5hbSOJlMz1bHgtfLqDK7Yuzq7ZOpfa2IVRrLxrHoOrP7VF 3dZg== MIME-Version: 1.0 X-Received: by 10.13.212.196 with SMTP id w187mr19752378ywd.137.1444668202963; Mon, 12 Oct 2015 09:43:22 -0700 (PDT) Received: by 10.37.71.84 with HTTP; Mon, 12 Oct 2015 09:43:22 -0700 (PDT) Received: by 10.37.71.84 with HTTP; Mon, 12 Oct 2015 09:43:22 -0700 (PDT) In-Reply-To: References: Date: Mon, 12 Oct 2015 12:43:22 -0400 Message-ID: Subject: Re: PCI Passthrough on FreeBSD host AMD64 IOMMU AMD-Vi From: Nathan Bosley To: =?UTF-8?B?Tmljb2zDoXMgQ8Opc2FyIFByZXN0YSBBdWJyw6ll?= Cc: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2015 16:43:24 -0000 I was looking to try this as well, based on an example found on Google: http://0xfeedface.org/2014/12/11/FreeBSD-Intel-wifi-via-bhyve.html In that example, a wifi nic is passed through to the host. I have not had a chance to try it yet personally. Should do sometime this week. Cheers. On Oct 12, 2015 2:00 AM, "Nicol=C3=A1s C=C3=A9sar Presta Aubr=C3=A9e" < nicolascpa88@outlook.com> wrote: > Hello, want to use FreeBSD as a VM Host with PCI Passthrough > is bhyve able to do PCI Passthrough on AMD64 host?VirtualBox's PCI > Passthrough works on FreeBSD? > i've used PCI Passthrough on GNU/Linux with QEMU+KVM+VFIO, KVM exist on > FreeBSD? > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > From owner-freebsd-virtualization@freebsd.org Tue Oct 13 11:43:31 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F8D3A11837 for ; Tue, 13 Oct 2015 11:43:31 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-outbound.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.userve.net", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 10BA0C97 for ; Tue, 13 Oct 2015 11:43:30 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) by smtp-outbound.userve.net (8.15.1/8.15.1) with ESMTPS id t9DBHlIv001769 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Tue, 13 Oct 2015 12:17:49 +0100 (BST) (envelope-from matt.churchyard@userve.net) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=201508; t=1444735069; bh=B4qcQHvh5Qy0jAZ2UXDo+aM3Uw+FDlpYjsr39pD/Ogw=; h=From:To:Subject:Date; b=ARk++UUdsWCmA3xkMc3E83rgWaufL64DjhUbql/09HZnmK5PC1FzJuWhticSqS9al E7OWDwgqLGGwnuO4swbR4whiOfWsALStf4bn9RddyaLN2TpR5N84qmEpDgCn3m9qz3 bYdlznHcVnTTLNbMSdDVe1K/OKUWdAjkBvEUCTiU= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Tue, 13 Oct 2015 12:17:41 +0100 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Tue, 13 Oct 2015 12:17:41 +0100 From: Matt Churchyard To: "freebsd-virtualization@freebsd.org" Subject: Illumos boot Thread-Topic: Illumos boot Thread-Index: AdEFpuLuEiXyF4rFT0iOg0wo9KoWVQ== Date: Tue, 13 Oct 2015 11:17:40 +0000 Message-ID: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 11:43:31 -0000 Hello In my quest to continue expanding guest support in my vm-bhyve utility (See= https://github.com/churchers/vm-bhyve :) ), I've found the Windows support= pretty solid once I got clear on the slot requirements. I'm now trying an = OS that requires CSM (Illumos) but unfortunately I'm currently struggling t= o get it to boot up correctly. Here's an example of the command I'm generating at the moment (This is runn= ing on an Intel Core-i3): bhyve -c 2 -m 2G -s 0,hostbridge -s 31,lpc \ -s 3,ahci-cd,/data/vm/.iso/smartos-latest.iso \ -s 4:0,ahci-hd,/data/vm/smartos/disk0.img \ -s 5:0,virtio-net,tap0 \ -l com1,stdio -l com2,/dev/nmdm2A \ -H -l bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd \ smartos I have com1 set to stdio so I can easily watch the output as it runs. It tends to get as far as "Legacy INT19 Boot...", then fall over. Depending on whether I put the network interface directly in the slot after= the HDD, I seem to get different errors - slot 3 - cd slot 4 - hdd slot 5 - virtio-net panic[cpu0]/thread=3Dffffff01457cdb40: BAD TRAP: type=3De (#pf Page fault) = rp=3Dffffff0004a69a60 addr=3D40 occurred in module "genunix" due to a NULL = pointer dereference slot 3 - cd slot 4 - hdd slot 7 - virtio-net panic[cpu1]/thread=3Dffffff0004002c40: BAD TRAP: type=3Dd (#gp General prot= ection) rp=3Dffffff0004002740 addr=3D0 On com2 I see the boot menu, then one and a half lines of dots. The second = line of dots stops about 2/3 of the way across. Interestingly, my code normally puts the CD after the HDD, which Windows se= ems happy with as long as the slots are consecutive. In SmartOS this gives me a different error: slot 3 - hdd slot 4 - cd slot 5 - virtio-net PlatformBdsBootFail Boot Failed. Harddisk 1 !!!! Find PE image /home/grehan/proj/stock_edk2/Build/BhyveX64/DEBUG_GCC48/= X64/UefiCpuPkg/CpuDxe/CpuDxe/DEBUG/CpuDxe.dll (ImageBase=3D000000007F8DC000= , EntryPoint=3D000000007F8DC2AF) !!!! Regards, Matt Churchyard From owner-freebsd-virtualization@freebsd.org Tue Oct 13 12:42:11 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1DE97A12D52 for ; Tue, 13 Oct 2015 12:42:11 +0000 (UTC) (envelope-from tycho.nightingale@pluribusnetworks.com) Received: from mail-qg0-x22d.google.com (mail-qg0-x22d.google.com [IPv6:2607:f8b0:400d:c04::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CF340D92 for ; Tue, 13 Oct 2015 12:42:10 +0000 (UTC) (envelope-from tycho.nightingale@pluribusnetworks.com) Received: by qgt47 with SMTP id 47so12482284qgt.2 for ; Tue, 13 Oct 2015 05:42:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pluribusnetworks.com; s=google; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=6kyRTe8yqXvrkEevYf51V3kypvweyvypXti6Rf0DtSw=; b=WQcPtqDS4ai2mS/Z2kj+jWNwITOCmXtHlgNK2lKXhs1HHiMvt5IycqblvDZJYDX6yI MgnNzT4AW6yH3qXi91UPTaGNTXLwRNguRFJsxUyxdb0gppgcjlCrjDBcw3e0TKNeFeuf uQUXE9PJlYB3cRRtkgMeW/6N4V842ts/keCr0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-type:mime-version:subject:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=6kyRTe8yqXvrkEevYf51V3kypvweyvypXti6Rf0DtSw=; b=avN4+Zzd7kE3yL5ai45b15R69eDO2P7o1v1LEkhyBu5o5euGOhsekIARbP8SvjtcqX jSzqVcH1V9iPbLwd/q5aqPn/LI+9C3M/6NUJI2oyPqkf/tgk9jT4lB7TPeDKdrTg5mvw 6+YnLc2FBhbRIF8D83H+KyjzK4OCcD48wVAj71E9/R9yNtdEDVzZtG35wi+aMplT5t4u hxUZkSwqE2ALXhXeCQ4++Kh15XoiKvQJ2tmNRyv6pgt+M4wt0HY0HXKgAuOIuH/tkMr5 V5DmiDYxu3jF+AeHZnaw1SVncqcdNOoAfNFyQkNpz/2MaOMiqqC3aRjvK3GpWQMs0FsS T/6g== X-Gm-Message-State: ALoCoQnnUB1jFK7SPJtBM1e3Lfap5SVVKqSNP/J4dcMQHZlCrTCUgdDfRtMvcGbkiQgzoHvI6oGX X-Received: by 10.140.109.74 with SMTP id k68mr40358256qgf.77.1444740129653; Tue, 13 Oct 2015 05:42:09 -0700 (PDT) Received: from [10.0.1.15] (209-6-121-211.c3-0.arl-ubr1.sbo-arl.ma.cable.rcn.com. [209.6.121.211]) by smtp.gmail.com with ESMTPSA id o39sm1072574qgd.48.2015.10.13.05.42.09 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 13 Oct 2015 05:42:09 -0700 (PDT) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Illumos boot From: Tycho Nightingale In-Reply-To: Date: Tue, 13 Oct 2015 08:42:07 -0400 Cc: "freebsd-virtualization@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <17936712-FB88-4017-8C0D-BEC4926D698E@pluribusnetworks.com> References: To: Matt Churchyard X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 12:42:11 -0000 Hi, Please see inline. On Oct 13, 2015, at 7:17 AM, Matt Churchyard via freebsd-virtualization = wrote: > In my quest to continue expanding guest support in my vm-bhyve utility = (See https://github.com/churchers/vm-bhyve :) ), I've found the Windows = support pretty solid once I got clear on the slot requirements. I'm now = trying an OS that requires CSM (Illumos) but unfortunately I'm currently = struggling to get it to boot up correctly. >=20 > Here's an example of the command I'm generating at the moment (This is = running on an Intel Core-i3): >=20 > bhyve -c 2 -m 2G -s 0,hostbridge -s 31,lpc \ > -s 3,ahci-cd,/data/vm/.iso/smartos-latest.iso \ > -s 4:0,ahci-hd,/data/vm/smartos/disk0.img \ > -s 5:0,virtio-net,tap0 \ > -l com1,stdio -l com2,/dev/nmdm2A \ > -H -l bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd \ > smartos >=20 > I have com1 set to stdio so I can easily watch the output as it runs. > It tends to get as far as "Legacy INT19 Boot...", then fall over. > Depending on whether I put the network interface directly in the slot = after the HDD, I seem to get different errors - >=20 > slot 3 - cd > slot 4 - hdd > slot 5 - virtio-net >=20 > panic[cpu0]/thread=3Dffffff01457cdb40: BAD TRAP: type=3De (#pf Page = fault) rp=3Dffffff0004a69a60 addr=3D40 occurred in module "genunix" due = to a NULL pointer dereference >=20 > slot 3 - cd > slot 4 - hdd > slot 7 - virtio-net >=20 > panic[cpu1]/thread=3Dffffff0004002c40: BAD TRAP: type=3Dd (#gp General = protection) rp=3Dffffff0004002740 addr=3D0 >=20 > On com2 I see the boot menu, then one and a half lines of dots. The = second line of dots stops about 2/3 of the way across. Have you tried booting illumos in verbose mode =97 edit the grub command = line and provide =91-v=92. That may give you a better backtrace than a = program counter. > Interestingly, my code normally puts the CD after the HDD, which = Windows seems happy with as long as the slots are consecutive. > In SmartOS this gives me a different error: >=20 > slot 3 - hdd > slot 4 - cd > slot 5 - virtio-net >=20 > PlatformBdsBootFail > Boot Failed. Harddisk 1 > !!!! Find PE image = /home/grehan/proj/stock_edk2/Build/BhyveX64/DEBUG_GCC48/X64/UefiCpuPkg/Cpu= Dxe/CpuDxe/DEBUG/CpuDxe.dll (ImageBase=3D000000007F8DC000, = EntryPoint=3D000000007F8DC2AF) !!!! This error is from the UEFI code. It implies that the CSM boot failed = or was never invoked. If the HDD isn=92t bootable, yet the CD is, that = is the most likely source as the CSM assumes the first block device it = encounters is the desired boot source. Tycho= From owner-freebsd-virtualization@freebsd.org Tue Oct 13 13:36:11 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ADB74A126A5 for ; Tue, 13 Oct 2015 13:36:11 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-outbound.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.userve.net", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 459FB10CD for ; Tue, 13 Oct 2015 13:36:10 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) by smtp-outbound.userve.net (8.15.1/8.15.1) with ESMTPS id t9DDa1NZ014677 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=FAIL); Tue, 13 Oct 2015 14:36:01 +0100 (BST) (envelope-from matt.churchyard@userve.net) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=201508; t=1444743365; bh=/eIiyZnRgLxk0tbF8rfcbWjShG1KZaerySdbK84zaQM=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=bTY3lD2vJT3KBbKJaVR4O2D+y62aEabAgI0iwRm6nEZrP2k6Z3UjB6X40CwIYbfNQ PYKqWJ3r8nVD0Wri4jThpQuizCzfJrLGVWFuo12cAF0B7FvpmgeRkx2fJJt8WGRONK YamQs4LWdz4O3u5Ay9kUNcJ/j2CeARbeETwgULiI= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Tue, 13 Oct 2015 14:35:55 +0100 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Tue, 13 Oct 2015 14:35:55 +0100 From: Matt Churchyard To: Tycho Nightingale CC: "freebsd-virtualization@freebsd.org" Subject: RE: Illumos boot Thread-Topic: Illumos boot Thread-Index: AdEFpuLuEiXyF4rFT0iOg0wo9KoWVQABU0uAAAObRxA= Date: Tue, 13 Oct 2015 13:35:55 +0000 Message-ID: <568ad0b2decf4537bff8f310a703c40b@SERVER.ad.usd-group.com> References: <17936712-FB88-4017-8C0D-BEC4926D698E@pluribusnetworks.com> In-Reply-To: <17936712-FB88-4017-8C0D-BEC4926D698E@pluribusnetworks.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 13:36:11 -0000 >Hi, >Please see inline. >On Oct 13, 2015, at 7:17 AM, Matt Churchyard via freebsd-virtualization wrote: > In my quest to continue expanding guest support in my vm-bhyve utility (S= ee https://github.com/churchers/vm-bhyve :) ), I've found the Windows suppo= rt pretty solid once I got clear on the slot requirements. I'm now trying a= n OS that requires CSM (Illumos) but unfortunately I'm currently struggling= to get it to boot up correctly. >=20 > Here's an example of the command I'm generating at the moment (This is ru= nning on an Intel Core-i3): >=20 > bhyve -c 2 -m 2G -s 0,hostbridge -s 31,lpc \ > -s 3,ahci-cd,/data/vm/.iso/smartos-latest.iso \ > -s 4:0,ahci-hd,/data/vm/smartos/disk0.img \ > -s 5:0,virtio-net,tap0 \ > -l com1,stdio -l com2,/dev/nmdm2A \ > -H -l bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd \ > smartos >=20 > I have com1 set to stdio so I can easily watch the output as it runs. > It tends to get as far as "Legacy INT19 Boot...", then fall over. > Depending on whether I put the network interface directly in the slot aft= er the HDD, I seem to get different errors - >=20 > slot 3 - cd > slot 4 - hdd > slot 5 - virtio-net >=20 > panic[cpu0]/thread=3Dffffff01457cdb40: BAD TRAP: type=3De (#pf Page fault= ) rp=3Dffffff0004a69a60 addr=3D40 occurred in module "genunix" due to a NUL= L pointer dereference >=20 > slot 3 - cd > slot 4 - hdd > slot 7 - virtio-net >=20 > panic[cpu1]/thread=3Dffffff0004002c40: BAD TRAP: type=3Dd (#gp General pr= otection) rp=3Dffffff0004002740 addr=3D0 >=20 > On com2 I see the boot menu, then one and a half lines of dots. The secon= d line of dots stops about 2/3 of the way across. >Have you tried booting illumos in verbose mode - edit the grub command lin= e and provide '-v'. That may give you a better backtrace than a >program c= ounter. This is what I get from a verbose boot: Bhyve-HandleProtocol: Copying DevPath: PciRoot(0x0)/Pci(0x3,0x0)/Sata(0x0,0= x0,0x0) [32] Legacy INT19 Boot... cpu0: x86 (chipid 0x0 GenuineIntel 206A7 family 6 model 42 step 7 clock 310= 9 MHz) cpu0: Intel(r) Core(tm) i3-2100 CPU @ 3.10GHz pseudo-device: stmf_sbd0 stmf_sbd0 is /pseudo/stmf_sbd@0 pseudo-device: lofi0 lofi0 is /pseudo/lofi@0 pseudo-device: devinfo0 devinfo0 is /pseudo/devinfo@0 acpinex0 at root acpinex0 is /fw iscsi0 at root iscsi0 is /iscsi xsvc0 at root: space 0 offset 0 xsvc0 is /xsvc@0,0 acpinex: sb@0, acpinex1 acpinex1 is /fw/sb@0 pseudo-device: pseudo1 pseudo1 is /pseudo/zconsnex@1 pseudo-device: pseudo2 pseudo2 is /pseudo/zfdnex@2 /pci@0,0/pci8086,2821@3 : SATA CD/DVD (ATAPI) device at port 0 model BHYVE SATA DVD ROM firmware 001 serial number BHYVE-EA14-A68A-54FA supported features: DMA SATA Gen3 signaling speed (6.0Gbps) pseudo-device: llc10 llc10 is /pseudo/llc1@0 pseudo-device: power0 power0 is /pseudo/power@0 pseudo-device: ramdisk1024 ramdisk1024 is /pseudo/ramdisk@1024 pseudo-device: ucode0 ucode0 is /pseudo/ucode@0 pseudo-device: zfs0 zfs0 is /pseudo/zfs@0 pseudo-device: srn0 srn0 is /pseudo/srn@0 pseudo-device: dtrace0 dtrace0 is /pseudo/dtrace@0 pseudo-device: dcpc0 dcpc0 is /pseudo/dcpc@0 pseudo-device: fasttrap0 fasttrap0 is /pseudo/fasttrap@0 pseudo-device: fbt0 fbt0 is /pseudo/fbt@0 pseudo-device: profile0 profile0 is /pseudo/profile@0 pseudo-device: lockstat0 lockstat0 is /pseudo/lockstat@0 pseudo-device: sdt0 sdt0 is /pseudo/sdt@0 pseudo-device: systrace0 systrace0 is /pseudo/systrace@0 pseudo-device: ipd0 ipd0 is /pseudo/ipd@0 pseudo-device: stmf0 stmf0 is /pseudo/stmf@0 sd0 at ahci0: target 0 lun 0 sd0 is /pci@0,0/pci8086,2821@3/cdrom@0,0 pseudo-device: fssnap0 fssnap0 is /pseudo/fssnap@0 /pci@0,0/pci8086,2821@3/cdrom@0,0 (sd0) online /pci@0,0/pci8086,2821@4 : SATA disk device at port 0 model BHYVE SATA DISK firmware 001 serial number BHYVE-3083-1AF1-1754 supported features: 48-bit LBA, DMA, Native Command Queueing SATA Gen3 signaling speed (6.0Gbps) Supported queue depth 32 capacity =3D 62914560 sectors WARNING: kvm: no hardware support pseudo-device: pool0 pool0 is /pseudo/pool@0 pseudo-device: bpf0 bpf0 is /pseudo/bpf@0 sd1 at ahci1: target 0 lun 0 sd1 is /pci@0,0/pci8086,2821@4/disk@0,0 pseudo-device: pm0 pm0 is /pseudo/pm@0 pseudo-device: nsmb0 nsmb0 is /pseudo/nsmb@0 pseudo-device: tap0 tap0 is /pseudo/tap@0 /pci@0,0/pci8086,2821@4/disk@0,0 (sd1) online NOTICE: vioif0: Got MAC address from host: e4:94:1:0:ff:ff pseudo-device: tun0 tun0 is /pseudo/tun@0 pseudo-device: lx_systrace0 lx_systrace0 is /pseudo/lx_systrace@0 panic[cpu0]/thread=3Dffffff0002566c40: BAD TRAP: type=3Dd (#gp General prot= ection) rp=3Dffffff00025664c0 addr=3D20 sched: #gp General protection addr=3D0x20 pid=3D0, pc=3D0xfffffffff80d375a, sp=3D0xffffff00025665b0, eflags=3D0x10282 cr0: 8005003b cr4: 406b8 cr2: fed3b5accr3: 1dc00000cr8: c rdi: 7f1a90ffffff00c3 rsi: ffffff00c33321f8 rdx: ffffff00c37f5828 rcx: ffffff00c3868603 r8: ffffff00ca7aa600 r9: 2ba6 rax: 0 rbx: ffffff00c30d0ef0 rbp: ffffff00025665c0 r10: fffffffffb8554c4 r11: 1 r12: 1f r13: ffffff00c319f880 r14: 10 r15: 20 fsb: 0 gsb: fffffffffbc326a0 ds: 4b es: 4b fs: 0 gs: 1c3 trp: d err: 0 rip: fffffffff80d375a cs: 30 rfl: 10282 rsp: ffffff00025665b0 ss: 38 This is the log of the bhyve options used (apart from 1 cpu, 1G ram) Oct 13 14:22:43: [bhyve devices: -s 0,hostbridge -s 31,lpc -s 4:0,ahci-hd,= /data/vm/smartos/disk0.img -s 7:0,virtio-net,tap1] Oct 13 14:22:43: [bhyve console: -l com1,/dev/nmdm1A -l com2,/dev/nmdm2A] Oct 13 14:22:43: [bhyve options: -Hw -l bootrom,/data/vm/.config/BHYVE_UEF= I_CSM.fd] Oct 13 14:22:43: [bhyve iso device: -s 3:0,ahci-cd,/data/vm/.iso/smartos-l= atest.iso] > Interestingly, my code normally puts the CD after the HDD, which Windows = seems happy with as long as the slots are consecutive. > In SmartOS this gives me a different error: >=20 > slot 3 - hdd > slot 4 - cd > slot 5 - virtio-net >=20 > PlatformBdsBootFail > Boot Failed. Harddisk 1 > !!!! Find PE image /home/grehan/proj/stock_edk2/Build/BhyveX64/DEBUG_GCC4= 8/X64/UefiCpuPkg/CpuDxe/CpuDxe/DEBUG/CpuDxe.dll (ImageBase=3D000000007F8DC0= 00, EntryPoint=3D000000007F8DC2AF) !!!! >This error is from the UEFI code. It implies that the CSM boot failed or = was never invoked. If the HDD isn't bootable, yet the CD is, that is the >= most likely source as the CSM assumes the first block device it encounters = is the desired boot source. Ok, so the boot semantics are currently different between the CSM and non-C= SM firmware? CSM will try and boot the first device and fail if it's not bo= otable, whereas non-CSM will always boot CD if it's bootable, regardless of= order (from Windows instructions). Matt From owner-freebsd-virtualization@freebsd.org Tue Oct 13 21:15:45 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 628FAA124D8 for ; Tue, 13 Oct 2015 21:15:45 +0000 (UTC) (envelope-from tycho.nightingale@pluribusnetworks.com) Received: from mail-pa0-x236.google.com (mail-pa0-x236.google.com [IPv6:2607:f8b0:400e:c03::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 345501A1B for ; Tue, 13 Oct 2015 21:15:45 +0000 (UTC) (envelope-from tycho.nightingale@pluribusnetworks.com) Received: by padcn9 with SMTP id cn9so916870pad.2 for ; Tue, 13 Oct 2015 14:15:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pluribusnetworks.com; s=google; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=QteF8JTn5xwYp7vqJ7FOtZUD0LnGSf9kGzS04fJ1y70=; b=JqPYlskiggYhQjuE71Bi7VfCrDHE+R+FdqcM1JSaox8E7wgNi82efbDoiqB9jdsW8g eyOu29l4fqZ/rGoBdPagK5U6IU9rU7XaiHUaAPGe5ALV3fA14Q3aOSEWsctdOENKRlmE dnCMs2LBd/oQuZgp9/IH5DecxC0b4pRbr6AnA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to; bh=QteF8JTn5xwYp7vqJ7FOtZUD0LnGSf9kGzS04fJ1y70=; b=MohxMmAdtloKWdWavmG5awTC7VTjaOMaeMN8JR7AC9uHvyYs4YufNkqYYLEuxH10TN yqM9FTP9Kp12VP3VFR3aXeOUDrywvIWhOnoTWwk9UvLG8VAVHmZGk06njPZxxvGtVUB6 GGt6DSms+KwPDIcGXUkg9yj6nltu3H6lwWs2sMDvmcp5AMdERbdjInh3YvOOzX7/quXX B+7VhEEP6hKJn/H9NXgar6OYbg9jLd+Pq4vuum3qPzaU9/utW54+SBheBPGHXEDFft8F Aqz9CfjOFYehDn/AfhHUOkytivZTyuPem/d3IoU6nyY3icK4QcXh3fbPRR9TEAbNPS4V 7M4g== X-Gm-Message-State: ALoCoQl3UdKDtN/OJOBwdkcmLbezY8l7PYNXAmC7HmpHK1zpGKmXCg5LHsNy/cAoue5gT2Bomgbe X-Received: by 10.68.242.2 with SMTP id wm2mr43340551pbc.31.1444770944357; Tue, 13 Oct 2015 14:15:44 -0700 (PDT) Received: from [10.0.1.16] (209-6-121-211.c3-0.arl-ubr1.sbo-arl.ma.cable.rcn.com. [209.6.121.211]) by smtp.gmail.com with ESMTPSA id fb1sm5613020pab.9.2015.10.13.14.15.43 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 13 Oct 2015 14:15:43 -0700 (PDT) Subject: Re: Illumos boot Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=us-ascii From: Tycho Nightingale In-Reply-To: <568ad0b2decf4537bff8f310a703c40b@SERVER.ad.usd-group.com> Date: Tue, 13 Oct 2015 17:15:43 -0400 Cc: "freebsd-virtualization@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <83C11159-D5DE-4E02-972C-A66A80155EB8@pluribusnetworks.com> References: <17936712-FB88-4017-8C0D-BEC4926D698E@pluribusnetworks.com> <568ad0b2decf4537bff8f310a703c40b@SERVER.ad.usd-group.com> To: Matt Churchyard X-Mailer: Apple Mail (2.1283) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2015 21:15:45 -0000 Hi, On Oct 13, 2015, at 9:35 AM, Matt Churchyard wrote: >=20 >> On Oct 13, 2015, at 7:17 AM, Matt Churchyard via = freebsd-virtualization wrote: >=20 >> In my quest to continue expanding guest support in my vm-bhyve = utility (See https://github.com/churchers/vm-bhyve :) ), I've found the = Windows support pretty solid once I got clear on the slot requirements. = I'm now trying an OS that requires CSM (Illumos) but unfortunately I'm = currently struggling to get it to boot up correctly. >>=20 >> Here's an example of the command I'm generating at the moment (This = is running on an Intel Core-i3): >>=20 >> bhyve -c 2 -m 2G -s 0,hostbridge -s 31,lpc \ >> -s 3,ahci-cd,/data/vm/.iso/smartos-latest.iso \ >> -s 4:0,ahci-hd,/data/vm/smartos/disk0.img \ >> -s 5:0,virtio-net,tap0 \ >> -l com1,stdio -l com2,/dev/nmdm2A \ >> -H -l bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd \ >> smartos >>=20 >> I have com1 set to stdio so I can easily watch the output as it runs. >> It tends to get as far as "Legacy INT19 Boot...", then fall over. >> Depending on whether I put the network interface directly in the slot = after the HDD, I seem to get different errors - >>=20 >> slot 3 - cd >> slot 4 - hdd >> slot 5 - virtio-net >>=20 >> panic[cpu0]/thread=3Dffffff01457cdb40: BAD TRAP: type=3De (#pf Page = fault) rp=3Dffffff0004a69a60 addr=3D40 occurred in module "genunix" due = to a NULL pointer dereference >>=20 >> slot 3 - cd >> slot 4 - hdd >> slot 7 - virtio-net >>=20 >> panic[cpu1]/thread=3Dffffff0004002c40: BAD TRAP: type=3Dd (#gp = General protection) rp=3Dffffff0004002740 addr=3D0 >>=20 >> On com2 I see the boot menu, then one and a half lines of dots. The = second line of dots stops about 2/3 of the way across. >=20 >> Have you tried booting illumos in verbose mode - edit the grub = command line and provide '-v'. That may give you a better backtrace = than a >program counter. >=20 > This is what I get from a verbose boot: >=20 > Bhyve-HandleProtocol: Copying DevPath: = PciRoot(0x0)/Pci(0x3,0x0)/Sata(0x0,0x0,0x0) [32] > Legacy INT19 Boot... > cpu0: x86 (chipid 0x0 GenuineIntel 206A7 family 6 model 42 step 7 = clock 3109 MHz) > cpu0: Intel(r) Core(tm) i3-2100 CPU @ 3.10GHz > pseudo-device: stmf_sbd0 > stmf_sbd0 is /pseudo/stmf_sbd@0 > pseudo-device: lofi0 > lofi0 is /pseudo/lofi@0 > pseudo-device: devinfo0 > devinfo0 is /pseudo/devinfo@0 > acpinex0 at root > acpinex0 is /fw > iscsi0 at root > iscsi0 is /iscsi > xsvc0 at root: space 0 offset 0 > xsvc0 is /xsvc@0,0 > acpinex: sb@0, acpinex1 > acpinex1 is /fw/sb@0 > pseudo-device: pseudo1 > pseudo1 is /pseudo/zconsnex@1 > pseudo-device: pseudo2 > pseudo2 is /pseudo/zfdnex@2 > /pci@0,0/pci8086,2821@3 : > SATA CD/DVD (ATAPI) device at port 0 > model BHYVE SATA DVD ROM > firmware 001 > serial number BHYVE-EA14-A68A-54FA > supported features: > DMA > SATA Gen3 signaling speed (6.0Gbps) > pseudo-device: llc10 > llc10 is /pseudo/llc1@0 > pseudo-device: power0 > power0 is /pseudo/power@0 > pseudo-device: ramdisk1024 > ramdisk1024 is /pseudo/ramdisk@1024 > pseudo-device: ucode0 > ucode0 is /pseudo/ucode@0 > pseudo-device: zfs0 > zfs0 is /pseudo/zfs@0 > pseudo-device: srn0 > srn0 is /pseudo/srn@0 > pseudo-device: dtrace0 > dtrace0 is /pseudo/dtrace@0 > pseudo-device: dcpc0 > dcpc0 is /pseudo/dcpc@0 > pseudo-device: fasttrap0 > fasttrap0 is /pseudo/fasttrap@0 > pseudo-device: fbt0 > fbt0 is /pseudo/fbt@0 > pseudo-device: profile0 > profile0 is /pseudo/profile@0 > pseudo-device: lockstat0 > lockstat0 is /pseudo/lockstat@0 > pseudo-device: sdt0 > sdt0 is /pseudo/sdt@0 > pseudo-device: systrace0 > systrace0 is /pseudo/systrace@0 > pseudo-device: ipd0 > ipd0 is /pseudo/ipd@0 > pseudo-device: stmf0 > stmf0 is /pseudo/stmf@0 > sd0 at ahci0: target 0 lun 0 > sd0 is /pci@0,0/pci8086,2821@3/cdrom@0,0 > pseudo-device: fssnap0 > fssnap0 is /pseudo/fssnap@0 > /pci@0,0/pci8086,2821@3/cdrom@0,0 (sd0) online > /pci@0,0/pci8086,2821@4 : > SATA disk device at port 0 > model BHYVE SATA DISK > firmware 001 > serial number BHYVE-3083-1AF1-1754 > supported features: > 48-bit LBA, DMA, Native Command Queueing > SATA Gen3 signaling speed (6.0Gbps) > Supported queue depth 32 > capacity =3D 62914560 sectors > WARNING: kvm: no hardware support >=20 > pseudo-device: pool0 > pool0 is /pseudo/pool@0 > pseudo-device: bpf0 > bpf0 is /pseudo/bpf@0 > sd1 at ahci1: target 0 lun 0 > sd1 is /pci@0,0/pci8086,2821@4/disk@0,0 > pseudo-device: pm0 > pm0 is /pseudo/pm@0 > pseudo-device: nsmb0 > nsmb0 is /pseudo/nsmb@0 > pseudo-device: tap0 > tap0 is /pseudo/tap@0 > /pci@0,0/pci8086,2821@4/disk@0,0 (sd1) online > NOTICE: vioif0: Got MAC address from host: e4:94:1:0:ff:ff > pseudo-device: tun0 > tun0 is /pseudo/tun@0 > pseudo-device: lx_systrace0 > lx_systrace0 is /pseudo/lx_systrace@0 >=20 > panic[cpu0]/thread=3Dffffff0002566c40: BAD TRAP: type=3Dd (#gp General = protection) rp=3Dffffff00025664c0 addr=3D20 >=20 > sched: #gp General protection > addr=3D0x20 > pid=3D0, pc=3D0xfffffffff80d375a, sp=3D0xffffff00025665b0, = eflags=3D0x10282 > cr0: 8005003b cr4: = 406b8 > cr2: fed3b5accr3: 1dc00000cr8: c >=20 > rdi: 7f1a90ffffff00c3 rsi: ffffff00c33321f8 rdx: = ffffff00c37f5828 > rcx: ffffff00c3868603 r8: ffffff00ca7aa600 r9: = 2ba6 > rax: 0 rbx: ffffff00c30d0ef0 rbp: = ffffff00025665c0 > r10: fffffffffb8554c4 r11: 1 r12: = 1f > r13: ffffff00c319f880 r14: 10 r15: = 20 > fsb: 0 gsb: fffffffffbc326a0 ds: = 4b > es: 4b fs: 0 gs: = 1c3 > trp: d err: 0 rip: = fffffffff80d375a > cs: 30 rfl: 10282 rsp: = ffffff00025665b0 > ss: 38 >=20 > This is the log of the bhyve options used (apart from 1 cpu, 1G ram) >=20 > Oct 13 14:22:43: [bhyve devices: -s 0,hostbridge -s 31,lpc -s = 4:0,ahci-hd,/data/vm/smartos/disk0.img -s 7:0,virtio-net,tap1] > Oct 13 14:22:43: [bhyve console: -l com1,/dev/nmdm1A -l = com2,/dev/nmdm2A] > Oct 13 14:22:43: [bhyve options: -Hw -l = bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd] > Oct 13 14:22:43: [bhyve iso device: -s = 3:0,ahci-cd,/data/vm/.iso/smartos-latest.iso] Ouch, even with the additional verbosity, the output isn't very = insightful. All I can glean is that you are reasonably far along in the = boot process. You could try to run with KMDB (-k boot option) and do = some disassembly around that program counter to see if any specific = module is implicated. Alternately, I'd omit the network device and see how far that gets you. >> Interestingly, my code normally puts the CD after the HDD, which = Windows seems happy with as long as the slots are consecutive. >> In SmartOS this gives me a different error: >>=20 >> slot 3 - hdd >> slot 4 - cd >> slot 5 - virtio-net >>=20 >> PlatformBdsBootFail >> Boot Failed. Harddisk 1 >> !!!! Find PE image = /home/grehan/proj/stock_edk2/Build/BhyveX64/DEBUG_GCC48/X64/UefiCpuPkg/Cpu= Dxe/CpuDxe/DEBUG/CpuDxe.dll (ImageBase=3D000000007F8DC000, = EntryPoint=3D000000007F8DC2AF) !!!! >=20 >> This error is from the UEFI code. It implies that the CSM boot = failed or was never invoked. If the HDD isn't bootable, yet the CD is, = that is the >most likely source as the CSM assumes the first block = device it encounters is the desired boot source. >=20 > Ok, so the boot semantics are currently different between the CSM and = non-CSM firmware? CSM will try and boot the first device and fail if = it's not bootable, whereas non-CSM will always boot CD if it's bootable, = regardless of order (from Windows instructions). Yes, the boot semantics are different. The UEFI (non-CSM) path is = somewhat more tolerant of trying the next device in the boot-order if = the current one is deemed "unbootable". The CSM path is cruder in that = it just searches for the first block device and goes for it. If that = device is unbootable it will fall thorough to the UEFI path somewhat = ungracefully as UEFI isn't really expecting CSM to ever return. Tycho= From owner-freebsd-virtualization@freebsd.org Thu Oct 15 01:25:33 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B179A1290E for ; Thu, 15 Oct 2015 01:25:33 +0000 (UTC) (envelope-from hosting.windows@aruba.it) Received: from smartcmd0185.aruba.it (smartcmd0187.aruba.it [62.149.158.87]) by mx1.freebsd.org (Postfix) with ESMTP id B84151591 for ; Thu, 15 Oct 2015 01:25:31 +0000 (UTC) (envelope-from hosting.windows@aruba.it) Received: from webs1001.aruba.it ([62.149.131.11]) by smartcmd01.ad.aruba.it with bizsmtp id VDRP1r00A0EuK6e01DRPPC; Thu, 15 Oct 2015 03:25:23 +0200 Received: from webs1001 ([127.0.0.1]) by webs1001.aruba.it with Microsoft SMTPSVC(6.0.3790.4675); Thu, 15 Oct 2015 03:25:23 +0200 Subject: We could not deliver your parcel, #0000833841 To: freebsd-virtualization@freebsd.org Date: Thu, 15 Oct 2015 03:25:23 +0200 From: "FedEx International Economy" Reply-To: "FedEx International Economy" Message-ID: <10365641a87acaf85ada358770981ecf@professionalequipment.it> X-Priority: 3 MIME-Version: 1.0 X-OriginalArrivalTime: 15 Oct 2015 01:25:23.0230 (UTC) FILETIME=[5CF69BE0:01D106E8] Content-Type: text/plain; charset=us-ascii X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2015 01:25:33 -0000 Dear Customer, We could not deliver your item. You can review complete details of your order in the find attached. Sincerely, Bob Hudson, Sr. Delivery Manager. From owner-freebsd-virtualization@freebsd.org Thu Oct 15 08:03:24 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BB0DFA12832 for ; Thu, 15 Oct 2015 08:03:24 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-outbound.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.userve.net", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6D1B31E55 for ; Thu, 15 Oct 2015 08:03:23 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) by smtp-outbound.userve.net (8.15.1/8.15.1) with ESMTPS id t9F838iq099694 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=FAIL); Thu, 15 Oct 2015 09:03:08 +0100 (BST) (envelope-from matt.churchyard@userve.net) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=201508; t=1444896192; bh=YATtGnt4A/REt7Ar6QmjykdhFk/MnJ5uo3oGi5froPg=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=Na3e3EqEkHMgTe/032zTDC0o0fYHom34SOkmC3TQvOc9V8O/JmT3ckKGnuBijh4U/ d9X7/9i869btlv4epx+GFfrjP5+pw124qtJ43husBSdlUoQqlQzyVYFFR89bPH4PiS FGJcSQgcHKwBNS83HTqNtcPec+5TR7f3Ki1DFkCU= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Thu, 15 Oct 2015 09:03:02 +0100 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Thu, 15 Oct 2015 09:03:02 +0100 From: Matt Churchyard To: Tycho Nightingale CC: "freebsd-virtualization@freebsd.org" Subject: RE: Illumos boot Thread-Topic: Illumos boot Thread-Index: AdEFpuLuEiXyF4rFT0iOg0wo9KoWVQABU0uAAAObRxAADlSqgABK4nfw Date: Thu, 15 Oct 2015 08:03:01 +0000 Message-ID: <051593da46cc417fa3e3b8fef3018746@SERVER.ad.usd-group.com> References: <17936712-FB88-4017-8C0D-BEC4926D698E@pluribusnetworks.com> <568ad0b2decf4537bff8f310a703c40b@SERVER.ad.usd-group.com> <83C11159-D5DE-4E02-972C-A66A80155EB8@pluribusnetworks.com> In-Reply-To: <83C11159-D5DE-4E02-972C-A66A80155EB8@pluribusnetworks.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.0.10] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2015 08:03:24 -0000 > Hi, >> .............. > /pci@0,0/pci8086,2821@4/disk@0,0 (sd1) online > NOTICE: vioif0: Got MAC address from host: e4:94:1:0:ff:ff > pseudo-device: tun0 > tun0 is /pseudo/tun@0 > pseudo-device: lx_systrace0 > lx_systrace0 is /pseudo/lx_systrace@0 >=20 > panic[cpu0]/thread=3Dffffff0002566c40: BAD TRAP: type=3Dd (#gp General=20 > protection) rp=3Dffffff00025664c0 addr=3D20 >=20 > sched: #gp General protection > addr=3D0x20 > pid=3D0, pc=3D0xfffffffff80d375a, sp=3D0xffffff00025665b0, eflags=3D0x102= 82 > cr0: 8005003b cr4:=20 > 406b8 > cr2: fed3b5accr3: 1dc00000cr8: c >=20 > rdi: 7f1a90ffffff00c3 rsi: ffffff00c33321f8 rdx: ffffff00c37f5828 > rcx: ffffff00c3868603 r8: ffffff00ca7aa600 r9: 2ba6 > rax: 0 rbx: ffffff00c30d0ef0 rbp: ffffff00025665c0 > r10: fffffffffb8554c4 r11: 1 r12: 1f > r13: ffffff00c319f880 r14: 10 r15: 20 > fsb: 0 gsb: fffffffffbc326a0 ds: 4b > es: 4b fs: 0 gs: 1c3 > trp: d err: 0 rip: fffffffff80d375a > cs: 30 rfl: 10282 rsp: ffffff00025665b0 > ss: 38 >=20 > This is the log of the bhyve options used (apart from 1 cpu, 1G ram) >=20 > Oct 13 14:22:43: [bhyve devices: -s 0,hostbridge -s 31,lpc -s=20 > 4:0,ahci-hd,/data/vm/smartos/disk0.img -s 7:0,virtio-net,tap1] Oct 13=20 > 14:22:43: [bhyve console: -l com1,/dev/nmdm1A -l com2,/dev/nmdm2A]=20 > Oct 13 14:22:43: [bhyve options: -Hw -l=20 > bootrom,/data/vm/.config/BHYVE_UEFI_CSM.fd] > Oct 13 14:22:43: [bhyve iso device: -s=20 > 3:0,ahci-cd,/data/vm/.iso/smartos-latest.iso] > Ouch, even with the additional verbosity, the output isn't very insightfu= l. All I can glean is that you are reasonably far along in the boot > proc= ess. You could try to run with KMDB (-k boot option) and do some disassemb= ly around that program counter to see if any specific module is > implicate= d. > Alternately, I'd omit the network device and see how far that gets you. It does appear that I can get as far as a login console without the virtio-= net device Matt From owner-freebsd-virtualization@freebsd.org Thu Oct 15 16:57:08 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A2F6EA157B3 for ; Thu, 15 Oct 2015 16:57:08 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 6181417CA for ; Thu, 15 Oct 2015 16:57:08 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTP id 726E6114F8 for ; Fri, 16 Oct 2015 02:57:00 +1000 (EST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 0BAE2280F54 for ; Fri, 16 Oct 2015 02:57:00 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Z8j8aUodhMBO for ; Fri, 16 Oct 2015 02:56:59 +1000 (AEST) Received: from Peters-MacBook-Pro.local (c-50-184-135-69.hsd1.ca.comcast.net [50.184.135.69]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 38BB12804DB; Fri, 16 Oct 2015 02:56:55 +1000 (AEST) Subject: Re: Illumos boot To: Matt Churchyard , Tycho Nightingale References: <17936712-FB88-4017-8C0D-BEC4926D698E@pluribusnetworks.com> <568ad0b2decf4537bff8f310a703c40b@SERVER.ad.usd-group.com> <83C11159-D5DE-4E02-972C-A66A80155EB8@pluribusnetworks.com> <051593da46cc417fa3e3b8fef3018746@SERVER.ad.usd-group.com> Cc: "freebsd-virtualization@freebsd.org" , Andriy Gapon From: Peter Grehan Message-ID: <561FDAD6.4030006@freebsd.org> Date: Thu, 15 Oct 2015 09:56:54 -0700 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0) Gecko/20100101 Thunderbird/38.3.0 MIME-Version: 1.0 In-Reply-To: <051593da46cc417fa3e3b8fef3018746@SERVER.ad.usd-group.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2015 16:57:08 -0000 Hi Matt, >> -s 0,hostbridge Things should work if you leave out the hostbridge. The PCIe capability that is tacked on to this will make Illumos use MSI/MSIx for the virtio adapter which apparently hits a bug in the driver. Without it, the virtio driver will fall back to legacy interrupts. This also means that the virtio adapter will be confined to slots 3/4/5/6. [root@smartos ~]# echo ::interrupts | mdb -k IRQ Vect IPL Bus Trg Type CPU Share APIC/INT# ISR(s) 1 0x40 5 ISA Edg Fixed 0 1 0x0/0x1 i8042_intr 3 0xb1 12 ISA Edg Fixed 0 1 0x0/0x3 asyintr 4 0xb0 12 ISA Edg Fixed 0 1 0x0/0x4 asyintr 9 0x81 9 PCI Lvl Fixed 1 1 0x0/0x9 acpi_wrapper_isr 12 0x41 5 ISA Edg Fixed 1 1 0x0/0xc i8042_intr 16 0x42 5 PCI Lvl Fixed 1 1 0x0/0x10 ahci_intr 17 0x43 5 PCI Lvl Fixed 0 1 0x0/0x11 ahci_intr 18 0x60 6 PCI Lvl Fixed 1 1 0x0/0x12 virtio_intx_dispatch 160 0xa0 0 Edg IPI all 0 - poke_cpu 208 0xd0 14 Edg IPI all 1 - kcpc_hw_overflow_intr 209 0xd1 14 Edg IPI all 1 - cbe_fire 210 0xd3 14 Edg IPI all 1 - cbe_fire 240 0xe0 15 Edg IPI all 1 - xc_serv 241 0xe1 15 Edg IPI all 1 - apic_error_intr (I believe Andriy Gapon (cc'd) has a fix for this in Illumos) later, Peter. From owner-freebsd-virtualization@freebsd.org Thu Oct 15 17:25:48 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C2567A15DB4 for ; Thu, 15 Oct 2015 17:25:48 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from smtp-outbound.userve.net (smtp-outbound.userve.net [217.196.1.22]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.userve.net", Issuer "Go Daddy Secure Certificate Authority - G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6CA67FA5; Thu, 15 Oct 2015 17:25:47 +0000 (UTC) (envelope-from matt.churchyard@userve.net) Received: from owa.usd-group.com (owa.usd-group.com [217.196.1.2]) by smtp-outbound.userve.net (8.15.1/8.15.1) with ESMTPS id t9FHPbI5027888 (version=TLSv1 cipher=ECDHE-RSA-AES256-SHA bits=256 verify=FAIL); Thu, 15 Oct 2015 18:25:40 +0100 (BST) (envelope-from matt.churchyard@userve.net) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=userve.net; s=201508; t=1444929940; bh=GWFJontVuno0htYgnxUDqzJPmw4ieq+F6tzXITUUl3Y=; h=From:To:CC:Subject:Date:References:In-Reply-To; b=g5xXIQFWfRmQL4z8/AzqttGRfJohaVmFxnSViohOQm9KYd5fnYfHI4qmpSGuNPTLn IlaW6PBa2ARoqRf8y6nYSgfGJ8EnUQ125gU6TQr0JuffWPaz4aCvaWmXqd8NJV6rSl uBPOaIhaUvPWhVBGc30z0XTYFVwRsvTgbgJawsqY= Received: from SERVER.ad.usd-group.com (192.168.0.1) by SERVER.ad.usd-group.com (192.168.0.1) with Microsoft SMTP Server (TLS) id 15.0.847.32; Thu, 15 Oct 2015 18:25:31 +0100 Received: from SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9]) by SERVER.ad.usd-group.com ([fe80::b19d:892a:6fc7:1c9%12]) with mapi id 15.00.0847.030; Thu, 15 Oct 2015 18:25:31 +0100 From: Matt Churchyard To: Peter Grehan CC: "freebsd-virtualization@freebsd.org" Subject: RE: Illumos boot Thread-Topic: Illumos boot Thread-Index: AdEFpuLuEiXyF4rFT0iOg0wo9KoWVQABU0uAAAObRxAADlSqgABK4nfwABCovQAAAue4ig== Date: Thu, 15 Oct 2015 17:25:30 +0000 Message-ID: References: <17936712-FB88-4017-8C0D-BEC4926D698E@pluribusnetworks.com> <568ad0b2decf4537bff8f310a703c40b@SERVER.ad.usd-group.com> <83C11159-D5DE-4E02-972C-A66A80155EB8@pluribusnetworks.com> <051593da46cc417fa3e3b8fef3018746@SERVER.ad.usd-group.com>, <561FDAD6.4030006@freebsd.org> In-Reply-To: <561FDAD6.4030006@freebsd.org> Accept-Language: en-GB, en-US Content-Language: en-GB X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [89.145.244.131] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2015 17:25:49 -0000 =0A= =0A= >> -s 0,hostbridge=0A= =0A= > Things should work if you leave out the hostbridge. The PCIe=0A= > capability that is tacked on to this will make Illumos use MSI/MSIx for= =0A= > the virtio adapter which apparently hits a bug in the driver. Without=0A= > it, the virtio driver will fall back to legacy interrupts. This also=0A= > means that the virtio adapter will be confined to slots 3/4/5/6.=0A= =0A= > [root@smartos ~]# echo ::interrupts | mdb -k=0A= > IRQ Vect IPL Bus Trg Type CPU Share APIC/INT# ISR(s)=0A= > 1 0x40 5 ISA Edg Fixed 0 1 0x0/0x1 i8042_intr=0A= > 3 0xb1 12 ISA Edg Fixed 0 1 0x0/0x3 asyintr=0A= > 4 0xb0 12 ISA Edg Fixed 0 1 0x0/0x4 asyintr=0A= > 9 0x81 9 PCI Lvl Fixed 1 1 0x0/0x9 acpi_wrapper_isr=0A= > 12 0x41 5 ISA Edg Fixed 1 1 0x0/0xc i8042_intr=0A= > 16 0x42 5 PCI Lvl Fixed 1 1 0x0/0x10 ahci_intr=0A= > 17 0x43 5 PCI Lvl Fixed 0 1 0x0/0x11 ahci_intr=0A= > 18 0x60 6 PCI Lvl Fixed 1 1 0x0/0x12 virtio_intx_dispatch= =0A= > 160 0xa0 0 Edg IPI all 0 - poke_cpu=0A= > 208 0xd0 14 Edg IPI all 1 - kcpc_hw_overflow_intr= =0A= > 209 0xd1 14 Edg IPI all 1 - cbe_fire=0A= > 210 0xd3 14 Edg IPI all 1 - cbe_fire=0A= > 240 0xe0 15 Edg IPI all 1 - xc_serv=0A= > 241 0xe1 15 Edg IPI all 1 - apic_error_intr=0A= =0A= =0A= > (I believe Andriy Gapon (cc'd) has a fix for this in Illumos)=0A= =0A= > later,=0A= =0A= > Peter.=0A= =0A= Thanks Peter. I can't believe how many times I checked you Illumos instruct= ions making sure there were no obvious differences, and didn't notice the h= ostbridge was missing. I think I just had it in my mind that the hostbridge= was a basic required device.=0A= =0A= Regards=0A= Matt= From owner-freebsd-virtualization@freebsd.org Fri Oct 16 16:39:56 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D884DA16F72 for ; Fri, 16 Oct 2015 16:39:56 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id E53EB131E; Fri, 16 Oct 2015 16:39:55 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA25892; Fri, 16 Oct 2015 19:39:53 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1Zn82r-000O8x-E1; Fri, 16 Oct 2015 19:39:53 +0300 Subject: Re: Illumos boot To: Peter Grehan , Matt Churchyard , Tycho Nightingale References: <17936712-FB88-4017-8C0D-BEC4926D698E@pluribusnetworks.com> <568ad0b2decf4537bff8f310a703c40b@SERVER.ad.usd-group.com> <83C11159-D5DE-4E02-972C-A66A80155EB8@pluribusnetworks.com> <051593da46cc417fa3e3b8fef3018746@SERVER.ad.usd-group.com> <561FDAD6.4030006@freebsd.org> Cc: "freebsd-virtualization@freebsd.org" From: Andriy Gapon Message-ID: <56212820.1010008@FreeBSD.org> Date: Fri, 16 Oct 2015 18:38:56 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <561FDAD6.4030006@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2015 16:39:56 -0000 On 15/10/2015 18:56, Peter Grehan wrote: > Hi Matt, > >>> -s 0,hostbridge > > Things should work if you leave out the hostbridge. The PCIe capability > that is tacked on to this will make Illumos use MSI/MSIx for the virtio > adapter which apparently hits a bug in the driver. Without it, the > virtio driver will fall back to legacy interrupts. This also means that > the virtio adapter will be confined to slots 3/4/5/6. > > [root@smartos ~]# echo ::interrupts | mdb -k > IRQ Vect IPL Bus Trg Type CPU Share APIC/INT# ISR(s) > 1 0x40 5 ISA Edg Fixed 0 1 0x0/0x1 i8042_intr > 3 0xb1 12 ISA Edg Fixed 0 1 0x0/0x3 asyintr > 4 0xb0 12 ISA Edg Fixed 0 1 0x0/0x4 asyintr > 9 0x81 9 PCI Lvl Fixed 1 1 0x0/0x9 acpi_wrapper_isr > 12 0x41 5 ISA Edg Fixed 1 1 0x0/0xc i8042_intr > 16 0x42 5 PCI Lvl Fixed 1 1 0x0/0x10 ahci_intr > 17 0x43 5 PCI Lvl Fixed 0 1 0x0/0x11 ahci_intr > 18 0x60 6 PCI Lvl Fixed 1 1 0x0/0x12 virtio_intx_dispatch > 160 0xa0 0 Edg IPI all 0 - poke_cpu > 208 0xd0 14 Edg IPI all 1 - kcpc_hw_overflow_intr > 209 0xd1 14 Edg IPI all 1 - cbe_fire > 210 0xd3 14 Edg IPI all 1 - cbe_fire > 240 0xe0 15 Edg IPI all 1 - xc_serv > 241 0xe1 15 Edg IPI all 1 - apic_error_intr > > > (I believe Andriy Gapon (cc'd) has a fix for this in Illumos) Yes, it's here https://www.illumos.org/rb/r/86/ I plan to RTI that change soon-ish. -- Andriy Gapon From owner-freebsd-virtualization@freebsd.org Fri Oct 16 23:05:37 2015 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4D8CA16B87 for ; Fri, 16 Oct 2015 23:05:37 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from forward13o.cmail.yandex.net (forward13o.cmail.yandex.net [IPv6:2a02:6b8:0:1a72::1e3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Certum Level IV CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5FA427E8 for ; Fri, 16 Oct 2015 23:05:36 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from smtp3o.mail.yandex.net (smtp3o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::27]) by forward13o.cmail.yandex.net (Yandex) with ESMTP id 84049218A9 for ; Sat, 17 Oct 2015 02:05:24 +0300 (MSK) Received: from smtp3o.mail.yandex.net (localhost [127.0.0.1]) by smtp3o.mail.yandex.net (Yandex) with ESMTP id 567281E0067 for ; Sat, 17 Oct 2015 02:05:24 +0300 (MSK) Received: by smtp3o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id IocDHNTb9m-5NLK8Y2Y; Sat, 17 Oct 2015 02:05:23 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) X-Yandex-ForeignMX: US To: freebsd-virtualization@freebsd.org From: Boris Samorodov Subject: libvirt + zvol + bhyve: no storage pool with matching name X-Enigmail-Draft-Status: N1110 Message-ID: <562182B3.9040601@passap.ru> Date: Sat, 17 Oct 2015 02:05:23 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2015 23:05:37 -0000 Hi All, I try to use libvirt to run a zvol-backed vm. The vm starts just fine if run from a command line. But libvirt does not create a domain (raw file images work though). Here is the output: ----- % virsh create test.xml error: Failed to create domain from test.xml error: Storage pool not found: no storage pool with matching name 'zroot' ----- The relevant part from the config: ----- ----- The host: ----- zfs get type,volmode zroot/test NAME PROPERTY VALUE SOURCE zroot/test type volume - zroot/test volmode dev local % uname -a FreeBSD bb055.bsnet 11.0-CURRENT FreeBSD 11.0-CURRENT #2 r289095: Sat Oct 10 05:10:17 MSK 2015 bsam@bb055.bsnet:/usr/obj/usr/src/sys/BB64X amd64 % pkg info -x virt libvirt-1.2.20 ----- Is it just me, a bug or something else? Thanks! -- WBR, bsam