From owner-freebsd-virtualization@freebsd.org Sun Jan 8 03:55:56 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0A362CA5683 for ; Sun, 8 Jan 2017 03:55:56 +0000 (UTC) (envelope-from g_amanakis@yahoo.com) Received: from nm21-vm3.bullet.mail.ne1.yahoo.com (nm21-vm3.bullet.mail.ne1.yahoo.com [98.138.91.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BEFE41504 for ; Sun, 8 Jan 2017 03:55:55 +0000 (UTC) (envelope-from g_amanakis@yahoo.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1483847591; bh=bKhAEvc9PY3X8utLQN3QzuTTluwMTOSFRK38DyIIhsg=; h=Subject:From:To:Date:In-Reply-To:References:From:Subject; b=i8dPwNr0G7br3x0Qy4863986es+78rhl685k2TlZ7A/MQdjCK3OdmFlcXD45BSCXkjtZz9B1j04GMiI+ClJ2pdkSpwOWpK5iwl3+2TG1xx5mxojkDacqCxWGWrUVRUqpL/0AvviPj7P9gzjYfJFbCn7sYT/XZ/JZbNFcMR4g3SN33Dq0I6AgI16heuAqcKpldgzOphjVRzAc1wNIyFwCv5xYZHMwRYJRIgGFpjooiY56d+Psu8ZzO05Zpoj6SCo514nq+4v8dmbACtwpm72ScVFu2ZXOiqHqXBCRh7YAoQCepzSFKtHDqnbXU+mv9p2rpH6mb0E3PDyQcPHb59UySg== Received: from [98.138.101.132] by nm21.bullet.mail.ne1.yahoo.com with NNFMP; 08 Jan 2017 03:53:11 -0000 Received: from [98.138.226.57] by tm20.bullet.mail.ne1.yahoo.com with NNFMP; 08 Jan 2017 03:53:11 -0000 Received: from [127.0.0.1] by smtp208.mail.ne1.yahoo.com with NNFMP; 08 Jan 2017 03:53:11 -0000 X-Yahoo-Newman-Id: 383549.2073.bm@smtp208.mail.ne1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: WE9bjQAVM1kkOAydTtym8YXLI2WOoKABvJpvQcPpqvV1D4e 3vOV8HtZwn8zvlhasPRroi2YryoKCojqpROzPIUwBza.Jwsv46Du7bpTyUcZ uFznDMUxhs1rsTYA2ZsupYbuGSa8NlOPgYyfhHIKwzpbytGFuc8HBmcHVWkC Uwcc37h7Zwaobub.4GPfu4lKwvk4WvcwbQCujaMAJZF6CAPDIZKJ_2UkimgM guTh.G0y9WpiX0tR4UAJ.dRVxRuTkKR4oODctG_K7.7wqU085vxVZ2MV.6mR c9veY7vOjmZ8GLox5Z6R9JrUcFbPSbawNF.bDg3mJ2wnuR9u_CUg.CALgzt6 XwWEmqktrX8WmxtnmtMlnZ5cUKWmz1yVm6V0RFnc1bRK4Mv7l_13jD1rJ9LW 2WxrRh0VqpLtenLuQ3.6GfYSvHqYLW0nkfGrcysVLDXGPXxjp8cflTADxhtL r6gL_MXS.d1ERts.Se2Ym.hQTvLP7pwz2PCQo8nyVafT03amGc3MAGx7jtL7 WUm.32.gRIyfh853G_XZnOEIVYcBw3FZ5BTgRz76NdF7l6RE4KTc- X-Yahoo-SMTP: 6sUo5IiswBDB2TZm6JKJ6DaI.Rsz4O0- Message-ID: <1483847589.2604.1.camel@yahoo.com> Subject: Re: Arch Linux UEFI Bhyve Setup From: Georgios Amanakis To: freebsd-virtualization@freebsd.org Date: Sat, 07 Jan 2017 22:53:09 -0500 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.22.3 Mime-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jan 2017 03:55:56 -0000 Dear Vincent, GRUB2 in Archlinux normally boots in "graphical" mode. You have toswitch it= to "text" mode. This is done by adding:=20 -------------8<-------------- GRUB_TERMINAL=3D"serial console" GRUB_SERIAL_COMMAND=3D"serial --unit 0 --speed=3D115200 --stop=3D1 -- parity=3Dno --word=3D8" -------------8<-------------- into /etc/default/grub, and then run=C2=A0 "grub-mkconfig -o /boot/grub/grub.cfg". This should let you boot it under UEFI on bhyve by running: "bhyve -u -A -H -P -s 0:0,hostbridge -s 31:0,lpc -s 14:0,virtio- net,tap1 -s 3:0,virtio-blk,./linux.img -l com1,stdio -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd -c 1 -m 1024M linuxguest". Please, let us know if this solved your problem. Best regards, George From owner-freebsd-virtualization@freebsd.org Sun Jan 8 15:18:57 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE2B3CA524D for ; Sun, 8 Jan 2017 15:18:57 +0000 (UTC) (envelope-from internationils@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5091C1626 for ; Sun, 8 Jan 2017 15:18:56 +0000 (UTC) (envelope-from internationils@gmx.net) Received: from [10.0.2.15] ([87.174.152.48]) by mail.gmx.com (mrgmx101 [212.227.17.168]) with ESMTPSA (Nemesis) id 0M1BMy-1ck6Vi22mm-00tEel for ; Sun, 08 Jan 2017 16:18:54 +0100 To: freebsd-virtualization@freebsd.org From: Nils Subject: Help getting vbox .vdi imares to run under Freenas Message-ID: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> Date: Sun, 8 Jan 2017 16:18:53 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.5.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:mDeFOdQeF3Pk9MEUtxTiuxPTXmh6bFfUVh0RDVabM1LyI4QFNcp QDxGhl603gNjKz/yaVgZUIhyUuj4Rq2nrza/HlqeT1qpn0TwIc8+uSJ9lqWJGk5x9vTo1Ix XC5vcQW0bUHNwKbj2iiYVGFbhXRoMp44vDlujxO8Y4iS6T/KwYt3Nn92zelnyj2CAprf0nu gG7j3e+7JZfQMj98rc1wg== X-UI-Out-Filterresults: notjunk:1;V01:K0:B93dQ8E+5Rg=:OFWNyrXWG+9UwDoxbUGclF Oa9dPuBE2bwmqOtWpWUMG/KFG+a8sH8N6V1Fh6nN3YQDTxxbfDhqbxOgWE8XUIQKkRgaSlk06 TjVrSVuOa70SECEju/9A11HSv/8Z9bf7gQFFKLTXLzFJ4tWoi3zgq0ol7p23HB8jhUFoN8407 cv95Cn7fjuq9jCzViCYf4nYIqjZK9/OAVYaLsY1zZ93dXEEUZ3wCH1h4dj8A6Wr2jGm9ibjVz m0gib+RFdfHdiOlE1TVt9y4H4iwpvdMoAj9P/wmf/0jTzNv1hnRzEpp1P8je0dV3FclZZvDDJ T2Cgd8nn/Y2a24V6EPgTsG7x2kqyJuBGHT8aleV8WoG11ZwkwH4kftl/yqO+k1MHUF2Ka3C8m td+PxZyq9V4A6NrrPV+I07n7hJ27aO5VU1f012A5zy5CZfThBjC9aTnsvxO1jT98uk2OR6+Kt drZGiI73vVbqO3caQlaE59finx5oEpvRytBk6n9DHicTwxM8NIdC3Lvfdbbr6auVjKSx5cKHm 8irS2swzDQmo4LE33cYyHjtEQLGG/kLk1kmgVm248ntIXhtUWBH4M1sRg/Cscx1yWp85vJFRN YLy//ubbZZg701W9EHhskOxU7hq8EplqaONAqDPJD9+7ceZH/T756aZP17YGj9YGMSmXDhKvo 25uTUnhckJKsmLW1nmLSI7NGWbBYP/I25oHrUqM1hshn86tld6yKrjshmdciXEITVGX4AWMWh zsif/g9TQvkpPQNtaPwuIICF2PlYK5vvvjW+XWkcqQQun5w2PRzeLUrWLyBcEJNX+bRgUrW5Z o9TNVCv X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jan 2017 15:18:58 -0000 Hello, I'm fighting to get vbox vdi images to run under FreeNAS and don't know what else to try. I've filed and commented on these two bugs: https://github.com/pr1ntf/iohyve/issues/227 https://github.com/pr1ntf/iohyve/issues/228 ...but I think the problem is not with bhyve itself or with iohyve, but either with grub-bhyve or ZFS. Running installation ISOs etc. works fine, but I need to get the VDIs going., Any pointers are welcome... Thanks Nils From owner-freebsd-virtualization@freebsd.org Sun Jan 8 17:01:52 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15973CA5638 for ; Sun, 8 Jan 2017 17:01:52 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EE4B519B2 for ; Sun, 8 Jan 2017 17:01:51 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 8D24E13394 for ; Sun, 8 Jan 2017 17:01:50 +0000 (UTC) Subject: Re: Help getting vbox .vdi imares to run under Freenas To: freebsd-virtualization@freebsd.org References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> From: Allan Jude Message-ID: <4b65f427-07dc-8b0c-1ac9-59c3c1533336@freebsd.org> Date: Sun, 8 Jan 2017 12:01:50 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 MIME-Version: 1.0 In-Reply-To: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jan 2017 17:01:52 -0000 On 2017-01-08 10:18, Nils wrote: > Hello, I'm fighting to get vbox vdi images to run under FreeNAS and > don't know what else to try. I've filed and commented on these two bugs: > https://github.com/pr1ntf/iohyve/issues/227 > https://github.com/pr1ntf/iohyve/issues/228 > > ...but I think the problem is not with bhyve itself or with iohyve, but > either with grub-bhyve or ZFS. > Running installation ISOs etc. works fine, but I need to get the VDIs > going., > > Any pointers are welcome... > Thanks > Nils > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > How are you converting the .VDI to a raw image? bhyve does not yet support the .VDI format, only raw. -- Allan Jude From owner-freebsd-virtualization@freebsd.org Sun Jan 8 17:07:16 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64EB6CA56EB for ; Sun, 8 Jan 2017 17:07:16 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3A8C91BF5 for ; Sun, 8 Jan 2017 17:07:15 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 2A248133AA for ; Sun, 8 Jan 2017 17:07:14 +0000 (UTC) Subject: Re: Help getting vbox .vdi imares to run under Freenas To: freebsd-virtualization@freebsd.org References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> From: Allan Jude Message-ID: Date: Sun, 8 Jan 2017 12:07:13 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 MIME-Version: 1.0 In-Reply-To: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="QJxB5cs6qX5hnn5459oRcp9hnpRcWHSrq" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jan 2017 17:07:16 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --QJxB5cs6qX5hnn5459oRcp9hnpRcWHSrq Content-Type: multipart/mixed; boundary="kHi0GRs7tatAUHX04nOH9itH1jTTrnF2f"; protected-headers="v1" From: Allan Jude To: freebsd-virtualization@freebsd.org Message-ID: Subject: Re: Help getting vbox .vdi imares to run under Freenas References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> In-Reply-To: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> --kHi0GRs7tatAUHX04nOH9itH1jTTrnF2f Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2017-01-08 10:18, Nils wrote: > Hello, I'm fighting to get vbox vdi images to run under FreeNAS and > don't know what else to try. I've filed and commented on these two bugs= : > https://github.com/pr1ntf/iohyve/issues/227 > https://github.com/pr1ntf/iohyve/issues/228 >=20 > ...but I think the problem is not with bhyve itself or with iohyve, but= > either with grub-bhyve or ZFS. > Running installation ISOs etc. works fine, but I need to get the VDIs > going., >=20 > Any pointers are welcome... > Thanks > Nils > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fr= eebsd.org" >=20 I wonder if you are having the same problem as this user: https://lists.freebsd.org/pipermail/freebsd-virtualization/2017-January/0= 05088.html That the .VDI you are booting is starting grub in graphical mode, hence your lack of feedback on the serial console. --=20 Allan Jude --kHi0GRs7tatAUHX04nOH9itH1jTTrnF2f-- --QJxB5cs6qX5hnn5459oRcp9hnpRcWHSrq Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJYcnHBAAoJEBmVNT4SmAt+WwMQANXNlbb3zFmfF9Y7RRc5e5tr 95yIC+9duLXzKuZRCk4Dg/pyGisz3KIFGP55lns05BlwiYnKJXD6QP4fLzTWgYo4 YYqAVEFnin/NypJSHow9SpNFaOd3H4Ncil7q7JCT0lL/QACEDucTTTP92iwyQPfY dtSzYmfedO+tOe66pffbeouMEnjHoLSHt9EDL4oeK/t3GejMl0RQzx7bLeJ2/4fH HwYxxlrixZcEDUHvLyCn+T5WJUhYgD4Dm+6yyQVkUAvPufxN9GDAk6xaKxpCeex1 jEfXWktEFFLvgYz6/RXgl7xkNjKwt5MyjE7N2q4xAzawEo1CneTUW1XWwJWUUILb sPx4DMSq+k5Iem3Vxh786L8GQeoRIZCWjh+D0JhDnPCAqE9WnZXccotjo4FLF+Jy X6K7Upgeo1wSC9LlsKcJkrAd75Z4iJ5EbCMVXV/c8iHXwLdL669lSp5qg7gDmdXG EXts52GCErQl7oUvXCKBrTLEJ0GX0E45LSeBbjwQSVaXdHR/JVAoCWD3UO1RmD3b 9hKJnxtzvDmkSdkYBPxtD4beWlg2xt2OmI3Si6VGLrL4xISWZ85f6/hK25Wzmaqf YSQokOBmTox/MxwIgtBUuVvUBcWSjIa1otYx/43OIPiUGY0CqpI0g22Mv4rI6vOD t8ICoNPTfCdoR8+xufTX =zuag -----END PGP SIGNATURE----- --QJxB5cs6qX5hnn5459oRcp9hnpRcWHSrq-- From owner-freebsd-virtualization@freebsd.org Sun Jan 8 17:22:12 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 20976CA5C9F for ; Sun, 8 Jan 2017 17:22:12 +0000 (UTC) (envelope-from internationils@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 75FAF12D9 for ; Sun, 8 Jan 2017 17:22:10 +0000 (UTC) (envelope-from internationils@gmx.net) Received: from [10.0.2.15] ([87.174.152.48]) by mail.gmx.com (mrgmx102 [212.227.17.168]) with ESMTPSA (Nemesis) id 0LxMgg-1cWM0r2yfi-016zFd for ; Sun, 08 Jan 2017 18:21:59 +0100 Subject: Re: Help getting vbox .vdi imares to run under Freenas To: freebsd-virtualization@freebsd.org References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> <4b65f427-07dc-8b0c-1ac9-59c3c1533336@freebsd.org> From: Nils Message-ID: <201dbbc3-53de-ba98-6a88-16dd06444bc5@gmx.net> Date: Sun, 8 Jan 2017 18:21:58 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.5.1 MIME-Version: 1.0 In-Reply-To: <4b65f427-07dc-8b0c-1ac9-59c3c1533336@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:06Qo9ua3KV68RFEV0gCYAV20A2OiKSbYWqEKKa1fvxzVxE2vOFp QJdzIj2E8uF3iuWrLuywSUQtXSzgOQhj3+eUoHBjMlx7XITtWqIQnxpzc2XoaNqIoUkdMEl IdNng/i7mrBJ96Wt2usJK9S+UR2Wa255lPfDemCctV7geW5Y4a01VJxXZpMVQnoNAiwfeQ6 EZznX1hdYjTt/xWIy0CSQ== X-UI-Out-Filterresults: notjunk:1;V01:K0:btVhgAP5Xcg=:RF1Xn2oQ5lQIZbBhCCv9SE 4xofApGjdzvqesLSwV0UksiofPvpSlZByv/qzaxZh7uToMDbfTG9EAO4ruf/nu/PkY/e8H8LJ mg5FcK2rMhcYsVcyceFSGshesuLyujb8TGfBxU0kDhQunHDiQxwowBuqfi2AGEOXlJ8J+FM16 Ck3/8w/tqSXs4LbmqJJGY1eTUy2QYlwHpVvePE996Uy/19VN0EfSYfGgpEfnWQyiazqRr2NoQ 9pxwBn2ENNx3XEbak6+xWZBjbIz9G9VeBgmKOF4VU7tfXdq+RPxZ+PTjpf6X0rBS2SFetZ/84 R1fFRqSImVbrzTfe6SAhQSmxgCGobxtEDZRVPN7h9We/3nYIiezVCSnZJjsnBwkp2ExYnHGY/ 1EyY7YygQzRd5afVu4ih/l3avk7tTdmCz6UMH5lWkj0BHQTMuohkBhb1UeYr9zWz8X9Z09pjE eZDXmhXrPn8BTnmJshIEymIGNQ+iJZcd0ynoFeoom4nx1PeUEibI1W/gcWkuzU88Z72Y7qdv3 yR0vf7ZTAhkZfvwaT3cKNtiIIFsTTmXULBozYNjkA3dK7aupi4BFohzcC51g+yzOhVQGywDID hewr0a4dh8n9s7isEC1uailqY7ILv5NuAQ47UVYqrg0wbGYmVWeopwVPnskgOabiQVRXibjcy +5IGOltY0atnfAFlrAMv4QrJnqfL5PYWFWLDZauU0A4RMpqOAb3ilhJXuGFLfJlt1TvHc/h0b /DlSalxM9mfM7KnNjAY3/R1c30pSlYFgeYiXSbH5whmISxgZcmkvfq1qBL4ROPC6E7/M6ZKY1 ZwJhvoC0ZkTxKhCv8nh7LRC8Z+xVJQPgefY5Yk2hM4EyD00LroV8DiH2QkN0STU9yHO0DD8IO uBcmG5j2AdKJq3oAfPZb2Hl0FSPgySR4gY1AbmRr8kxnUi4QcoF+++U7Kr2wDzFgtbV6N1pyn 8afj0x/U/Yz6oJswBF8Do9o9Dw3C3Dr8fJZpzB//oIAYp2cmwNegk5jWZvgYL1epMsAOsSKzA KjM5wbQUPjy/E8KyoqEmlSI= X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jan 2017 17:22:12 -0000 On 2017-01-08 18:01, Allan Jude wrote: > On 2017-01-08 10:18, Nils wrote: >> Hello, I'm fighting to get vbox vdi images to run under FreeNAS and >> don't know what else to try. I've filed and commented on these two bugs: >> https://github.com/pr1ntf/iohyve/issues/227 >> https://github.com/pr1ntf/iohyve/issues/228 >> >> ...but I think the problem is not with bhyve itself or with iohyve, but >> either with grub-bhyve or ZFS. >> Running installation ISOs etc. works fine, but I need to get the VDIs >> going., >> >> Any pointers are welcome... >> Thanks >> Nils >> _______________________________________________ >> freebsd-virtualization@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" >> > How are you converting the .VDI to a raw image? bhyve does not yet > support the .VDI format, only raw. > I've done the conversion to raw with both VBoxManage and qemu-img, same result. I'm not sure where to add teh text flag, but I don't think that it's a problem, as grub should be running in a text console. What bothers me is that grub at the prompt claims not to recognize the (hd0): |grub> ls (hd0) Device hd0: No known filesystem detected - Total size 16777216 sectors ... where extracting the MBR and looking at it with fdisk shows the partitions... | From owner-freebsd-virtualization@freebsd.org Sun Jan 8 17:35:32 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BC98DCA5F6D for ; Sun, 8 Jan 2017 17:35:32 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8340418EB for ; Sun, 8 Jan 2017 17:35:32 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id C865D1341E for ; Sun, 8 Jan 2017 17:35:24 +0000 (UTC) Subject: Re: Help getting vbox .vdi imares to run under Freenas To: freebsd-virtualization@freebsd.org References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> <4b65f427-07dc-8b0c-1ac9-59c3c1533336@freebsd.org> <201dbbc3-53de-ba98-6a88-16dd06444bc5@gmx.net> From: Allan Jude Message-ID: <13af4cda-44d7-5b2c-7dbe-862b3fc8be40@freebsd.org> Date: Sun, 8 Jan 2017 12:35:21 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.6.0 MIME-Version: 1.0 In-Reply-To: <201dbbc3-53de-ba98-6a88-16dd06444bc5@gmx.net> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0baTpKHqAlk8OobqDVgePJwgr0XamP4J7" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jan 2017 17:35:32 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --0baTpKHqAlk8OobqDVgePJwgr0XamP4J7 Content-Type: multipart/mixed; boundary="m5MqIhtB6Bu9VPnK8xAGGeOPNQKsL2JvP"; protected-headers="v1" From: Allan Jude To: freebsd-virtualization@freebsd.org Message-ID: <13af4cda-44d7-5b2c-7dbe-862b3fc8be40@freebsd.org> Subject: Re: Help getting vbox .vdi imares to run under Freenas References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> <4b65f427-07dc-8b0c-1ac9-59c3c1533336@freebsd.org> <201dbbc3-53de-ba98-6a88-16dd06444bc5@gmx.net> In-Reply-To: <201dbbc3-53de-ba98-6a88-16dd06444bc5@gmx.net> --m5MqIhtB6Bu9VPnK8xAGGeOPNQKsL2JvP Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2017-01-08 12:21, Nils wrote: > On 2017-01-08 18:01, Allan Jude wrote: >> On 2017-01-08 10:18, Nils wrote: >>> Hello, I'm fighting to get vbox vdi images to run under FreeNAS and >>> don't know what else to try. I've filed and commented on these two bu= gs: >>> https://github.com/pr1ntf/iohyve/issues/227 >>> https://github.com/pr1ntf/iohyve/issues/228 >>> >>> ...but I think the problem is not with bhyve itself or with iohyve, b= ut >>> either with grub-bhyve or ZFS. >>> Running installation ISOs etc. works fine, but I need to get the VDIs= >>> going., >>> >>> Any pointers are welcome... >>> Thanks >>> Nils >>> _______________________________________________ >>> freebsd-virtualization@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >>> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@= freebsd.org" >>> >> How are you converting the .VDI to a raw image? bhyve does not yet >> support the .VDI format, only raw. >> >=20 > I've done the conversion to raw with both VBoxManage and qemu-img, same= > result. >=20 > I'm not sure where to add teh text flag, but I don't think that it's a > problem, as grub should be running in a text console. What bothers me i= s > that grub at the prompt claims not to recognize the (hd0): >=20 > |grub> ls (hd0) Device hd0: No known filesystem detected - Total size > 16777216 sectors ... where extracting the MBR and looking at it with > fdisk shows the partitions... | >=20 > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@fr= eebsd.org" >=20 Well, you are not telling grub to USE a partition, you are asking it to read (hd0) has a file system You likely want something like: ls (hd0,msdos1) --=20 Allan Jude --m5MqIhtB6Bu9VPnK8xAGGeOPNQKsL2JvP-- --0baTpKHqAlk8OobqDVgePJwgr0XamP4J7 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJYcnhcAAoJEBmVNT4SmAt+DQcP/A/Lg9PV72LpDdoZGcBlCXiH w40hj2g/1GQSo75RUcuiJlX3nTmpFW3ZRXJZh9F14i9fUs5c+K9pVoj1JJHIZ2VH EHl7ea5SzCSAWkdgVdMngEKJ/bYk3xX49lkpuIb9FX+0iQGjIeXRnoyjzoVYgyEB EYtSoEFQ/5ITa9Og2cdfMjEtlHSwY6qU+Zibyu7Fw5bkBhkhYTNpB3NoqqCPPSHT sqWMl3V1RTmuknGf9v7p8nA+i2LCHbxuMgDR2qACbsCwqzDLqfVSJzYVosD2wG3U Pzz+tphgkuxiSa4ele2dr9/TyVpiEuW2VnaZmK5F16MoHzVflU0JBI9jzVYZSc3w jh4PZwplU2P+IADUJZzwOWLSXvRB9Qq/2TteOUOPuu7furacEMjAyLNiC1OwkmDG 6ewpyd7jFt8SCePPmbGcxs02xsv56bSsvuhTI+D3/WBY8gokJZarLVcAsxixQxTU qa0gW91VuGh2I8snqyAk50oUf8189IMg5c5hBtYWjloMk1kpI8hJzjOrpNSUEGRs mw4ndIaztZs+65C0sBxU/k3qfycUHVhW/wDh0uGDjNWWRe3DVsCSt9F4PMYRS+YR WiZs8XJd0kgleGH8WApzhRJpp85OLdWsaoHGlSF8wrxT0x8Am+ehXcykHbLGDAIy /bhPc9psPn4wNhDIss3F =OQ6C -----END PGP SIGNATURE----- --0baTpKHqAlk8OobqDVgePJwgr0XamP4J7-- From owner-freebsd-virtualization@freebsd.org Sun Jan 8 18:04:46 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CAA43CA59F7 for ; Sun, 8 Jan 2017 18:04:46 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 8AD4019D6 for ; Sun, 8 Jan 2017 18:04:46 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 5564A20B4A16 for ; Mon, 9 Jan 2017 04:04:25 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 462D9280991 for ; Mon, 9 Jan 2017 04:04:25 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id YCPh6n7oL2rH for ; Mon, 9 Jan 2017 04:04:25 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 875142808C8; Mon, 9 Jan 2017 04:04:23 +1000 (AEST) Subject: Re: Help getting vbox .vdi imares to run under Freenas To: Allan Jude References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> <4b65f427-07dc-8b0c-1ac9-59c3c1533336@freebsd.org> <201dbbc3-53de-ba98-6a88-16dd06444bc5@gmx.net> <13af4cda-44d7-5b2c-7dbe-862b3fc8be40@freebsd.org> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: Date: Sun, 8 Jan 2017 10:04:38 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <13af4cda-44d7-5b2c-7dbe-862b3fc8be40@freebsd.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=NEAV23lmAAAA:8 a=bpN-wHtoY4BIDvUZYzAA:9 a=pILNOxqGKmIA:10 a=VlZU0XKO32wA:10 a=Bn2pgwyD2vrAyMmN8A2t:22 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jan 2017 18:04:46 -0000 >> |grub> ls (hd0) Device hd0: No known filesystem detected - Total size >> 16777216 sectors ... where extracting the MBR and looking at it with >> fdisk shows the partitions... | ... > Well, you are not telling grub to USE a partition, you are asking it to > read (hd0) has a file system > > You likely want something like: ls (hd0,msdos1) There's some additional info at https://github.com/pr1ntf/iohyve/issues/228 - > grub> ls > (hd0) (cd0) (host) > grub> ls (hd0) > Device hd0: No known filesystem detected - Total size 16777216 sectors ... so it looks like grub isn't able to auto-detect the partitions. later, Peter. From owner-freebsd-virtualization@freebsd.org Sun Jan 8 18:04:47 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9A8DFCA59F9 for ; Sun, 8 Jan 2017 18:04:47 +0000 (UTC) (envelope-from internationils@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.15.19]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 12C3619D8 for ; Sun, 8 Jan 2017 18:04:46 +0000 (UTC) (envelope-from internationils@gmx.net) Received: from [10.0.2.15] ([87.174.152.48]) by mail.gmx.com (mrgmx002 [212.227.17.190]) with ESMTPSA (Nemesis) id 0MMk99-1cOsWH2e6R-008X7d for ; Sun, 08 Jan 2017 19:04:38 +0100 Subject: Re: Help getting vbox .vdi imares to run under Freenas To: freebsd-virtualization@freebsd.org References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> <4b65f427-07dc-8b0c-1ac9-59c3c1533336@freebsd.org> <201dbbc3-53de-ba98-6a88-16dd06444bc5@gmx.net> <13af4cda-44d7-5b2c-7dbe-862b3fc8be40@freebsd.org> From: Nils Message-ID: <8e19f594-e281-6676-fecc-e1af058e2e22@gmx.net> Date: Sun, 8 Jan 2017 19:04:37 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.5.1 MIME-Version: 1.0 In-Reply-To: <13af4cda-44d7-5b2c-7dbe-862b3fc8be40@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:wHhBttm4K2IuwSNuYBCcz/7RqRR5+7qqHgbn647cUUJaU/kxgNF lgIjCOJyHLMuFXSPQ8Ko8q22FQIv0bnK2Bh2oZ4fXcMszehovvGoN9VBMmWoxM53bAgq2co KXI97n7H4uSM1jgp0Lwq36ELATCQtD2oqaPL1B5tOH11LYpJF6F66hG93xczweB5xahrTe4 WT+g1QHWB/4qWdrltywxg== X-UI-Out-Filterresults: notjunk:1;V01:K0:WkpBCxintAo=:LGJ5Xd/ebBk79AFlMlQcDi kPciGOY9orNCSq+N/xX6VGilKBvxdV036X2GCYaU7xmndXw3Bgdrc8YF1xewICUoqBI2CT0Wb R168lcRl5y/TFn3H8pG/XbomHTio7ofsnCl1LSlI0n0jbfnYW6wseXWF9nTKsFiVdryczaMho dfqLo/P4U33PAFuL9tbYcd8vWK2GAWBxGzmaDeY8uImnpFRnS0mhVO6Vm5JSJwIlCXdpx6cF7 j/Gpbf6g+w1tetf5CJ6X3JlFlf17e5lZIeLmz4Jycno52abvQkxmUzQu4Cx6yEVN5BHxdrBBr xmgOJHrxpw25iQ2a/pio/N9+fDFXYi8aox5M6NZRjS+vJk89qOu6cAMn7hIEwF9Br11CJpWEm V9H+JpCD7wiq8Qgo/FLq2HThfcyrzOuL4aKDml9HYs4WnLmuN9Qa8Ul/xjyNvCGZTcUobYE5y u6m3jSkMnz1eLzLudv3PBKLZ3fBwYfhm+8fpTBJXIpZxT+zTA62gTKqxQ5BEuveI7Vjv5QA5r QYBalijybYozuwfszSec93qONgHo0i++jTXT6YqK5PBXgg8SpCOaj74smYNLs2Daw2Z13X1pl LIL605vhezPzhDiOb54bb65AGz/nO8NSoGCiNk4u4qC5Ua6liAYwIjpp55YpRClmxaChax7KW GpNyCESXUrU9nBVNUcO07VaKRUYTAT2yerR13AJBd05nFp8m8FqI9UPUL7VprHq94xPsvnUh+ lanAcWZo80DhC7Spxus7S6aDDNhg+8u9KEQP5PqYDy51dNJ+nUemJ47U4BIihVJwIaRNY1kYq Ha11t+f X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jan 2017 18:04:47 -0000 On 2017-01-08 18:35, Allan Jude wrote: > On 2017-01-08 12:21, Nils wrote: >> On 2017-01-08 18:01, Allan Jude wrote: >>> On 2017-01-08 10:18, Nils wrote: >>>> Hello, I'm fighting to get vbox vdi images to run under FreeNAS and >>>> don't know what else to try. I've filed and commented on these two bugs: >>>> https://github.com/pr1ntf/iohyve/issues/227 >>>> https://github.com/pr1ntf/iohyve/issues/228 >>>> >>>> ...but I think the problem is not with bhyve itself or with iohyve, but >>>> either with grub-bhyve or ZFS. >>>> Running installation ISOs etc. works fine, but I need to get the VDIs >>>> going., >>>> >>>> Any pointers are welcome... >>>> Thanks >>>> Nils >>>> _______________________________________________ >>>> freebsd-virtualization@freebsd.org mailing list >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >>>> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" >>>> >>> How are you converting the .VDI to a raw image? bhyve does not yet >>> support the .VDI format, only raw. >>> >> I've done the conversion to raw with both VBoxManage and qemu-img, same >> result. >> >> I'm not sure where to add teh text flag, but I don't think that it's a >> problem, as grub should be running in a text console. What bothers me is >> that grub at the prompt claims not to recognize the (hd0): >> >> |grub> ls (hd0) Device hd0: No known filesystem detected - Total size >> 16777216 sectors ... where extracting the MBR and looking at it with >> fdisk shows the partitions... | >> >> _______________________________________________ >> freebsd-virtualization@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" >> > Well, you are not telling grub to USE a partition, you are asking it to > read (hd0) has a file system > > You likely want something like: ls (hd0,msdos1) > > grub> ls (hd0) Device hd0: No known filesystem detected - Total size 16777216 sectors grub> ls (hd0,msdos1) error: disk `hd0,msdos1' not found. grub> ...looking at the MBR (extracted with dd if=disk0 of=disk0-mbr count=1 bs=512) under linux with fdisk -l shows: |nils@dnet64:/mnt/nas/backup/tmp$ sudo fdisk -l disk0 Disk disk0: 8 GiB, 8589934592 bytes, 16777216 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: dos Disk identifier: 0x000f139a Device Boot Start End Sectors Size Id Type disk0p1 * 63 15952544 15952482 7,6G 83 Linux disk0p2 15952545 16771859 819315 400,1M 5 Extended disk0p5 15952608 16771859 819252 400M 82 Linux swap / Solaris| From owner-freebsd-virtualization@freebsd.org Sun Jan 8 20:59:42 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D22DBCA64D1 for ; Sun, 8 Jan 2017 20:59:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B8A6414C8 for ; Sun, 8 Jan 2017 20:59:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v08Kxgbe091807 for ; Sun, 8 Jan 2017 20:59:42 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 212711] [typo] bhyve: virtio-rnd PCI ID doesn't match viorng driver from Redhat Date: Sun, 08 Jan 2017 20:59:42 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: patch X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: mfc-stable10? mfc-stable11? X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jan 2017 20:59:43 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212711 --- Comment #4 from commit-hook@freebsd.org --- A commit references this bug: Author: grehan Date: Sun Jan 8 20:58:58 UTC 2017 New revision: 311702 URL: https://svnweb.freebsd.org/changeset/base/311702 Log: Use correct PCI device id for virtio-rng. This prevented the device from attaching with a Windows guest (most other guests use the device type for matching) PR: 212711 Submitted by: jbeich MFC after: 3 days Changes: head/usr.sbin/bhyve/virtio.h --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Sun Jan 8 21:00:46 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F31ABCA6704 for ; Sun, 8 Jan 2017 21:00:46 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D15DC19DE for ; Sun, 8 Jan 2017 21:00:46 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v08L01qS092914 for ; Sun, 8 Jan 2017 21:00:46 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201701082100.v08L01qS092914@kenobi.freebsd.org> From: bugzilla-noreply@FreeBSD.org To: freebsd-virtualization@FreeBSD.org Subject: Problem reports for freebsd-virtualization@FreeBSD.org that need special attention Date: Sun, 08 Jan 2017 21:00:46 +0000 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Jan 2017 21:00:47 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 212820 | FreeBSD 10-STABLE from latest HEAD and 11-RELEASE Open | 212721 | FreeBSD 11.0-RC2/RC3/RELEASE fails on Hyper-V 201 2 problems total for which you should take action. From owner-freebsd-virtualization@freebsd.org Mon Jan 9 03:10:15 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 60848CA6426 for ; Mon, 9 Jan 2017 03:10:15 +0000 (UTC) (envelope-from trentnthompson@gmail.com) Received: from mail-yw0-x22e.google.com (mail-yw0-x22e.google.com [IPv6:2607:f8b0:4002:c05::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2185616F3 for ; Mon, 9 Jan 2017 03:10:15 +0000 (UTC) (envelope-from trentnthompson@gmail.com) Received: by mail-yw0-x22e.google.com with SMTP id a10so415267237ywa.3 for ; Sun, 08 Jan 2017 19:10:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to:cc; bh=ZA6rlvdAHZ0riyOEfdu0oWMs14wMo3Iec0J41gTyZkA=; b=Nr12wYN94lAUiksm1OLz9YpvBOrHncPB7fM3Xj41Re6ogniCYZFzFwfcuSfOO4lW8r KQATOGi1ReIbBqBIIDUE8j/whLPfSmq4ZjG9MpCiiXLwvq7FKpVcWu+6oVYVqK1oLL8n iZPs3BgjbOQGDXBy9jj+c/S6v/AfNhE6TJEv/eAM/2DJIqV7KSwMAhZ5OF7CdmBdhWaS qKIVqqRBYzoI4Vs1hcPThLVXS3TAZ5J9nvYcpvbUdqQoANoUjhXjNSUW6JGZT1+Ml4Ag GVAePyoLGwSZv2UEJpoIQL2AZZ8KpDyuF3C0YS9zeWbUUbBA72hpY4tYfckHL6hMLUoW ryMA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=ZA6rlvdAHZ0riyOEfdu0oWMs14wMo3Iec0J41gTyZkA=; b=ILfS9ESLEFlbLFdfhl/CJ0ejfl+r8Q3Yu8BuxmjhXogh1j/SGghQA/3cvSWz90Tj4k aG15FdT2tUM6QGPby0UpSjCuuDSoAs3EjdW7zPp8evLKmGQVIYoIW/a6uFpa/VQ8Xyp8 8yvlbb/0JykQIWJjUj6JUDUfCXrtxHtdBuQ9Rp/Vj5yEUcyEg9wLdl3RuER7c25AC0NK bc1jiaVvtLf3l0PgVCeSKziZI85fc1Qcwaj/D71yh8xUC9bqq46giD1kYPc0MVBo9plD IS5pVq9Koy93+SukOQd2Rqo3xymOin3bgAGZCKCU3wg+R9kMs+J6Vz2A6rMA7VOq2Sxh qBPw== X-Gm-Message-State: AIkVDXK4EAEHBT0NvbjdHTX8eGhqAqgUvxnVC0e/rB1Su1TKJDgISg5Uh/9frK69NBeaZiWAzIDt+Qts7DtlVw== X-Received: by 10.129.172.36 with SMTP id k36mr15295839ywh.286.1483931414100; Sun, 08 Jan 2017 19:10:14 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.172.149 with HTTP; Sun, 8 Jan 2017 19:10:13 -0800 (PST) From: Trent Thompson Date: Sun, 8 Jan 2017 20:10:13 -0700 Message-ID: Subject: Re: Booting 9front on bhyve To: Piotr Kubaj Cc: FreeBSD virtualization Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jan 2017 03:10:15 -0000 Piotr, Thanks to work from Peter G, bhyve has been patched[1] to fix the PS/2 mouse issue in 9front. With this fixed, I was able to get 9front installed and running from a bhyve virtual machine using virtio-blk emulation. I was even able to set up networking[2] with virtio-net. Using ahci-hd and Intel 1000 emulation did not work for me. I'd also like to note that I had better success with partitioning the virtual hard drive as MBR in the installer, over GPT. I have heard of GPT working for some people, though. I uploaded a simple script to Gist[3] that I used to get this done, but as long as you use Virtio drivers you should be fine. [1] https://svnweb.freebsd.org/base?view=revision&revision=311699 [2] https://twitter.com/pr1ntf/status/817895393824382976/ [3] https://gist.github.com/pr1ntf/fb62e316b3007f55590d2397a08b0e52 Happy Hacking! -Trent From owner-freebsd-virtualization@freebsd.org Mon Jan 9 06:28:23 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ED470CA646B for ; Mon, 9 Jan 2017 06:28:23 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 7DEDE103D for ; Mon, 9 Jan 2017 06:28:23 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 22AB52011484 for ; Mon, 9 Jan 2017 16:28:02 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 106892809CC for ; Mon, 9 Jan 2017 16:28:02 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id D4rDuW19yG2O for ; Mon, 9 Jan 2017 16:28:01 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 56F4D280991; Mon, 9 Jan 2017 16:27:59 +1000 (AEST) Subject: Re: Arch Linux UEFI Bhyve Setup To: Vincent Olivier References: <680F62F7-9FD7-4FB3-A60A-DED3C784BE7F@up4.com> <8D5F1A67-DC29-4443-8128-B590CCC4B219@up4.com> From: Peter Grehan Cc: freebsd-virtualization@freebsd.org Message-ID: <850ef0cd-131a-8c18-ee0a-fe044293360d@freebsd.org> Date: Sun, 8 Jan 2017 22:28:16 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <8D5F1A67-DC29-4443-8128-B590CCC4B219@up4.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=IkcTkHD0fZMA:10 a=IgFoBzBjUZAA:10 a=GqK1-ANopE_QyggxADoA:9 a=QEXdDO2ut3YA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jan 2017 06:28:24 -0000 Hi Vincent, > Nothing works with UEFI console for me so I think I=E2=80=99m going to = stick > with BIOS-MBR-GRUB for now, unless you have an idea I can try=E2=80=A6 The live ISO can be repacked to work with a serial console. The only=20 complication is that the systemd-boot (aka gummiboot, Arch's UEFI=20 loader) are embedded in a FAT filesystem image. The repack steps are: 1. Install the cdrtools-devel package to get mkisofs 2. Extract the contents of the Arch iso into a working directory # cd /path/to/work_dir # tar xf /path/to/archlinux-2017.01.01-dual.iso 3. Find the label of the ISO (to be used in the repack) # isoinfo -d -i /path/to/archlinux-2017.01.01-dual.iso | grep=20 ^Volume\ id Volume id: ARCH_201701 4. Set up a vnode-backed md for the FAT boot image and mount it # mdconfig -f EFI/archiso/efiboot.img # mount_msdosfs /dev/md0 /mnt 5. Edit the config file, adding serial console lines to the 'options' lin= e # vi /mnt/loader/entries/archiso-x86_64.conf .... options archisobasedir=3Darch archisolabel=3DARCH_201701 console=3Dtt= y0=20 console=3DttyS0 6. Unmount the filesystem (updating the FAT filesystem image) and=20 cleanup the md # umount /mnt # mdconfig -d -u 0 7. Repack the ISO, using the volume ID from step 3. Change the name of the repacked iso to whatever is suitable with the '-o' parameter. # mkisofs \ -iso-level 3 \ -V "ARCH_201701" \ -J -joliet-long \ -b isolinux/isolinux.bin -no-emul-boot -c isolinux/boot.cat \ -boot-load-size 4 -boot-info-table \ -eltorito-alt-boot -eltorito-platform 0xEF \ -eltorito-boot EFI/archiso/efiboot.img \ -no-emul-boot \ -o ../arch_repack.iso . The system can now be booted with a serial console. An example command=20 line with serial output on the nmdm device is bhyve \ -c 2 \ -s 0,hostbridge \ -s 3,ahci-cd,/path/to/arch_repack.iso \ -s 10,virtio-net,tap0 \ -l bootrom,/path/to/BHYVE_UEFI_20160526.fd \ -l com1,/dev/nmdm0A \ -s 31,lpc \ -m 2G -H -w \ arch later, Peter. From owner-freebsd-virtualization@freebsd.org Mon Jan 9 09:46:56 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5756FCA577B for ; Mon, 9 Jan 2017 09:46:56 +0000 (UTC) (envelope-from lars@e-new.0x20.net) Received: from mail.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "0x20.net", Issuer "StartCom Class 1 DV Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 136041FFE for ; Mon, 9 Jan 2017 09:46:56 +0000 (UTC) (envelope-from lars@e-new.0x20.net) Received: from e-new.0x20.net (mail.0x20.net [IPv6:2001:aa8:fffb:1::3]) (using TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits)) (No client certificate requested) by mail.0x20.net (Postfix) with ESMTPS id 979AE6E0081; Mon, 9 Jan 2017 10:46:53 +0100 (CET) Received: from e-new.0x20.net (localhost [127.0.0.1]) by e-new.0x20.net (8.14.7/8.14.7) with ESMTP id v099krtR069333; Mon, 9 Jan 2017 10:46:53 +0100 (CET) (envelope-from lars@e-new.0x20.net) Received: (from lars@localhost) by e-new.0x20.net (8.14.7/8.14.7/Submit) id v099kqi2068803; Mon, 9 Jan 2017 10:46:52 +0100 (CET) (envelope-from lars) Date: Mon, 9 Jan 2017 10:46:52 +0100 From: Lars Engels To: Trent Thompson Cc: Piotr Kubaj , FreeBSD virtualization Subject: Re: Booting 9front on bhyve Message-ID: <20170109094652.GQ25026@e-new.0x20.net> References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="8RsyD0KswhpoK73Z" Content-Disposition: inline In-Reply-To: X-Editor: VIM - Vi IMproved 7.4 X-Operation-System: FreeBSD 8.4-RELEASE-p35 User-Agent: Mutt/1.5.23 (2014-03-12) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jan 2017 09:46:56 -0000 --8RsyD0KswhpoK73Z Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sun, Jan 08, 2017 at 08:10:13PM -0700, Trent Thompson wrote: > Piotr, > Thanks to work from Peter G, bhyve has been patched[1] to fix the PS/2 > mouse issue in 9front. With this fixed, I was able to get 9front installed > and running from a bhyve virtual machine using virtio-blk emulation. I w= as > even able to set up networking[2] with virtio-net. Using ahci-hd and Intel > 1000 emulation did not work for me. I'd also like to note that I had bett= er > success with partitioning the virtual hard drive as MBR in the installer, > over GPT. I have heard of GPT working for some people, though. I uploaded= a > simple script to Gist[3] that I used to get this done, but as long as you > use Virtio drivers you should be fine. >=20 > [1] https://svnweb.freebsd.org/base?view=3Drevision&revision=3D311699 > [2] https://twitter.com/pr1ntf/status/817895393824382976/ > [3] https://gist.github.com/pr1ntf/fb62e316b3007f55590d2397a08b0e52 >=20 Will you incorporate all your OS testing in iohyve? :) --8RsyD0KswhpoK73Z Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQF8BAEBCgBmBQJYc1wMXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RjQwMDE3RTRERjUzMTI1N0FGRTUxNDlF NTRDQjM3RDNBMDg5RDZEAAoJEOVMs306CJ1tGGEIALPycXbXBQem8id+iXXTZFOs 1bVaz2Ny+r2zJp7gFkUZDfHlnRx6n3/W3pijww6EE0jm14iF2EoAHhWum3wrvZTh +6HRde7wJ9PSGztu9Tt3v84DrZyKqbmZzgMlMN9OHimyFbXwj1enwrQ7gnKZqpuh 0HIHLjToyjmouo1GSreCgOlDxX7CS/N12qVGpGZg2J7lpPLtmHgdqKPOKDFQ+OH7 wUrkbzvZ05/vlKOg78a+VYaA6zaXJMiJfg9BRvwy2bx8fQD+NpRA/Swu/eGH7cLk thi9waTTUaWV/QCxzGSik6qgWW5OLFPJSTSBHd/yUgqLCwAiLlHWKL+ORprJxhw= =cnyC -----END PGP SIGNATURE----- --8RsyD0KswhpoK73Z-- From owner-freebsd-virtualization@freebsd.org Mon Jan 9 11:56:06 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0459DCA6324 for ; Mon, 9 Jan 2017 11:56:06 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 964E5120E; Mon, 9 Jan 2017 11:56:05 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id v09Bu1kb047431; Mon, 9 Jan 2017 12:56:01 +0100 (CET) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (titan.inop.mo1.omnilan.net [IPv6:2001:a60:f0bb:1::3:1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id ED376CB0; Mon, 9 Jan 2017 12:56:00 +0100 (CET) Message-ID: <58737A50.30506@omnilan.de> Date: Mon, 09 Jan 2017 12:56:00 +0100 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Peter Grehan CC: freebsd-virtualization@freebsd.org Subject: Re: [Bug 215740] [bhyve] utilizing passthru breaks raw device usage with virtio-blk | ahci-hd References: In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Greylist: ACL 119 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Mon, 09 Jan 2017 12:56:01 +0100 (CET) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jan 2017 11:56:06 -0000 Bezüglich bugzilla-noreply@freebsd.org's Nachricht vom 04.01.2017 21:37 (localtime): > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=215740 > … > --- Comment #1 from Peter Grehan --- > Would you be able to post a verbose dmesg (boot -v) ? Due to lack of knowledge I experimented with BIOS VT-d settings "Remap Interrupt enable" and "Remap DMA enable". No difference in any combination. Is there anything else I can provide to help narrowing down the problem? Has anybody else ever tried such a VM (using character special file as virtio-blk backend together with a passthru device)? With success? Thanks, -harry From owner-freebsd-virtualization@freebsd.org Mon Jan 9 12:41:49 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E97A7CA5682 for ; Mon, 9 Jan 2017 12:41:49 +0000 (UTC) (envelope-from pkubaj@anongoth.pl) Received: from mail.anongoth.pl (anongoth.pl [88.156.79.165]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "anongoth.pl", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 684D411FB for ; Mon, 9 Jan 2017 12:41:48 +0000 (UTC) (envelope-from pkubaj@anongoth.pl) Received: from mail (unknown [127.0.1.10]) by mail.anongoth.pl (Postfix) with ESMTP id 60C47774A; Mon, 9 Jan 2017 13:36:05 +0100 (CET) X-Virus-Scanned: amavisd-new at anongoth.pl Received: from mail.anongoth.pl ([127.0.1.10]) by mail (anongoth.pl [127.0.1.10]) (amavisd-new, port 10024) with LMTP id l_kk2As_BJyE; Mon, 9 Jan 2017 13:36:02 +0100 (CET) Received: from anongoth.pl (unknown [46.248.161.165]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) (Authenticated sender: pkubaj@anongoth.pl) by mail.anongoth.pl (Postfix) with ESMTPSA id DD97E7731; Mon, 9 Jan 2017 13:36:01 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=anongoth.pl; s=ANONGOTH; t=1483965362; bh=OnhJjTGbKkeMNAsFzSSLj0eWukLYTzaBkUMgItwQmqg=; h=Date:From:To:Cc:Subject:References:In-Reply-To; b=sRaM/W8ThFv77CvXt9st2AQWcOGV4LeY8D0A+AxMe1Dz6e/KjUicooQG5n+d6gkob 4yzjpyDNmOH68FKXJRVXw3kLTTNtx3ZP8n1voB+74/pTMap4zTT3HBMzmkUuA2Iqbp ex/m5jZnClOzIpGPVMGDPgJrZXp10x139TjBhu01MOK54T1lzPCXLf1+XP50I6Brfl wNNJTV+87zu/sG8PU76m/cCYoKLlAL0wkU49i80wI22sDvqZGdyQjuDtC5STV8/wCf y6CLO7kr8ykfNSg9gKWMOb6NdpTLRtyknxZjkrTv1mamES0u6nfb5ECQlOMgA69noY Xm9L7YtZZgL1Q== Date: Mon, 9 Jan 2017 13:36:00 +0100 From: Piotr Kubaj To: Lars Engels Cc: Trent Thompson , FreeBSD virtualization Subject: Re: Booting 9front on bhyve Message-ID: <20170109123600.GA131@chujemuje> References: <20170109094652.GQ25026@e-new.0x20.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="PNTmBPCT7hxwcZjr" Content-Disposition: inline In-Reply-To: <20170109094652.GQ25026@e-new.0x20.net> User-Agent: Mutt/1.7.2 (2016-11-26) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jan 2017 12:41:50 -0000 --PNTmBPCT7hxwcZjr Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Thanks, I've merged it manually with 11.0-RELEASE and will try to use it :) On 17-01-09 10:46:52, Lars Engels wrote: > On Sun, Jan 08, 2017 at 08:10:13PM -0700, Trent Thompson wrote: > > Piotr, > > Thanks to work from Peter G, bhyve has been patched[1] to fix the PS/2 > > mouse issue in 9front. With this fixed, I was able to get 9front instal= led > > and running from a bhyve virtual machine using virtio-blk emulation. I= was > > even able to set up networking[2] with virtio-net. Using ahci-hd and In= tel > > 1000 emulation did not work for me. I'd also like to note that I had be= tter > > success with partitioning the virtual hard drive as MBR in the installe= r, > > over GPT. I have heard of GPT working for some people, though. I upload= ed a > > simple script to Gist[3] that I used to get this done, but as long as y= ou > > use Virtio drivers you should be fine. > >=20 > > [1] https://svnweb.freebsd.org/base?view=3Drevision&revision=3D311699 > > [2] https://twitter.com/pr1ntf/status/817895393824382976/ > > [3] https://gist.github.com/pr1ntf/fb62e316b3007f55590d2397a08b0e52 > >=20 >=20 > Will you incorporate all your OS testing in iohyve? :) --=20 _______________________________________=20 / Had this been an actual emergency, we \ | would have fled in terror, and you | \ would not have been informed. / ---------------------------------------=20 \ ^__^ \ (oo)\_______ (__)\ )\/\ ||----w | || || --PNTmBPCT7hxwcZjr Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEycyIeNkkgohzsoorelmbhSCDnJ0FAlhzg68ACgkQelmbhSCD nJ0zBg//cG2k9EX2Exmip4uEcbYHhNT6s8631R6Xqm3zxXuICU4AnUk/Qlu/nImv cmufcMgAccHDNFXcklyY0rYmbS24l7al7Jr7AVaqAbt/UWx+502pn5eI7tojOk/R jJmCehBSlaE71N1W/ooKhfPXQt7dsEJ3uw0+Vzep1BHdzi8RsLdUnxVFiVkSfqxa LwP99GiiQ6QcSJp9GKsKgQpp94NOswnpB3Wm5eXo+w3veSziMQAwl7mnYvmRwAz/ oZwuYU/PlR1tIhCLomYd4B6+aWefFSbe1DuAAxJDVHNjyZ87paOaA8yePGIiL8Y+ zws4y2oLqu4YnbRt+Vt/GboOFR0fCnuaZ5eHlyT5U1O49dpUE+San2MkINoM5jIi JswepVquVadkRYw5TMCYMi55CPHJJu8fWdrGKcJZQDDodLLc2K8J0Uo6gJyafAUB JbiFffKfGNQLLdHA6JUXZf0MdgPrDeVlgx8wguxT9lWYYbc6yQ/7ijjlwygS8+oB 2vKaoEX/+pWNMrl6arbBfzQd2Uxt2nX9iJA6Es5rY9BGOphBDpqIUQuDR4yMGcio N+7zJqkMnEE0MFo60ZJzzrwT7ME1eD3SBnYTr/L40O6lhdJ0lw/uANxwqxELAvx5 +aeIEwPYlhwQ0GawwHTwAE4mZ7JF38rWeyC737Fb59ow6J/OYa4= =tNxS -----END PGP SIGNATURE----- --PNTmBPCT7hxwcZjr-- From owner-freebsd-virtualization@freebsd.org Mon Jan 9 13:35:29 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8007ACA6D44 for ; Mon, 9 Jan 2017 13:35:29 +0000 (UTC) (envelope-from tycho.nightingale@pluribusnetworks.com) Received: from mail-qt0-x236.google.com (mail-qt0-x236.google.com [IPv6:2607:f8b0:400d:c0d::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 487F11527 for ; Mon, 9 Jan 2017 13:35:29 +0000 (UTC) (envelope-from tycho.nightingale@pluribusnetworks.com) Received: by mail-qt0-x236.google.com with SMTP id x49so69432360qtc.2 for ; Mon, 09 Jan 2017 05:35:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pluribusnetworks.com; s=google; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=2p7Li431n/0NZTRrAoy+H0CHj/7r6/p83cMoQn/Jy9I=; b=maiZQeNGqhXzyWD3EWmgSf1+Dc3yFlbj/i5Nqx6p3L+RAh2yhSW31/uux/sjzaAU6W /NBoM/TRzOM4igdpX2yphU8cenODESAflZYEGZHeWk8FnH6nGrlpw4HXfz9GeCl0WUFT KIDXs908/sVFz/xdgIZ3AcpZoIXCnE3lVPdCY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=2p7Li431n/0NZTRrAoy+H0CHj/7r6/p83cMoQn/Jy9I=; b=BXHEX2kzBp00N7HKjeEGUpQkLjTfj1exTRQSBeYRsJq882xM8IWnZL+vzPj8D6Ena7 mbKDgS3B01x7YhLAmzfHWcAeeqOXvOWp8ywuMmAPkZ0o1WbRLYivMqpA5JGqcAqjGW5F ojc1G25wirylnLk+/aI00XiAmtQJdoTYVQ2beDYa+Zgl4GKrmL0ypzrA2DTtK1AYbaFQ tR9V7x5E8ZCscxsPazrOw6ZwDmIN7SC4/eWUQcUfVplujnlqrIxPCvvc5Q92ZiYNEbDg W52SolsCw8sHezAPffsJaomgH3hVQCLO6pZ1jmwVyV+i/4bV+H105xZ8+h9B9AxnB0Q3 dOtw== X-Gm-Message-State: AIkVDXICiwPiNCp46gkAvGTxPTmZDI29cIDKqc0/O+l3s7W9+zHkoRpBG4l1Tk2x3N3uTjdM X-Received: by 10.200.50.112 with SMTP id y45mr8468363qta.108.1483968928324; Mon, 09 Jan 2017 05:35:28 -0800 (PST) Received: from [10.0.1.5] (209-6-121-211.c3-0.arl-ubr1.sbo-arl.ma.cable.rcn.com. [209.6.121.211]) by smtp.gmail.com with ESMTPSA id s20sm5927304qtc.39.2017.01.09.05.35.27 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Mon, 09 Jan 2017 05:35:27 -0800 (PST) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Booting 9front on bhyve From: Tycho Nightingale In-Reply-To: <20161214224429.GA76434@DESKTOP1> Date: Mon, 9 Jan 2017 08:35:25 -0500 Cc: Peter Grehan , freebsd-virtualization@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <20161214181909.GA20065@chujemuje> <58c8124e-847b-3af9-eef0-faa27a45605a@freebsd.org> <20161214224429.GA76434@DESKTOP1> To: Piotr Kubaj X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 09 Jan 2017 13:35:29 -0000 Hi, On Dec 14, 2016, at 5:44 PM, Piotr Kubaj via freebsd-virtualization = wrote: > Is it possible to use other mouse drivers, ps2intellimouse etc.? Not at this point as the emulation doesn=92t currently support the ps2 = intellimouse. Tycho= From owner-freebsd-virtualization@freebsd.org Tue Jan 10 08:52:32 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72806CA7A1F for ; Tue, 10 Jan 2017 08:52:32 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 607C71FF5 for ; Tue, 10 Jan 2017 08:52:32 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 36466F802; Tue, 10 Jan 2017 00:33:33 -0800 (PST) Date: Tue, 10 Jan 2017 00:33:32 -0800 From: To: , Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170110003332.7cf8ba15@mscad14> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jan 2017 08:52:32 -0000 Howdy, virtualization zealots! This is in reply to maillist thread [0]. It so happens that I have to get GPU-accelerated OpenCL working on my machine, so I had a play with bhyve & PCI-e passthrough for VGA. I was using nVidia Quadro 600 (GF108) for testing (planning to use AMD/ATI for OpenCL, of course). I tried a Linux guest with the proprietary nVidia driver, and the result was that the driver couldn't init the VGA during boot: [ 1.394726] nvidia: module license 'NVIDIA' taints kernel. [ 1.395140] Disabling lock debugging due to kernel taint [ 1.412132] nvidia: module verification failed: signature and/or required key missing - tainting kernel [ 1.419359] nvidia 0000:00:04.0: can't derive routing for PCI INT A [ 1.419807] nvidia 0000:00:04.0: PCI INT A: no GSI [ 1.420157] NVRM: This PCI I/O region assigned to your NVIDIA device is invalid: [ 1.420157] NVRM: BAR1 is 0M @ 0x0 (PCI:0000:00:04.0) [ 1.421023] NVRM: The system BIOS may have misconfigured your GPU. [ 1.421476] nvidia: probe of 0000:00:04.0 failed with error -1 [ 1.437301] nvidia-nvlink: Nvlink Core is being initialized, major device number 247 [ 1.440094] NVRM: The NVIDIA probe routine failed for 1 device(s). [ 1.440530] NVRM: None of the NVIDIA graphics adapters were initialized! After adding the "pci=nocrs" Linux boot option (which, from what I understand, magically helps to [partially] workaround bhyve assigning addresses beyond host CPU's physically addressable space for PCIe memory-mapped registers), the guest couldn't finish booting, because bhyve would segfault. Turns out the what peripherals are used, and their order on the command line, are important. Edit: actually, looks like it's the number of CPUs (the '-c' flag's argument) that makes the difference; the machine has a CPU with 4 core, no multithreading. This didn't work (segfault): `bhyve -A -H -P -s 0:0,hostbridge -s 1:0,lpc -s 2:0,virtio-net,tap0 \ -s 3:0,virtio-blk,./bhyve_lunix.img \ -s 4:0,ahci-cd,./ubuntu-16.04.1-server-amd64.iso \ -s 5:0,passthru,1/0/0 -l com1,stdio -c 4 -m 1024M -S lunixguest` [...] [ OK ] Listening on Load/Save RF Kill Switch Status /dev/rfkill Watch. [ OK ] Reached target Swap. Assertion failed: (pi->pi_bar[baridx].type == PCIBAR_IO), function passthru_write, file /usr/src/usr.sbin/bhyve/pci_passthru.c, line 850. Abort (core dumped) But his worked, finally: `bhyve -c 1 -m 1024M -S -A -H -P -s 0:0,hostbridge -s 1:0,lpc \ -s 2:0,virtio-net,tap0 -s 3:0,virtio-blk,./bhyve_lunix.img \ -s 4:0,passthru,1/0/0 -l com1,stdio lunixguest` So, the guest booted, and didn't complain about non-addressable- -by-CPU BARs anymore. However, the same fate befall me as Dom had in this thread -- the driver loaded: [ 1.691216] nvidia: module verification failed: signature and/or required key missing - tainting kernel [ 1.696641] nvidia 0000:00:04.0: can't derive routing for PCI INT A [ 1.698093] nvidia 0000:00:04.0: PCI INT A: no GSI [ 1.699277] vgaarb: device changed decodes: PCI:0000:00:04.0,olddecodes=io+mem,decodes=none:owns=io+mem [ 1.701461] nvidia-nvlink: Nvlink Core is being initialized, major device number 247 [ 1.702649] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 375.26 Thu Dec 8 18:36:43 PST 2016 (using threaded interrupts) [ 1.705481] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 375.26 Thu Dec 8 18:04:14 PST 2016 [ 1.708941] [drm] [nvidia-drm] [GPU ID 0x00000004] Loading driver but couldn't talk to the card: [lost the log, but it was the same as Dom's: "NVRM: rm_init_adapter failed"]. So I decided to try test in a FreeBSD 10.3-STABLE guest. With older driver, or just loading 'nvidia' without modesetting, I got guest kernel panics [1]. I loaded 'nvidia-modeset', there was more success: Linux ELF exec handler installed Linux x86-64 ELF exec handler installed nvidia0: on vgapci0 vgapci0: child nvidia0 requested pci_enable_io vgapci0: attempting to allocate 1 MSI vectors (1 supported) msi: routing MSI IRQ 269 to local APIC 2 vector 51 vgapci0: using IRQ 269 for MSI vgapci0: child nvidia0 requested pci_enable_io nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 367.44 Wed Aug 17 22:05:09 PDT 2016 But: # nvidia-smi NVRM: Xid (PCI:0000:00:04): 62, !2369(0000) NVRM: RmInitAdapter failed! (0x26:0x65:1072) nvidia0: NVRM: rm_init_adapter() failed! No devices were found It also panicked after starting Xorg. After stumbling upon some Xen forums, I found the solution: nVidia crippled the driver so that it detects virtualization environment, and refuses to attach to anything but high-end pro cards! Those bastards [if the speculation is true]! GTX960 didn't work. Quadro 600 didn't work. So I tried with a Quadro 2000: root@fbsd12tst:~ # sync root@fbsd12tst:~ # kldload nvidia-modeset Linux ELF exec handler installed nvidia0: on vgapci0 vgapci0: child nvidia0 requested pci_enable_io vgapci0: attempting to allocate 1 MSI vectors (1 supported) msi: routing MSI IRQ 269 to local APIC 3 vector 51 vgapci0: using IRQ 269 for MSI vgapci0: child nvidia0 requested pci_enable_io random: harvesting attach, 8 bytes (4 bits) from nvidia0 [a bit more]Success! However: root@fbsd12tst:~ # nvidia-smi acquiring duplicate lock of same type: "os.lock_sx" 1st os.lock_sx @ nvidia_os.c:599 2nd os.lock_sx @ nvidia_os.c:599 stack backtrace: #0 0xffffffff80aa6780 at witness_debugger+0x70 #1 0xffffffff80aa6683 at witness_checkorder+0xde3 #2 0xffffffff80a4fac2 at _sx_xlock+0x72 #3 0xffffffff82a515c2 at os_acquire_mutex+0x32 #4 0xffffffff82a21068 at _nv016673rm+0x18 Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0xfffffe004f601088 fault code = supervisor write data, reserved bits in PTE instruction pointer = 0x20:0xffffffff82a512e3 stack pointer = 0x28:0xfffffe0000221138 frame pointer = 0x28:0xfffffe0001a76ba8 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 634 (nvidia-smi) [ thread pid 634 tid 100100 ] Stopped at os_mem_copy+0xf3: movb %dl,(%rax) db> bt Tracing pid 634 tid 100100 td 0xfffff8000b866500 os_mem_copy() at os_mem_copy+0xf3/frame 0xfffffe0001a76ba8 ??() at 0xfffff8000b8beb00 db> (I upgraded to FreeBSD 12.0-CURRENT (GENERIC) #0 r311659, but initially did the test with Quadro 2000 on the same 10.3-STABLE as before, with the same results). Linux succeeds loading the driver with Quadro 2000, too: [ 1.374925] nvidia: module license 'NVIDIA' taints kernel. [ 1.375348] Disabling lock debugging due to kernel taint [ 1.400506] nvidia: module verification failed: signature and/or required key missing - tainting kernel [ 1.413539] nvidia 0000:00:04.0: can't derive routing for PCI INT A [ 1.414003] nvidia 0000:00:04.0: PCI INT A: no GSI [ 1.414417] vgaarb: device changed decodes: PCI:0000:00:04.0,olddecodes=io+mem,decodes=none:owns=io+mem [ 1.421807] nvidia-nvlink: Nvlink Core is being initialized, major device number 247 [ 1.422369] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 375.26 Thu Dec 8 18:36:43 PST 2016 (using threaded interrupts) [ 1.424568] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 375.26 Thu Dec 8 18:04:14 PST 2016 [ 1.426837] [drm] [nvidia-drm] [GPU ID 0x00000004] Loading driver But I get the same assertion and segfault from bhyve if I try to run `nvidia-smi` after the OS finished booting [at least it seemed to before, but can't get it to finish booting now, just hangs]. And now to the point: how would one go about fixing bhyve's tendency to segfault because of assert (it is saying that something is still very wrong?), and get Linux working with the GPU? And what to do about FreeBSD's guest kernel panics? P.S.: please CC, as I'm not subscribed. [0] https://lists.freebsd.org/pipermail/freebsd-virtualization/2016-September/004704.html [1] Fatal trap 12: page fault while in kernel mode cpuid = 3; apic id = 03 fault virtual address = 0xfffffe003d7c508c fault code = supervisor write data, reserved bits in PTE instruction pointer = 0x20:0xffffffff820bb5d5 stack pointer = 0x28:0xfffffe003d69d380 frame pointer = 0x28:0xfffffe000154ce68 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 5084 (nvidia-smi) trap number = 12 panic: page fault -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Tue Jan 10 09:05:38 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F3184CA6040 for ; Tue, 10 Jan 2017 09:05:38 +0000 (UTC) (envelope-from misc-freebsd@talk2dom.com) Received: from mail.shmtech.biz (mail.shmtech.biz [IPv6:2001:41c9:1:37::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9A0E2162D for ; Tue, 10 Jan 2017 09:05:38 +0000 (UTC) (envelope-from misc-freebsd@talk2dom.com) Received: from [10.0.1.100] (originc9729.pndsl.co.uk [84.92.225.78]) (authenticated bits=0) by mail.shmtech.biz (8.15.2/8.15.2) with ESMTPSA id v0A95Vmh087303 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Tue, 10 Jan 2017 09:05:32 GMT (envelope-from misc-freebsd@talk2dom.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=talk2dom.com; s=shmtech4; t=1484039132; x=1484643932; bh=NyjutCOsePJZza7W/w7DFfNdoAMRTjXCv1Ra7mQupLg=; h=Subject:To:References:From:Date:In-Reply-To; b=RStYwxXMVFJqcv0z5DOz9VcoUp8Des5Pb4uJPMcWEZ00i7OcGb6lWeAeVvHCqQ9BK AhhVhhxROwZg2yw5qR+i8vZ0I9XuJbKsOJCXGGFjBFmHEI6l6jrQ03mQ/2pOEhp/eI AA5Po/rmQ5xig0UkhJz16cKisr+NsyOR1sdIDVn4= X-Authentication-Warning: sendmail: Host originc9729.pndsl.co.uk [84.92.225.78] claimed to be [10.0.1.100] Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org, freebsd-virtualization@freebsd.org References: <20170110003332.7cf8ba15@mscad14> From: Dom Message-ID: <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> Date: Tue, 10 Jan 2017 09:05:31 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20170110003332.7cf8ba15@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jan 2017 09:05:39 -0000 With QEMU, they have the "kvm=off" option which hides hypervisor info from the guest. See: https://www.redhat.com/archives/libvir-list/2014-August/msg00512.html I did try to replicate this a while back but didn't have much success - maybe I missed a flag? The QEMU diff seems relatively small, see: http://lists.gnu.org/archive/html/qemu-devel/2014-06/msg00302.html Having another go at doing this is on my to-do list, but not very near the top! From owner-freebsd-virtualization@freebsd.org Tue Jan 10 09:39:49 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3264CCA6CF1 for ; Tue, 10 Jan 2017 09:39:49 +0000 (UTC) (envelope-from misc-freebsd@talk2dom.com) Received: from mail.shmtech.biz (mail.shmtech.biz [IPv6:2001:41c9:1:37::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B30EF1584 for ; Tue, 10 Jan 2017 09:39:48 +0000 (UTC) (envelope-from misc-freebsd@talk2dom.com) Received: from [10.0.1.100] (originc9729.pndsl.co.uk [84.92.225.78]) (authenticated bits=0) by mail.shmtech.biz (8.15.2/8.15.2) with ESMTPSA id v0A9diGV089583 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Tue, 10 Jan 2017 09:39:44 GMT (envelope-from misc-freebsd@talk2dom.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=talk2dom.com; s=shmtech4; t=1484041184; x=1484645984; bh=uo1NIAFWHbhLaFpyMURCeev6rC7n4WR4lSaAhRvagOg=; h=Subject:To:References:From:Date:In-Reply-To; b=Twa3HT1WAL25IBgzG9CjnF3zlue/uhpWG/reOLTnRGPopfEFpiEzN5cf4ssIb5IL6 0mvDn+YVDGCqmt8iOjWN5C2rjsnnsKGXb5DEKzI3H7iZR+/vQ90cJ1djDCchrY0iAp 6GKT2+6P78xGiIx925rxa46CyH/GHeEKQJreb6HU= X-Authentication-Warning: sendmail: Host originc9729.pndsl.co.uk [84.92.225.78] claimed to be [10.0.1.100] Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org, freebsd-virtualization@freebsd.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> From: Dom Message-ID: <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> Date: Tue, 10 Jan 2017 09:39:43 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 10 Jan 2017 09:39:49 -0000 Found my original attempt by modifying /usr/src/sys/amd64/vmm/x86.c Unified diff follows, but this didn't work for me. ("bhyve_id[]" commented out to prevent compiler complaints) There doesn't seem to be support for CPUID 0x40000001 in bhyve either. --- x86.c.orig 2016-09-11 14:40:22.410462000 +0100 +++ x86.c 2016-09-11 15:53:14.182186000 +0100 @@ -52,7 +52,7 @@ #define CPUID_VM_HIGH 0x40000000 -static const char bhyve_id[12] = "bhyve bhyve "; +/* static const char bhyve_id[12] = "bhyve bhyve "; */ static uint64_t bhyve_xcpuids; SYSCTL_ULONG(_hw_vmm, OID_AUTO, bhyve_xcpuids, CTLFLAG_RW, &bhyve_xcpuids, 0, @@ -236,7 +236,7 @@ regs[2] &= ~(CPUID2_VMX | CPUID2_EST | CPUID2_TM2); regs[2] &= ~(CPUID2_SMX); - regs[2] |= CPUID2_HV; + /* regs[2] |= CPUID2_HV; */ if (x2apic_state != X2APIC_DISABLED) regs[2] |= CPUID2_X2APIC; @@ -463,12 +463,15 @@ } break; + /* + * Don't expose KVM to guest case 0x40000000: regs[0] = CPUID_VM_HIGH; bcopy(bhyve_id, ®s[1], 4); bcopy(bhyve_id + 4, ®s[2], 4); bcopy(bhyve_id + 8, ®s[3], 4); break; + */ default: /* From owner-freebsd-virtualization@freebsd.org Wed Jan 11 02:01:20 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3E047CA9849 for ; Wed, 11 Jan 2017 02:01:20 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 2C1B51C4C for ; Wed, 11 Jan 2017 02:01:19 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id BCC47F802; Tue, 10 Jan 2017 18:01:18 -0800 (PST) Date: Tue, 10 Jan 2017 18:01:17 -0800 From: To: Cc: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170110180117.7f246b5a@mscad14> In-Reply-To: <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2017 02:01:20 -0000 > Found my original attempt by modifying /usr/src/sys/amd64/vmm/x86.c > Unified diff follows, but this didn't work for me. > ("bhyve_id[]" commented out to prevent compiler complaints) Who knows what sort of trickery nVidia's driver is up to besides CPUID when determining the presence of virtualization. Regardless of that, VGA PCIe passthrough does not work in bhyve even with Quadro 2000 (which Xen people have had success with). The problem appears to be in the area of assigning memory-mapped I/O ranges by bhyve for the VGA card to a region outside of the CPU's addressable space; i.e., bhyve does not check CPUID's 0x80000008 AL value (0x27 for my CPU, which is 39 bits -- while bhyve assigns 0xd000000000 & above for the large Prefetch Memory chunks, which requires 40 address bits). At least this is my understanding of why VGA passthrough does not work. This seems easy to fix. Could someone who knows better have a look? Unlike Linux, FreeBSD has no problem assigning BAR range outside addressable range, and then panics when trying to write to these virtual memory addresses. See [0] below. > There doesn't seem to be support for CPUID 0x40000001 in bhyve either. What is it supposed to do? [0] Linux dmesg: [ 0.204799] PCI: MMCONFIG for domain 0000 [bus 00-ff] at [mem 0xe0000000-0xefffffff] (base 0xe0000000) [ 0.205474] PCI: MMCONFIG at [mem 0xe0000000-0xefffffff] reserved in ACPI motherboard resources [ 0.206080] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.207306] ACPI: PCI Root Bridge [PC00] (domain 0000 [bus 00]) [ 0.207724] acpi PNP0A03:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI] [ 0.208291] acpi PNP0A03:00: _OSC failed (AE_NOT_FOUND); disabling ASPM [ 0.208759] acpi PNP0A03:00: host bridge window [mem 0xd000000000-0xd00c0fffff window] (ignored, not CPU addressable) [ 0.209517] PCI host bridge to bus 0000:00 [ 0.209808] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.210281] pci_bus 0000:00: root bus resource [io 0x0d00-0x1fff window] [ 0.210752] pci_bus 0000:00: root bus resource [io 0x2000-0x211f window] [ 0.211224] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xc40fffff window] [ 0.211743] pci_bus 0000:00: root bus resource [bus 00] [...] [ 0.223902] PCI: Using ACPI for IRQ routing [ 0.265987] pci 0000:00:03.0: can't claim BAR 1 [mem 0xd000000000-0xd007ffffff 64bit pref]: no compatible bridge window [ 0.266735] pci 0000:00:03.0: can't claim BAR 3 [mem 0xd008000000-0xd00bffffff 64bit pref]: no compatible bridge window [ 0.284717] pci 0000:00:03.0: can't claim BAR 6 [mem 0xf6000000-0xf607ffff pref]: no compatible bridge window [...] [ 0.285407] pci 0000:00:03.0: BAR 1: no space for [mem size 0x08000000 64bit pref] [ 0.285933] pci 0000:00:03.0: BAR 1: trying firmware assignment [mem 0xd000000000-0xd007ffffff 64bit pref] [ 0.286599] pci 0000:00:03.0: BAR 1: [mem 0xd000000000-0xd007ffffff 64bit pref] conflicts with PCI mem [mem 0x00000000-0x7fffffffff] [ 0.287419] pci 0000:00:03.0: BAR 1: failed to assign [mem size 0x08000000 64bit pref] [ 0.287968] pci 0000:00:03.0: BAR 3: no space for [mem size 0x04000000 64bit pref] [ 0.288506] pci 0000:00:03.0: BAR 3: trying firmware assignment [mem 0xd008000000-0xd00bffffff 64bit pref] [ 0.289173] pci 0000:00:03.0: BAR 3: [mem 0xd008000000-0xd00bffffff 64bit pref] conflicts with PCI mem [mem 0x00000000-0x7fffffffff] [ 0.289992] pci 0000:00:03.0: BAR 3: failed to assign [mem size 0x04000000 64bit pref] [ 0.290539] pci 0000:00:03.0: BAR 6: assigned [mem 0xc0080000-0xc00fffff pref] [ 0.291039] pci 0000:00:01.0: BAR 6: assigned [mem 0xc0002000-0xc00027ff pref] [ 0.291540] pci 0000:00:02.0: BAR 6: assigned [mem 0xc0002800-0xc0002fff pref] Cannot get output from Linux's `lspci -vvn` booted with "pci=nocrs" kernel option, as it hangs now close to the end of boot process (not sure why, was able to finish booting before). Another machine: vgapci0@pci0:1:0:0: class=0x030000 card=0x083510de chip=0x0df810de rev=0xa1 hdr=0x00 vendor = 'nVidia Corporation' device = 'GF108 [Quadro 600]' class = display subclass = VGA bar [10] = type Memory, range 32, base 0xfa000000, size 16777216, enabled bar [14] = type Prefetchable Memory, range 64, base 0xe8000000, size 134217728, enabled bar [1c] = type Prefetchable Memory, range 64, base 0xf0000000, size 33554432, enabled bar [24] = type I/O Port, range 32, base 0xe000, size 128, enabled hdac0@pci0:1:0:1: class=0x040300 card=0x083510de chip=0x0bea10de rev=0xa1 hdr=0x00 vendor = 'nVidia Corporation' device = 'GF108 High Definition Audio Controller' class = multimedia subclass = HDA bar [10] = type Memory, range 32, base 0xfb080000, size 16384, enabled Host: ppt0@pci0:1:0:0: class=0x030000 card=0x084a10de chip=0x0dd810de rev=0xa1 hdr=0x00 vendor = 'NVIDIA Corporation' device = 'GF106GL [Quadro 2000]' class = display subclass = VGA bar [10] = type Memory, range 32, base 0xf4000000, size 33554432, enabled bar [14] = type Prefetchable Memory, range 64, base 0xe0000000, size 134217728, enabled bar [1c] = type Prefetchable Memory, range 64, base 0xe8000000, size 67108864, enabled bar [24] = type I/O Port, range 32, base 0xe000, size 128, enabled ppt1@pci0:1:0:1: class=0x040300 card=0x084a10de chip=0x0be910de rev=0xa1 hdr=0x00 vendor = 'NVIDIA Corporation' device = 'GF106 High Definition Audio Controller' class = multimedia subclass = HDA bar [10] = type Memory, range 32, base 0xf6080000, size 16384, enabled Guest: vgapci0@pci0:0:4:0: class=0x030000 card=0x084a10de chip=0x0dd810de rev=0xa1 hdr=0x00 vendor = 'NVIDIA Corporation' device = 'GF106GL [Quadro 2000]' class = display subclass = VGA bar [10] = type Memory, range 32, base 0xc2000000, size 33554432, enabled bar [14] = type Prefetchable Memory, range 64, base 0xd000000000, size 134217728, enabled bar [1c] = type Prefetchable Memory, range 64, base 0xd008000000, size 67108864, enabled bar [24] = type I/O Port, range 32, base 0x2080, size 128, enabled Guest dmesg: found-> vendor=0x10de, dev=0x0dd8, revid=0xa1 domain=0, bus=0, slot=4, func=0 class=03-00-00, hdrtype=0x00, mfdev=0 cmdreg=0x0403, statreg=0x0010, cachelnsz=16 (dwords) lattimer=0x00 (0 ns), mingnt=0x00 (0 ns), maxlat=0x00 (0 ns) intpin=a, irq=16 powerspec 3 supports D0 D3 current D0 MSI supports 1 message, 64 bit map[10]: type Memory, range 32, base 0xc2000000, size 25, enabled pcib0: allocated type 3 (0xc2000000-0xc3ffffff) for rid 10 of pci0:0:4:0 map[14]: type Prefetchable Memory, range 64, base 0xd000000000, size 27, enabled pcib0: allocated type 3 (0xd000000000-0xd007ffffff) for rid 14 of pci0:0:4:0 map[1c]: type Prefetchable Memory, range 64, base 0xd008000000, size 26, enabled pcib0: allocated type 3 (0xd008000000-0xd00bffffff) for rid 1c of pci0:0:4:0 map[24]: type I/O Port, range 32, base 0x2080, size 7, enabled pcib0: allocated type 4 (0x2080-0x20ff) for rid 24 of pci0:0:4:0 pcib0: no PRT entry for 0.4.INTA -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Wed Jan 11 09:45:51 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9E7CCAAFD4 for ; Wed, 11 Jan 2017 09:45:51 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id B79351821 for ; Wed, 11 Jan 2017 09:45:51 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 9E5A9F804; Wed, 11 Jan 2017 01:45:45 -0800 (PST) Date: Wed, 11 Jan 2017 01:45:44 -0800 From: To: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111014544.70670784@mscad14> In-Reply-To: <20170110180117.7f246b5a@mscad14> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2017 09:45:51 -0000 > The problem appears to be in the area of assigning memory-mapped > I/O ranges by bhyve for the VGA card to a region outside of the > CPU's addressable space; i.e., bhyve does not check CPUID's > 0x80000008 AL value (0x27 for my CPU, which is 39 bits -- while > bhyve assigns 0xd000000000 & above for the large Prefetch Memory > chunks, which requires 40 address bits). At least this is my > understanding of why VGA passthrough does not work. To test this, I tried writing to PCI BARs in FreeBSD guest using `pciconf -w`. Not much use that was: I could read back the values written to the registers (e.g., `pciconf -r pci0:0:4:0 0x14:48`), but `pciconf -lvb` still showed the same huge base addresses -- they did not want to change. OK, I had enough of that. So I went to dig in the source, and changed the "#define PCI_EMUL_MEMBASE64" from '0xD000000000UL' to '0x3400000000UL' in src/usr.sbin/bhyve/pci_emul.c. Recompiled bhyve, booted up FreeBSD, and: # pciconf -lvb [...] vgapci0@pci0:0:4:0: class=0x030000 card=0x084a10de chip=0x0dd810de rev=0xa1 hdr=0x00 vendor = 'NVIDIA Corporation' device = 'GF106GL [Quadro 2000]' class = display subclass = VGA bar [10] = type Memory, range 32, base 0xc2000000, size 33554432, enabled bar [14] = type Prefetchable Memory, range 64, base 0x3400000000, size 134217728, enabled bar [1c] = type Prefetchable Memory, range 64, base 0x3408000000, size 67108864, enabled bar [24] = type I/O Port, range 32, base 0x2080, size 128, enabled ...a-a-and: # kldload nvidia-modeset Linux ELF exec handler installed nvidia0: on vgapci0 vgapci0: child nvidia0 requested pci_enable_io vgapci0: attempting to allocate 1 MSI vectors (1 supported) msi: routing MSI IRQ 269 to local APIC 3 vector 51 vgapci0: using IRQ 269 for MSI vgapci0: child nvidia0 requested pci_enable_io random: harvesting attach, 8 bytes (4 bits) from nvidia0 # nvidia-smi acquiring duplicate lock of same type: "os.lock_sx" 1st os.lock_sx @ nvidia_os.c:599 2nd os.lock_sx @ nvidia_os.c:599 stack backtrace: #0 0xffffffff80aa6780 at witness_debugger+0x70 #1 0xffffffff80aa6683 at witness_checkorder+0xde3 #2 0xffffffff80a4fac2 at _sx_xlock+0x72 #3 0xffffffff82a515c2 at os_acquire_mutex+0x32 #4 0xffffffff82a21068 at _nv016673rm+0x18 Tue Jan 10 17:06:48 2017 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 367.44 Driver Version: 367.44 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===============================+======================+======================| | 0 Quadro 2000 Off | 0000:00:04.0 Off | N/A | | 30% 35C P8 N/A / N/A | 0MiB / 963MiB | 0% Default | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | No running processes found | +-----------------------------------------------------------------------------+ Beauty! It's very slow to execute, though. And Xorg is not in a hurry to start working: [ 204.724] (--) PCI:*(0:0:4:0) 10de:0dd8:10de:084a rev 161, Mem @ 0xc2000000/33554432, 0x3400000000/134217728, 0x3408000000/67108864, I/O @ 0x00002080/128, BIOS @ 0x????????/65536 [...] [ 204.736] (**) NVIDIA(0): Depth 24, (--) framebuffer bpp 32 [ 204.736] (==) NVIDIA(0): RGB weight 888 [ 204.736] (==) NVIDIA(0): Default visual is TrueColor [ 204.736] (==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0) [ 204.738] (**) NVIDIA(0): Enabling 2D acceleration [ 213.674] (--) NVIDIA(0): Valid display device(s) on GPU-0 at PCI:0:4:0 [ 213.674] (--) NVIDIA(0): CRT-0 [ 213.674] (--) NVIDIA(0): DFP-0 (boot) [ 213.674] (--) NVIDIA(0): DFP-1 [ 213.674] (--) NVIDIA(0): DFP-2 [ 213.674] (--) NVIDIA(0): DFP-3 [ 213.675] (--) NVIDIA(0): DFP-4 [ 213.698] (--) NVIDIA(0): CRT-0: disconnected [ 213.698] (--) NVIDIA(0): CRT-0: 400.0 MHz maximum pixel clock [ 213.698] (--) NVIDIA(0): [ 213.743] (--) NVIDIA(0): DELL 2007FP (DFP-0): connected [ 213.743] (--) NVIDIA(0): DELL 2007FP (DFP-0): Internal TMDS [ 213.743] (--) NVIDIA(0): DELL 2007FP (DFP-0): 330.0 MHz maximum pixel clock [...] [ 213.747] (II) NVIDIA(0): NVIDIA GPU Quadro 2000 (GF106GL) at PCI:0:4:0 (GPU-0) [ 213.747] (--) NVIDIA(0): Memory: 1048576 kBytes [ 213.747] (--) NVIDIA(0): VideoBIOS: 70.06.0d.00.02 [ 213.747] (II) NVIDIA(0): Detected PCI Express Link width: 16X [ 213.748] (**) NVIDIA(0): Using HorizSync/VertRefresh ranges from the EDID for display [ 213.748] (**) NVIDIA(0): device DELL 2007FP (DFP-0) (Using EDID frequencies has [ 213.748] (**) NVIDIA(0): been enabled on all display devices.) [...] [ 213.751] (II) NVIDIA(0): Virtual screen size determined to be 1600 x 1200 [ 213.761] (--) NVIDIA(0): DPI set to (99, 98); computed from "UseEdidDpi" X config [ 213.761] (--) NVIDIA(0): option [ 213.761] (--) Depth 24 pixmap format is 32 bpp [ 213.767] (II) NVIDIA: Reserving 12288.00 MB of virtual memory for indirect memory [ 213.767] (II) NVIDIA: access. [ 216.789] (EE) NVIDIA(GPU-0): Failed to initialize DMA. [ 216.789] (EE) *** Aborting *** [ 216.791] (EE) NVIDIA(0): Failed to allocate push buffer [ 216.839] (EE) Fatal server error: [ 216.839] (EE) AddScreen/ScreenInit failed for driver 0 Linux still doesn't work (curse Ubuntu! what a mess. It tried to start Xorg at boot, so I managed to disable that, but no matter what, I couldn't stop it from trying to run 'nvidia-smi' at boot! And trust me, I tried a lot. I removed all the scripts related to nvidia, /etc/udev/ is basically empty [/etc just looks like a pile-up of crap, wow!], yet /usr/bin/nvidia-smi still tried to run by itself until I moved it away). dmesg: [ 1.390957] nvidia: module verification failed: signature and/or required key missing - tainting kernel [ 1.394715] nvidia 0000:00:04.0: can't derive routing for PCI INT A [ 1.395185] nvidia 0000:00:04.0: PCI INT A: no GSI [ 1.414173] vgaarb: device changed decodes: PCI:0000:00:04.0,olddecodes=io+mem,decodes=none:owns=io+mem [ 1.417062] nvidia-nvlink: Nvlink Core is being initialized, major device number 247 [ 1.417609] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 375.26 Thu Dec 8 18:36:43 PST 2016 (using threaded interrupts) [ 1.419820] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 375.26 Thu Dec 8 18:04:14 PST 2016 [ 1.422067] [drm] [nvidia-drm] [GPU ID 0x00000004] Loading driver [...] [ 3.904893] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 246 # lspci -vvn 00:04.0 0300: 10de:0dd8 (rev a1) (prog-if 00 [VGA controller]) Subsystem: 10de:084a Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B7F3CAA808 for ; Wed, 11 Jan 2017 10:08:27 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 2CFC91388 for ; Wed, 11 Jan 2017 10:08:26 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id D7DEC2041641 for ; Wed, 11 Jan 2017 20:07:59 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id D2E69280991 for ; Wed, 11 Jan 2017 20:07:59 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ncQi3bRcgIgB for ; Wed, 11 Jan 2017 20:07:59 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id B8513280500; Wed, 11 Jan 2017 20:07:57 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> Date: Wed, 11 Jan 2017 02:08:12 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170111014544.70670784@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=FO24ivJdlaGtZuhHtzoA:9 a=_0mYaXJXlwf1-dBP:21 a=SLvJyNt8ObrNJ6nn:21 a=pILNOxqGKmIA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2017 10:08:27 -0000 Hi, >> The problem appears to be in the area of assigning memory-mapped >> I/O ranges by bhyve for the VGA card to a region outside of the >> CPU's addressable space; i.e., bhyve does not check CPUID's >> 0x80000008 AL value (0x27 for my CPU, which is 39 bits -- while >> bhyve assigns 0xd000000000 & above for the large Prefetch Memory >> chunks, which requires 40 address bits). That's correct - it's a bug in bhyve. > To test this, I tried writing to PCI BARs in FreeBSD guest using > `pciconf -w`. Not much use that was: I could read back the values > written to the registers (e.g., `pciconf -r pci0:0:4:0 0x14:48`), > but `pciconf -lvb` still showed the same huge base addresses -- > they did not want to change. PCI passthru doesn't allow the BAR values to be modified (this could be changed, but it's a lot of work for little gain). > OK, I had enough of that. So I went to dig in the source, and > changed the "#define PCI_EMUL_MEMBASE64" from '0xD000000000UL' > to '0x3400000000UL' in src/usr.sbin/bhyve/pci_emul.c. Yep, that's a good way to test. > But: > # ./nvidia-smi > No devices were found > dmesg: > [ 173.498953] NVRM: RmInitAdapter failed! (0x53:0x3:1856) > [ 173.499115] NVRM: rm_init_adapter failed for device bearing minor number 0 Looks like you're getting close :) later, Peter. From owner-freebsd-virtualization@freebsd.org Wed Jan 11 10:41:59 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB037CAB3B3 for ; Wed, 11 Jan 2017 10:41:59 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 260821AB1 for ; Wed, 11 Jan 2017 10:41:58 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id v0BAfO2D043391; Wed, 11 Jan 2017 02:41:24 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id v0BAfOoV043390; Wed, 11 Jan 2017 02:41:24 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201701111041.v0BAfOoV043390@pdx.rh.CN85.dnsmgr.net> Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) In-Reply-To: <20170111014544.70670784@mscad14> To: soralx@cydem.org Date: Wed, 11 Jan 2017 02:41:24 -0800 (PST) CC: freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2017 10:41:59 -0000 IIRC the 367.44 version of the nvidia drivers do NOT support the Quadro 2000, you need to be using the 340.xx version of them. I ran into problems on native hardware. Also before you attempt to get VGA passthrough working it is best to make sure you can run native, have you tried running your guest on the host in a native configuration? I have fought this on other platforms many times only to find out that what I was trying would not ever run native, let alone in a virtualized environment. > > The problem appears to be in the area of assigning memory-mapped > > I/O ranges by bhyve for the VGA card to a region outside of the > > CPU's addressable space; i.e., bhyve does not check CPUID's > > 0x80000008 AL value (0x27 for my CPU, which is 39 bits -- while > > bhyve assigns 0xd000000000 & above for the large Prefetch Memory > > chunks, which requires 40 address bits). At least this is my > > understanding of why VGA passthrough does not work. > > To test this, I tried writing to PCI BARs in FreeBSD guest using > `pciconf -w`. Not much use that was: I could read back the values > written to the registers (e.g., `pciconf -r pci0:0:4:0 0x14:48`), > but `pciconf -lvb` still showed the same huge base addresses -- > they did not want to change. > > OK, I had enough of that. So I went to dig in the source, and > changed the "#define PCI_EMUL_MEMBASE64" from '0xD000000000UL' > to '0x3400000000UL' in src/usr.sbin/bhyve/pci_emul.c. Recompiled > bhyve, booted up FreeBSD, and: > # pciconf -lvb > [...] > vgapci0@pci0:0:4:0: class=0x030000 card=0x084a10de chip=0x0dd810de rev=0xa1 hdr=0x00 > vendor = 'NVIDIA Corporation' > device = 'GF106GL [Quadro 2000]' > class = display > subclass = VGA > bar [10] = type Memory, range 32, base 0xc2000000, size 33554432, enabled > bar [14] = type Prefetchable Memory, range 64, base 0x3400000000, size 134217728, enabled > bar [1c] = type Prefetchable Memory, range 64, base 0x3408000000, size 67108864, enabled > bar [24] = type I/O Port, range 32, base 0x2080, size 128, enabled > > ...a-a-and: > # kldload nvidia-modeset > Linux ELF exec handler installed > nvidia0: on vgapci0 > vgapci0: child nvidia0 requested pci_enable_io > vgapci0: attempting to allocate 1 MSI vectors (1 supported) > msi: routing MSI IRQ 269 to local APIC 3 vector 51 > vgapci0: using IRQ 269 for MSI > vgapci0: child nvidia0 requested pci_enable_io > random: harvesting attach, 8 bytes (4 bits) from nvidia0 > # nvidia-smi > acquiring duplicate lock of same type: "os.lock_sx" > 1st os.lock_sx @ nvidia_os.c:599 > 2nd os.lock_sx @ nvidia_os.c:599 > stack backtrace: > #0 0xffffffff80aa6780 at witness_debugger+0x70 > #1 0xffffffff80aa6683 at witness_checkorder+0xde3 > #2 0xffffffff80a4fac2 at _sx_xlock+0x72 > #3 0xffffffff82a515c2 at os_acquire_mutex+0x32 > #4 0xffffffff82a21068 at _nv016673rm+0x18 > Tue Jan 10 17:06:48 2017 > +-----------------------------------------------------------------------------+ > | NVIDIA-SMI 367.44 Driver Version: 367.44 | > |-------------------------------+----------------------+----------------------+ > | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | > | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | > |===============================+======================+======================| > | 0 Quadro 2000 Off | 0000:00:04.0 Off | N/A | > | 30% 35C P8 N/A / N/A | 0MiB / 963MiB | 0% Default | > +-------------------------------+----------------------+----------------------+ > > +-----------------------------------------------------------------------------+ > | Processes: GPU Memory | > | GPU PID Type Process name Usage | > |=============================================================================| > | No running processes found | > +-----------------------------------------------------------------------------+ > > Beauty! It's very slow to execute, though. And Xorg is not in a hurry > to start working: > [ 204.724] (--) PCI:*(0:0:4:0) 10de:0dd8:10de:084a rev 161, Mem @ 0xc2000000/33554432, 0x3400000000/134217728, 0x3408000000/67108864, I/O @ 0x00002080/128, BIOS @ 0x????????/65536 > [...] > [ 204.736] (**) NVIDIA(0): Depth 24, (--) framebuffer bpp 32 > [ 204.736] (==) NVIDIA(0): RGB weight 888 > [ 204.736] (==) NVIDIA(0): Default visual is TrueColor > [ 204.736] (==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0) > [ 204.738] (**) NVIDIA(0): Enabling 2D acceleration > [ 213.674] (--) NVIDIA(0): Valid display device(s) on GPU-0 at PCI:0:4:0 > [ 213.674] (--) NVIDIA(0): CRT-0 > [ 213.674] (--) NVIDIA(0): DFP-0 (boot) > [ 213.674] (--) NVIDIA(0): DFP-1 > [ 213.674] (--) NVIDIA(0): DFP-2 > [ 213.674] (--) NVIDIA(0): DFP-3 > [ 213.675] (--) NVIDIA(0): DFP-4 > [ 213.698] (--) NVIDIA(0): CRT-0: disconnected > [ 213.698] (--) NVIDIA(0): CRT-0: 400.0 MHz maximum pixel clock > [ 213.698] (--) NVIDIA(0): > [ 213.743] (--) NVIDIA(0): DELL 2007FP (DFP-0): connected > [ 213.743] (--) NVIDIA(0): DELL 2007FP (DFP-0): Internal TMDS > [ 213.743] (--) NVIDIA(0): DELL 2007FP (DFP-0): 330.0 MHz maximum pixel clock > [...] > [ 213.747] (II) NVIDIA(0): NVIDIA GPU Quadro 2000 (GF106GL) at PCI:0:4:0 (GPU-0) > [ 213.747] (--) NVIDIA(0): Memory: 1048576 kBytes > [ 213.747] (--) NVIDIA(0): VideoBIOS: 70.06.0d.00.02 > [ 213.747] (II) NVIDIA(0): Detected PCI Express Link width: 16X > [ 213.748] (**) NVIDIA(0): Using HorizSync/VertRefresh ranges from the EDID for display > [ 213.748] (**) NVIDIA(0): device DELL 2007FP (DFP-0) (Using EDID frequencies has > [ 213.748] (**) NVIDIA(0): been enabled on all display devices.) > [...] > [ 213.751] (II) NVIDIA(0): Virtual screen size determined to be 1600 x 1200 > [ 213.761] (--) NVIDIA(0): DPI set to (99, 98); computed from "UseEdidDpi" X config > [ 213.761] (--) NVIDIA(0): option > [ 213.761] (--) Depth 24 pixmap format is 32 bpp > [ 213.767] (II) NVIDIA: Reserving 12288.00 MB of virtual memory for indirect memory > [ 213.767] (II) NVIDIA: access. > [ 216.789] (EE) NVIDIA(GPU-0): Failed to initialize DMA. > [ 216.789] (EE) *** Aborting *** > [ 216.791] (EE) NVIDIA(0): Failed to allocate push buffer > [ 216.839] (EE) > Fatal server error: > [ 216.839] (EE) AddScreen/ScreenInit failed for driver 0 > > Linux still doesn't work (curse Ubuntu! what a mess. It tried to start > Xorg at boot, so I managed to disable that, but no matter what, I > couldn't stop it from trying to run 'nvidia-smi' at boot! And trust me, > I tried a lot. I removed all the scripts related to nvidia, /etc/udev/ > is basically empty [/etc just looks like a pile-up of crap, wow!], yet > /usr/bin/nvidia-smi still tried to run by itself until I moved it away). > > dmesg: > [ 1.390957] nvidia: module verification failed: signature and/or required key missing - tainting kernel > [ 1.394715] nvidia 0000:00:04.0: can't derive routing for PCI INT A > [ 1.395185] nvidia 0000:00:04.0: PCI INT A: no GSI > [ 1.414173] vgaarb: device changed decodes: PCI:0000:00:04.0,olddecodes=io+mem,decodes=none:owns=io+mem > [ 1.417062] nvidia-nvlink: Nvlink Core is being initialized, major device number 247 > [ 1.417609] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 375.26 Thu Dec 8 18:36:43 PST 2016 (using threaded interrupts) > [ 1.419820] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 375.26 Thu Dec 8 18:04:14 PST 2016 > [ 1.422067] [drm] [nvidia-drm] [GPU ID 0x00000004] Loading driver > [...] > [ 3.904893] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 246 > # lspci -vvn > 00:04.0 0300: 10de:0dd8 (rev a1) (prog-if 00 [VGA controller]) > Subsystem: 10de:084a > Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- > Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Latency: 0 > Interrupt: pin A routed to IRQ 16 > Region 0: Memory at c2000000 (32-bit, non-prefetchable) [size=32M] > Region 1: Memory at 3400000000 (64-bit, prefetchable) [size=128M] > Region 3: Memory at 3408000000 (64-bit, prefetchable) [size=64M] > Region 5: I/O ports at 2080 [size=128] > [virtual] Expansion ROM at c0080000 [disabled] [size=512K] > [...] > But: > # ./nvidia-smi > No devices were found > dmesg: > [ 173.498953] NVRM: RmInitAdapter failed! (0x53:0x3:1856) > [ 173.499115] NVRM: rm_init_adapter failed for device bearing minor number 0 > > Not sure what's happening. But I'll try with AMD/ATI card. > > -- > [SorAlx] ridin' VN2000 Classic LT > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Wed Jan 11 13:52:49 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9E963CA982E for ; Wed, 11 Jan 2017 13:52:49 +0000 (UTC) (envelope-from misc-freebsd@talk2dom.com) Received: from mail.shmtech.biz (mail.shmtech.biz [IPv6:2001:41c9:1:37::e]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 43ED71737 for ; Wed, 11 Jan 2017 13:52:48 +0000 (UTC) (envelope-from misc-freebsd@talk2dom.com) Received: from [10.0.1.100] (originc9729.pndsl.co.uk [84.92.225.78]) (authenticated bits=0) by mail.shmtech.biz (8.15.2/8.15.2) with ESMTPSA id v0BDqhBj052238 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO); Wed, 11 Jan 2017 13:52:44 GMT (envelope-from misc-freebsd@talk2dom.com) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=talk2dom.com; s=shmtech4; t=1484142764; x=1484747564; bh=V+ZJAicwvSt6f0NoK72u94LquylUUzp3J3smQbXXWiU=; h=Subject:To:References:Cc:From:Date:In-Reply-To; b=kvkZu5xw0wCXigMjUeW+wXdrfrY5wom8lq3MMqyecWdoXagVHkR4/rHmUmvaWoJ5p x8I8csenpg9Ff5D9gi1tQu5AOJgF8utvQKHiaPMZDZj/YwXtYSXLKTHxoTYM2smrT0 TjVhFO8edyYSLAi+09aJ2BVnBgHkSOgN/jsRCCLo= X-Authentication-Warning: sendmail: Host originc9729.pndsl.co.uk [84.92.225.78] claimed to be [10.0.1.100] Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> Cc: freebsd-virtualization@freebsd.org From: Dom Message-ID: <75abdb83-8902-1c6e-e881-5af24e5bba05@talk2dom.com> Date: Wed, 11 Jan 2017 13:52:43 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: <20170110180117.7f246b5a@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2017 13:52:49 -0000 On 11/01/2017 02:01, soralx@cydem.org wrote: > Dom wote: >> There doesn't seem to be support for CPUID 0x40000001 in bhyve either. > What is it supposed to do? As far as I can tell it's the Hypervisor extension flags list. The lack of these extensions/optimisations might explain why your FreeBSD VM runs slow but their presence also causes the nVidia driver to refuse to run. (Can't remember where I read this, sorry) With your change to PCI_EMUL_MEMBASE64 I can boot a CentOS VM without the "pci=nocrs" kernel option and nVidia card is assigned BARs without issue. However, even with reapplying the changes to vmm.ko to hide/remove the 0x40000000 CPUID support and CPUID2_HV, I still have the same "RmInitAdapter failed" issue. Allegedly[0] nVidia VM checking came in with driver version 337.88, with more checking after version 344.11. I couldn't install version 319 as it failed to build the Linux kernel module. I currently have 370.28 installed which supports both my GT610 and my GTX960. Maybe the next thing for me to try is to replicate your tests with a FreeBSD VM. [0] https://ubuntuforums.org/showthread.php?t=2266916 search for "337.88" From owner-freebsd-virtualization@freebsd.org Wed Jan 11 19:44:04 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2A275CAB791 for ; Wed, 11 Jan 2017 19:44:04 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-lf0-f44.google.com (mail-lf0-f44.google.com [209.85.215.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C66C210B5 for ; Wed, 11 Jan 2017 19:44:03 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-lf0-f44.google.com with SMTP id m78so122619158lfg.2 for ; Wed, 11 Jan 2017 11:44:03 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=aK91yyHRJQiti0zhaBnFKJYBReiVJWeQam1kJWEVUJw=; b=L1iEC+flMBjASfBzMz84wZi2sJIv/8UfzfplKhzDlTZl1kQIsWKeDXNuqbsHtoaW1i 8M1wxQ0yrvDTjQrffH4Mqz+A1OUmMyDPEiBcSsgiYZLCZdfJGrSG0UA2oANdEJhC5bs/ x3BfAyuiZR/lgierGMcd1YKjRei9AsdIl/ViYReSIOqOlTzBl/XTAx8ebFXODoB4avuR 4G6uTMTxWXaxJzEtvcpwqGCoJoo2Sgb5OKbMgz5ftZmXk2DoEzhdmBTUEmLnEV0DStVf OG6nxWrMZIOfV2GDjNShqE4kH8k850svZ26nz8p724gv+r4u2u6FcbpS4MCYr+Fpr+iU 0MXg== X-Gm-Message-State: AIkVDXJws+4ldO2gQWeKRRbS+C4sr4rj+kJ+hsfoO+sCAXQ9XLuSy8Z27/Tw2UBB8x/zkk/2+VVbzUejRQoxlA== X-Received: by 10.46.8.2 with SMTP id 2mr3759985lji.55.1484163431959; Wed, 11 Jan 2017 11:37:11 -0800 (PST) MIME-Version: 1.0 References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <75abdb83-8902-1c6e-e881-5af24e5bba05@talk2dom.com> In-Reply-To: <75abdb83-8902-1c6e-e881-5af24e5bba05@talk2dom.com> From: "K. Macy" Date: Wed, 11 Jan 2017 19:37:01 +0000 Message-ID: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: Dom , soralx@cydem.org Cc: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2017 19:44:04 -0000 Is the VM checking documented in the driver notes somewhere? I have a Titan X that I need to run CUDA on and would be much happier if I didn't have to actually switch back and forth between FreeBSD and Ubuntu on my desktop. Are we new fairly certain that this won't work? (Yet another reason to go with AMD if they ever deliver on ROCm) On Wed, Jan 11, 2017 at 05:53 Dom wrote: > On 11/01/2017 02:01, soralx@cydem.org wrote: > > > Dom wote: > > >> There doesn't seem to be support for CPUID 0x40000001 in bhyve either. > > > What is it supposed to do? > > > > As far as I can tell it's the Hypervisor extension flags list. The lack > > of these extensions/optimisations might explain why your FreeBSD VM runs > > slow but their presence also causes the nVidia driver to refuse to run. > > (Can't remember where I read this, sorry) > > > > With your change to PCI_EMUL_MEMBASE64 I can boot a CentOS VM without > > the "pci=nocrs" kernel option and nVidia card is assigned BARs without > > issue. > > > > However, even with reapplying the changes to vmm.ko to hide/remove the > > 0x40000000 CPUID support and CPUID2_HV, I still have the same > > "RmInitAdapter failed" issue. > > > > Allegedly[0] nVidia VM checking came in with driver version 337.88, with > > more checking after version 344.11. I couldn't install version 319 as it > > failed to build the Linux kernel module. I currently have 370.28 > > installed which supports both my GT610 and my GTX960. > > > > Maybe the next thing for me to try is to replicate your tests with a > > FreeBSD VM. > > > > [0] https://ubuntuforums.org/showthread.php?t=2266916 search for > > "337.88" > > _______________________________________________ > > freebsd-virtualization@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > > From owner-freebsd-virtualization@freebsd.org Wed Jan 11 22:04:23 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56C12CAB99F for ; Wed, 11 Jan 2017 22:04:23 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 1786312ED for ; Wed, 11 Jan 2017 22:04:22 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 6510320195B4 for ; Thu, 12 Jan 2017 08:04:01 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 5E36F280991 for ; Thu, 12 Jan 2017 08:04:01 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 7LHc-E20Be9G for ; Thu, 12 Jan 2017 08:04:01 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 5B8B4280900; Thu, 12 Jan 2017 08:03:58 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: Dom , soralx@cydem.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <75abdb83-8902-1c6e-e881-5af24e5bba05@talk2dom.com> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: Date: Wed, 11 Jan 2017 14:04:14 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <75abdb83-8902-1c6e-e881-5af24e5bba05@talk2dom.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=07d9gI8wAAAA:8 a=3QiXCMbFiH1qCH0bXdgA:9 a=pILNOxqGKmIA:10 a=H71i6ErUBJYA:10 a=e2CUPOnPG4QKp8I52DXD:22 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Jan 2017 22:04:23 -0000 Hi, >>> There doesn't seem to be support for CPUID 0x40000001 in bhyve either. >> What is it supposed to do? > > As far as I can tell it's the Hypervisor extension flags list. The lack > of these extensions/optimisations might explain why your FreeBSD VM runs > slow but their presence also causes the nVidia driver to refuse to run. That leaf is KVM-only. bhyve doesn't have any additional hypervisor leaves beyond 0x4000000 (the spec for this is https://lwn.net/Articles/301888/) later, Peter. From owner-freebsd-virtualization@freebsd.org Thu Jan 12 02:33:37 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3E0ACAB9C6 for ; Thu, 12 Jan 2017 02:33:37 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 8D65C1B6A for ; Thu, 12 Jan 2017 02:33:36 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 4D63AF805; Wed, 11 Jan 2017 18:33:36 -0800 (PST) Date: Wed, 11 Jan 2017 18:33:35 -0800 From: To: Cc: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111183335.50a93727@mscad14> In-Reply-To: <75abdb83-8902-1c6e-e881-5af24e5bba05@talk2dom.com> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <75abdb83-8902-1c6e-e881-5af24e5bba05@talk2dom.com> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 02:33:37 -0000 > As far as I can tell it's the Hypervisor extension flags list. The lack > of these extensions/optimisations might explain why your FreeBSD VM > runs slow The guest isn't slow, actually -- just the `nvidia-smi` tool was much slower than normal to produce output. CPU speed in the guest is less than 2% slower than bare hardware. Memory bandwidth is over 2.5 times slower. [0] Still, compiling world and ports feels quite snappy. Disk I/O is not bad, too! (but latencies are bit high) [1] > However, even with reapplying the changes to vmm.ko to hide/remove the > 0x40000000 CPUID support and CPUID2_HV, I still have the same > "RmInitAdapter failed" issue. Well, I, too, couldn't get nVidia driver to work at all in Linux. The driver loaded, but when trying to use it, gave the "RmInitAdapter failed! (0x53:0x3:1856)" errors. Is is the same in your case? Or the driver does not load at all (like I experienced with Quadro 600)? Maybe I should try the same driver version in Linux as in FreeBSD... > Allegedly[0] nVidia VM checking came in with driver version 337.88, > with more checking after version 344.11. I couldn't install version 319 > as it failed to build the Linux kernel module. I currently have 370.28 > installed which supports both my GT610 and my GTX960. `nvidia-smi -q` thinks it's running on bare hardware [when in VM]: GPU Virtualization Mode Virtualization mode : None Yet we know the driver refuses to load for Quadro 600 and your GTX. (that two-faced bastard nVidia!) So there must be multiple checks. > Maybe the next thing for me to try is to replicate your tests with a > FreeBSD VM. Yes, give it a try. I'm about to give up, for nVidia card at least (I was only using it for testing, as I didn't have AMD GPU handy until recently). [0] # Host with idle FreeBSD VM guest running # ubench Unix Benchmark Utility v.0.3 FreeBSD 10.3-STABLE FreeBSD 10.3-STABLE #0 r311343M: Thu Jan 5 02:31:50 PST 2017 xxx@yyy:/usr/obj/usr/src/sys/SORALX amd64 Ubench CPU: 2076542 Ubench MEM: 1296221 -------------------- Ubench AVG: 1686381 # Guest # ubench Unix Benchmark Utility v.0.3 FreeBSD 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r311659: Tue Jan 10 14:18:44 PST 2017 xxx@zzz:/usr/obj/usr/src/sys/GENERIC amd64 Ubench CPU: 2053983 Ubench MEM: 489953 -------------------- Ubench AVG: 1271968 [1] # Host # diskinfo -tv /dev/ada0 /dev/ada0 512 # sectorsize 250059350016 # mediasize in bytes (233G) 488397168 # mediasize in sectors 4096 # stripesize Seek times: Full stroke: 250 iter in 0.029146 sec = 0.117 msec Half stroke: 250 iter in 0.029950 sec = 0.120 msec Quarter stroke: 500 iter in 0.057272 sec = 0.115 msec Short forward: 400 iter in 0.045711 sec = 0.114 msec Short backward: 400 iter in 0.045748 sec = 0.114 msec Seq outer: 2048 iter in 0.072799 sec = 0.036 msec Seq inner: 2048 iter in 0.072256 sec = 0.035 msec Transfer rates: outside: 102400 kbytes in 0.222412 sec = 460407 kbytes/sec middle: 102400 kbytes in 0.222026 sec = 461207 kbytes/sec inside: 102400 kbytes in 0.222842 sec = 459518 kbytes/sec # Guest # diskinfo -tv /dev/vtbd0 /dev/vtbd0 512 # sectorsize 22548644864 # mediasize in bytes (21G) 44040322 # mediasize in sectors 32768 # stripesize Seek times: Full stroke: 250 iter in 0.112714 sec = 0.451 msec Half stroke: 250 iter in 0.100388 sec = 0.402 msec Quarter stroke: 500 iter in 0.132808 sec = 0.266 msec Short forward: 400 iter in 0.074504 sec = 0.186 msec Short backward: 400 iter in 0.096154 sec = 0.240 msec Seq outer: 2048 iter in 0.208239 sec = 0.102 msec Seq inner: 2048 iter in 0.061853 sec = 0.030 msec Transfer rates: outside: 102400 kbytes in 0.201375 sec = 508504 kbytes/sec middle: 102400 kbytes in 0.196044 sec = 522332 kbytes/sec inside: 102400 kbytes in 0.197697 sec = 517964 kbytes/sec -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Thu Jan 12 02:37:30 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9405CABAE9 for ; Thu, 12 Jan 2017 02:37:30 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id A85BE1C6D for ; Thu, 12 Jan 2017 02:37:30 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 14F24F805; Wed, 11 Jan 2017 18:37:30 -0800 (PST) Date: Wed, 11 Jan 2017 18:37:29 -0800 From: To: Cc: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111183729.6bd82398@mscad14> In-Reply-To: <201701111041.v0BAfOoV043390@pdx.rh.CN85.dnsmgr.net> References: <20170111014544.70670784@mscad14> <201701111041.v0BAfOoV043390@pdx.rh.CN85.dnsmgr.net> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 02:37:30 -0000 > IIRC the 367.44 version of the nvidia drivers do NOT support the > Quadro 2000, you need to be using the 340.xx version of them. I > ran into problems on native hardware. I pulled the Quadro 2000 out of my workstation [and put the 600 in], which is running fine with the latest driver from ports (367.44). > Also before you attempt to get VGA passthrough working it is best > to make sure you can run native, have you tried running your guest > on the host in a native configuration? Yes, I just installed the nVidia driver on the host, and it works fine. > I have fought this on other platforms many times only to find out > that what I was trying would not ever run native, let alone in a > virtualized environment. This gives me the idea to try a different driver version in Linux... -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Thu Jan 12 02:48:12 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49134CABDDD for ; Thu, 12 Jan 2017 02:48:12 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 36DED107B; Thu, 12 Jan 2017 02:48:11 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 951ACF805; Wed, 11 Jan 2017 18:48:11 -0800 (PST) Date: Wed, 11 Jan 2017 18:48:10 -0800 From: To: Cc: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111184810.5e9fb07f@mscad14> In-Reply-To: <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 02:48:12 -0000 > Hi, > > >> The problem appears to be in the area of assigning memory-mapped > >> I/O ranges by bhyve for the VGA card to a region outside of the > >> CPU's addressable space; i.e., bhyve does not check CPUID's > >> 0x80000008 AL value (0x27 for my CPU, which is 39 bits -- while > >> bhyve assigns 0xd000000000 & above for the large Prefetch Memory > >> chunks, which requires 40 address bits). > > That's correct - it's a bug in bhyve. Baking a proper fix will be more complicated by the fact that PCIe cards themselves may have limitations. For example, most nVidia GPUs have 40 bits DMA addressing capability, some 39, an a few (still modern) ones -- just 37 [ref. nVidia "README" in the driver package]. > PCI passthru doesn't allow the BAR values to be modified (this could > be changed, but it's a lot of work for little gain). Removing another signature of detecting virtualization and increasing compatibility would be negligible gain? Just asking... > > But: > > # ./nvidia-smi > > No devices were found > > dmesg: > > [ 173.498953] NVRM: RmInitAdapter failed! (0x53:0x3:1856) > > [ 173.499115] NVRM: rm_init_adapter failed for device bearing > > minor number 0 > > Looks like you're getting close :) Hmm, I'm not seeing myself getting much closer here. Do you know something I don't? ;) I really hope bhyve developers can spare a bit of time on getting GPU passthrough to work... I know nothing about these things, and where I waste half a day messing around, the problem could be fixed in half an hour by someone who knows. > later, > Peter. -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Thu Jan 12 02:54:55 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7453BCA909F for ; Thu, 12 Jan 2017 02:54:55 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 61ADD147A; Thu, 12 Jan 2017 02:54:55 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 040F6F805; Wed, 11 Jan 2017 18:54:55 -0800 (PST) Date: Wed, 11 Jan 2017 18:54:54 -0800 From: To: Cc: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111185454.59c47d1c@mscad14> In-Reply-To: References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <75abdb83-8902-1c6e-e881-5af24e5bba05@talk2dom.com> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 02:54:55 -0000 > Is the VM checking documented in the driver notes somewhere? I have a It's not in their driver's "README" file. > Titan X that I need to run CUDA on and would be much happier if I > didn't have to actually switch back and forth between FreeBSD and > Ubuntu on my desktop. Are we new fairly certain that this won't work? Not certain. The idea that nVidia artificially limits the use of the non-pro cards in VMs in their drivers are only speculations. There is a possibility that certain BIOS and/or hardware features are missing in the gaming cards. > (Yet another reason to go with AMD if they ever deliver on ROCm) Yeah, AMD are pretty good for computing. And they don't seem to limit floating-point performance as severely as nVidia for non-"professional" cards. -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Thu Jan 12 03:09:27 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0E85CA9734 for ; Thu, 12 Jan 2017 03:09:27 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 82F1F1D97 for ; Thu, 12 Jan 2017 03:09:27 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 6A44C20195A1 for ; Thu, 12 Jan 2017 13:09:06 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 6463A2809D3 for ; Thu, 12 Jan 2017 13:09:06 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id levYwLj62SOw for ; Thu, 12 Jan 2017 13:09:06 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 485DB280992; Thu, 12 Jan 2017 13:09:04 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111184810.5e9fb07f@mscad14> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: <839b4e54-fd7a-20fd-630f-f8aaee0c79ee@freebsd.org> Date: Wed, 11 Jan 2017 19:09:21 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170111184810.5e9fb07f@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=8SLKNYVD9Kc_1ussAjsA:9 a=iZbocsCRyau62-kp:21 a=WgxIE6v9D2JIhToZ:21 a=pILNOxqGKmIA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 03:09:28 -0000 Hi, >> That's correct - it's a bug in bhyve. > > Baking a proper fix will be more complicated by the fact that PCIe > cards themselves may have limitations. For example, most nVidia GPUs > have 40 bits DMA addressing capability, some 39, an a few (still > modern) ones -- just 37 [ref. nVidia "README" in the driver package]. That's a different issue - it's unlikely, if not impossible, to configure bhyve with enough RAM to hit 37 bits worth where that would become a problem. No need to worry about that. >> PCI passthru doesn't allow the BAR values to be modified (this could >> be changed, but it's a lot of work for little gain). > > Removing another signature of detecting virtualization and increasing > compatibility would be negligible gain? Just asking... There are lots of BIOS/UEFI implementations out there that have the same restriction. In general, there should be no need for a guest to reprogram device BARs. After changing the 64-bit BAR base address, did you still need the pci=nocrs option for Linux ? I'd hope this would be no longer necessary. >>> But: >>> # ./nvidia-smi >>> No devices were found >>> dmesg: >>> [ 173.498953] NVRM: RmInitAdapter failed! (0x53:0x3:1856) >>> [ 173.499115] NVRM: rm_init_adapter failed for device bearing >>> minor number 0 >> >> Looks like you're getting close :) > > Hmm, I'm not seeing myself getting much closer here. Do you know > something I don't? ;) I really hope bhyve developers can spare a > bit of time on getting GPU passthrough to work... I know nothing > about these things, and where I waste half a day messing around, > the problem could be fixed in half an hour by someone who knows. The problem is the knowledge set of graphics/GPU knowhow and equipment access, and bhyve/PCI programming, are disjoint. The time I've spent on it has been the inverse, where I feel that I've spent a half-day doing things that anyone who knew about graphics could get done in a half-hour :) For these type of issues, joint work is best to leverage the knowledge of both sides. From my point-of-view, the work you've done has been very helpful. later, Peter. From owner-freebsd-virtualization@freebsd.org Thu Jan 12 03:26:23 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DB6B6CA9FED for ; Thu, 12 Jan 2017 03:26:23 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 801051A87; Thu, 12 Jan 2017 03:26:23 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id v0C3Pgch047030; Wed, 11 Jan 2017 19:25:42 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id v0C3PgU7047029; Wed, 11 Jan 2017 19:25:42 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201701120325.v0C3PgU7047029@pdx.rh.CN85.dnsmgr.net> Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) In-Reply-To: <20170111184810.5e9fb07f@mscad14> To: soralx@cydem.org Date: Wed, 11 Jan 2017 19:25:42 -0800 (PST) CC: grehan@freebsd.org, freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 03:26:24 -0000 > > > Hi, > > > > >> The problem appears to be in the area of assigning memory-mapped > > >> I/O ranges by bhyve for the VGA card to a region outside of the > > >> CPU's addressable space; i.e., bhyve does not check CPUID's > > >> 0x80000008 AL value (0x27 for my CPU, which is 39 bits -- while > > >> bhyve assigns 0xd000000000 & above for the large Prefetch Memory > > >> chunks, which requires 40 address bits). > > > > That's correct - it's a bug in bhyve. > > Baking a proper fix will be more complicated by the fact that PCIe > cards themselves may have limitations. For example, most nVidia GPUs > have 40 bits DMA addressing capability, some 39, an a few (still > modern) ones -- just 37 [ref. nVidia "README" in the driver package]. > > > PCI passthru doesn't allow the BAR values to be modified (this could > > be changed, but it's a lot of work for little gain). > > Removing another signature of detecting virtualization and increasing > compatibility would be negligible gain? Just asking... I don't think we are going to try and defeat the NVidia virtualization checks, and I can probably assure you that they would patch them as fast as we bypassed them. > > > But: > > > # ./nvidia-smi > > > No devices were found > > > dmesg: > > > [ 173.498953] NVRM: RmInitAdapter failed! (0x53:0x3:1856) > > > [ 173.499115] NVRM: rm_init_adapter failed for device bearing > > > minor number 0 > > > > Looks like you're getting close :) > > Hmm, I'm not seeing myself getting much closer here. Do you know > something I don't? ;) I really hope bhyve developers can spare a > bit of time on getting GPU passthrough to work... I know nothing > about these things, and where I waste half a day messing around, > the problem could be fixed in half an hour by someone who knows. This is officially an item on my plate. Functional hardware that works doing VGA passthrough under ESXi was brought up about 2 weeks ago so I am past that stage. I have Quadro FX3800 on a Supermicro X9DAi. You have helped some in uncovering the next set of issues, but my plate is very large, and seems to have grown appendages that are holding all sorts of things :-) I am still coming up to speed on the bhyve code, so it wont be a half hour fix by me, but it well get fixed. > > later, > > Peter. > > -- > [SorAlx] ridin' VN2000 Classic LT > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Thu Jan 12 03:54:04 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5AA5ACAB96F for ; Thu, 12 Jan 2017 03:54:04 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 4308E1AC6 for ; Thu, 12 Jan 2017 03:54:03 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 403DEF805 for ; Wed, 11 Jan 2017 19:54:03 -0800 (PST) Date: Wed, 11 Jan 2017 19:54:02 -0800 From: To: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111195402.785f27c6@mscad14> In-Reply-To: <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 03:54:04 -0000 I had a bit more play with nVidia and FreeBSD guest. First, `nvidia-smi -q` output diff [0] is interesting. It suggests that the card may be in some incompletely initialized state: notice the "Unknown Error" instead of real UUID, and the P8 power state. Could it be that the driver doesn't put the card's BIOS in the right state? The command was run in both host and guest without Xorg loaded. Second, I was able to start Xorg by disabling rendering acceleration (Option "NoAccel"). Now nVidia's Xorg module does not fail to allocate DMA (I guess it does not try?), but oddly, reserves 48 GB (!?) of virtual memory instead. Sadly there is still no display for some reason. Relevant dmesg bits are below [1]. Of particular interest is the line "nvidia-modeset: Allocated GPU:0 () @ PCI:0000:00:00.0" -- the PCI address is obviously incorrect. Xorg log bits [2] show that X is up. But the monitor stays in sleep mode. With more options [3], I get this: [4]. Edit: actually, host reboot made it behave the same as just with "NoAccel", maybe. Clearly the driver is able to talk to the card: e.g., it attaches and responds to `nvidia-smi` [with the exception of UUID], reads EDID from the monitor. But some channel of communication is clearly missing or not working right. Any ideas how to go about finding out which one? [0] ==============NVSMI LOG============== -Timestamp : Wed Jan 11 19:40:54 2017 +Timestamp : Wed Jan 11 11:08:40 2017 Driver Version : 367.44 Attached GPUs : 1 -GPU 0000:01:00.0 +GPU 0000:00:04.0 Product Name : Quadro 2000 Product Brand : Quadro Display Mode : Enabled @@ -17,11 +17,11 @@ Current : N/A Pending : N/A Serial Number : N/A - GPU UUID : GPU-f6c71b8e-f6c8-5a42-260d-1164720bf4f2 + GPU UUID : Unknown Error Minor Number : 0 VBIOS Version : 70.06.0D.00.02 MultiGPU Board : No - Board ID : 0x100 + Board ID : 0x4 GPU Part Number : N/A Inforom Version Image Version : N/A @@ -34,16 +34,16 @@ GPU Virtualization Mode Virtualization mode : None PCI - Bus : 0x01 - Device : 0x00 + Bus : 0x00 + Device : 0x04 Domain : 0x0000 Device Id : 0x0DD810DE - Bus Id : 0000:01:00.0 + Bus Id : 0000:00:04.0 Sub System Id : 0x084A10DE GPU Link Info PCIe Generation Max : 2 - Current : 2 + Current : 1 Link Width Max : 16x Current : 16x @@ -54,7 +54,7 @@ Tx Throughput : N/A Rx Throughput : N/A Fan Speed : 30 % - Performance State : P0 + Performance State : P8 Clocks Throttle Reasons : N/A FB Memory Usage Total : 963 MiB @@ -113,7 +113,7 @@ Double Bit ECC : N/A Pending : N/A Temperature - GPU Current Temp : 38 C + GPU Current Temp : 35 C GPU Shutdown Temp : N/A GPU Slowdown Temp : N/A Power Readings @@ -125,10 +125,10 @@ Min Power Limit : N/A Max Power Limit : N/A Clocks - Graphics : 625 MHz - SM : 1251 MHz - Memory : 1304 MHz - Video : 540 MHz + Graphics : 405 MHz + SM : 810 MHz + Memory : 324 MHz + Video : 405 MHz Applications Clocks Graphics : N/A Memory : N/A [1] nvidia0: on vgapci0 vgapci0: child nvidia0 requested pci_enable_io vgapci0: attempting to allocate 1 MSI vectors (1 supported) msi: routing MSI IRQ 269 to local APIC 3 vector 51 vgapci0: using IRQ 269 for MSI vgapci0: child nvidia0 requested pci_enable_io nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 367.44 Wed Aug 17 22:05:09 PDT 2016 acquiring duplicate lock of same type: "os.lock_sx" 1st os.lock_sx @ nvidia_os.c:599 2nd os.lock_sx @ nvidia_os.c:599 stack backtrace: #0 0xffffffff80aa6780 at witness_debugger+0x70 #1 0xffffffff80aa6683 at witness_checkorder+0xde3 #2 0xffffffff80a4fac2 at _sx_xlock+0x72 #3 0xffffffff82a515c2 at os_acquire_mutex+0x32 #4 0xffffffff82a21068 at _nv016673rm+0x18 nvidia-modeset: Allocated GPU:0 () @ PCI:0000:00:00.0 nvidia-modeset: WARNING: GPU:0: Lost display notification; continuing. NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000000 NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000001 NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000002 NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000003 NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000004 NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000005 NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000006 NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000007 When rebooting, I get this: nvidia-modeset: ERROR: GPU:0: Idling display engine timed out: 0x0000857d:0:0:0x00000040 [2] [ 659.406] (--) PCI:*(0:0:4:0) 10de:0dd8:10de:084a rev 161, Mem @ 0xc2000000/33554432, 0x3400000000/134217728, 0x3408000000/67108864, I/O @ 0x00002080/128, BIOS @ 0x????????/65536 [ 659.407] (II) LoadModule: "glx" [ 659.412] (II) Loading /usr/local/lib/xorg/modules/extensions/libglx.so [ 659.594] (II) Module glx: vendor="NVIDIA Corporation" [ 659.594] compiled for 4.0.2, module version = 1.0.0 [ 659.594] Module class: X.Org Server Extension [ 659.594] (II) NVIDIA GLX Module 367.44 Wed Aug 17 22:01:17 PDT 2016 [ 659.595] (II) LoadModule: "nvidia" [ 659.595] (II) Loading /usr/local/lib/xorg/modules/drivers/nvidia_drv.so [ 659.603] (II) Module nvidia: vendor="NVIDIA Corporation" [ 659.603] compiled for 4.0.2, module version = 1.0.0 [ 659.603] Module class: X.Org Video Driver [ 659.604] (II) NVIDIA dlloader X Driver 367.44 Wed Aug 17 21:41:06 PDT 2016 [ 659.604] (II) NVIDIA Unified Driver for all Supported NVIDIA GPUs [ 659.604] (--) Using syscons driver with X support (version 2.0) [ 659.604] (--) using VT number 9 [...] [ 659.609] (WW) VGA arbiter: cannot open kernel arbiter, no multi-card support [ 659.609] (**) NVIDIA(0): Depth 24, (--) framebuffer bpp 32 [ 659.609] (==) NVIDIA(0): RGB weight 888 [ 659.609] (==) NVIDIA(0): Default visual is TrueColor [ 659.609] (==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0) [ 659.610] (**) NVIDIA(0): Option "NoPowerConnectorCheck" "yes" [ 659.610] (**) NVIDIA(0): Option "ThermalConfigurationCheck" "no" [ 659.610] (**) NVIDIA(0): Option "Accel" "no" [ 659.610] (**) NVIDIA(0): Disabling 2D acceleration [ 668.054] (--) NVIDIA(0): Valid display device(s) on GPU-0 at PCI:0:4:0 [ 668.054] (--) NVIDIA(0): CRT-0 [ 668.054] (--) NVIDIA(0): DFP-0 (boot) [...] [ 668.125] (II) NVIDIA(GPU-0): Skipping Power Connector Check. [ 668.125] (II) NVIDIA(GPU-0): Skipping Thermal Configuration Check. [ 668.125] (II) NVIDIA(GPU-0): Acceleration disabled. [ 668.125] (II) NVIDIA(0): NVIDIA GPU Quadro 2000 (GF106GL) at PCI:0:4:0 (GPU-0) [ 668.125] (--) NVIDIA(0): Memory: 1048576 kBytes [ 668.125] (--) NVIDIA(0): VideoBIOS: 70.06.0d.00.02 [ 668.125] (II) NVIDIA(0): Detected PCI Express Link width: 16X [ 668.125] (**) NVIDIA(0): Using HorizSync/VertRefresh ranges from the EDID for display [ 668.125] (**) NVIDIA(0): device DELL 2007FP (DFP-0) (Using EDID frequencies has [ 668.125] (**) NVIDIA(0): been enabled on all display devices.) [ 668.127] (==) NVIDIA(0): [ 668.127] (==) NVIDIA(0): No modes were requested; the default mode "nvidia-auto-select" [ 668.127] (==) NVIDIA(0): will be used as the requested mode. [ 668.127] (==) NVIDIA(0): [ 668.128] (II) NVIDIA(0): Validated MetaModes: [ 668.128] (II) NVIDIA(0): "DFP-0:nvidia-auto-select" [ 668.128] (II) NVIDIA(0): Virtual screen size determined to be 1600 x 1200 [ 668.139] (--) NVIDIA(0): DPI set to (99, 98); computed from "UseEdidDpi" X config [ 668.139] (--) NVIDIA(0): option [ 668.139] (--) Depth 24 pixmap format is 32 bpp [ 668.144] (II) NVIDIA: Reserving 49152.00 MB of virtual memory for indirect memory [ 668.144] (II) NVIDIA: access. [ 668.370] (II) NVIDIA(0): Setting mode "DFP-0:nvidia-auto-select" [ 675.574] (==) NVIDIA(0): Disabling shared memory pixmaps [ 675.574] (==) NVIDIA(0): Backing store enabled [ 675.574] (==) NVIDIA(0): Silken mouse enabled [ 675.575] (**) NVIDIA(0): DPMS enabled [ 675.575] (II) Loading sub module "dri2" [ 675.575] (II) LoadModule: "dri2" [ 675.575] (II) Module "dri2" already built-in [ 675.575] (II) NVIDIA(0): [DRI2] Setup complete [ 675.575] (II) NVIDIA(0): [DRI2] VDPAU driver: nvidia [ 675.576] (--) RandR disabled [ 675.577] (EE) Failed to initialize GLX extension (Compatible NVIDIA X driver not found) [ 675.577] (II) Loading sub module "shadow" [ 675.577] (II) LoadModule: "shadow" [ 675.577] (II) Loading /usr/local/lib/xorg/modules/libshadow.so [ 675.577] (II) Module shadow: vendor="X.Org Foundation" [ 675.577] compiled for 1.17.4, module version = 1.1.0 [ 675.577] ABI class: X.Org ANSI C Emulation, version 0.4 [ 675.638] (II) config/devd: probing input devices... [ 675.638] (II) config/devd: adding input device (null) (/dev/kbdmux) [ 675.638] (II) LoadModule: "kbd" [ 675.639] (II) Loading /usr/local/lib/xorg/modules/input/kbd_drv.so [ 675.640] (II) Module kbd: vendor="X.Org Foundation" [ 675.640] compiled for 1.17.4, module version = 1.8.1 [ 675.640] Module class: X.Org XInput Driver [ 675.640] ABI class: X.Org XInput driver, version 21.0 [ 675.640] (II) Using input driver 'kbd' for 'kbdmux' [ 675.640] (**) kbdmux: always reports core events [ 675.640] (**) kbdmux: always reports core events [ 675.640] (**) Option "Protocol" "standard" [ 675.640] (**) Option "XkbRules" "base" [ 675.640] (**) Option "XkbModel" "pc105" [ 675.640] (**) Option "XkbLayout" "us" [ 675.640] (**) Option "config_info" "devd:kbdmux" [ 675.640] (II) XINPUT: Adding extended input device "kbdmux" (type: KEYBOARD, id 6) [ 675.640] (II) config/devd: kbdmux is enabled, ignoring device atkbd0 [ 675.640] (II) config/devd: adding input device (null) (/dev/sysmouse) [ 675.640] (II) LoadModule: "mouse" [ 675.640] (II) Loading /usr/local/lib/xorg/modules/input/mouse_drv.so [ 675.642] (II) Module mouse: vendor="X.Org Foundation" [ 675.642] compiled for 1.17.4, module version = 1.9.1 [ 675.642] Module class: X.Org XInput Driver [ 675.642] ABI class: X.Org XInput driver, version 21.0 [ 675.642] (II) Using input driver 'mouse' for 'sysmouse' [ 675.642] (**) sysmouse: always reports core events [ 675.642] (**) Option "Device" "/dev/sysmouse" [ 675.642] (==) sysmouse: Protocol: "Auto" [ 675.642] (**) sysmouse: always reports core events [ 675.642] (==) sysmouse: Emulate3Buttons, Emulate3Timeout: 50 [ 675.642] (**) sysmouse: ZAxisMapping: buttons 4 and 5 [ 675.642] (**) sysmouse: Buttons: 5 [ 675.642] (**) Option "config_info" "devd:sysmouse" [ 675.642] (II) XINPUT: Adding extended input device "sysmouse" (type: MOUSE, id 7) [ 675.642] (**) sysmouse: (accel) keeping acceleration scheme 1 [ 675.642] (**) sysmouse: (accel) acceleration profile 0 [ 675.642] (**) sysmouse: (accel) acceleration factor: 2.000 [ 675.642] (**) sysmouse: (accel) acceleration threshold: 4 [ 675.642] (II) sysmouse: SetupAuto: hw.iftype is 4, hw.model is 0 [ 675.642] (II) sysmouse: SetupAuto: protocol is SysMouse [3] Option "Accel" "no" Option "RenderAccel" "no" Option "UBB" "no" Option "NoFlip" "yes" Option "Overlay" "no" Option "MultisampleCompatibility" "off" Option "NoPowerConnectorCheck" "yes" Option "ThermalConfigurationCheck" "no" Option "TripleBuffer" "off" Option "ModeDebug" "yes" Option "IndirectMemoryAccess" "no" Option "UseSysmemPixmapAccel" "no" #Option "UseDPLib" "off" [4] Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Display engine push buffer channel allocation failed Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Failed to allocate display engine core DMA push buffer Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x20 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff8243997b stack pointer = 0x28:0xfffffe007aec8ab8 frame pointer = 0x28:0xfffff80006a58808 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 3 current process = 631 (Xorg) [ thread pid 631 tid 100062 ] Stopped at _nv002035kms+0x3b: movq 0x20(%rax,%rdx,8),%rax db> -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Thu Jan 12 04:09:03 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81B7CCABFF0 for ; Thu, 12 Jan 2017 04:09:03 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-lf0-f47.google.com (mail-lf0-f47.google.com [209.85.215.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D17B41289 for ; Thu, 12 Jan 2017 04:09:02 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-lf0-f47.google.com with SMTP id v186so4977306lfa.1 for ; Wed, 11 Jan 2017 20:09:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=eiQbOFaxnPpzxqg+EonLCVdt0mpbDI2epjgWMeW5eZc=; b=YnjCSF4J69mdwwcc460T/uKBscJ0DD9Im0lgxcf4l63wj1BTeUnftU1Lx2UMi22LYv ioohhhdsgK1DoDynQCJyrcrzC6I9IXeo2DXGhc+pTCZ+MIciJoL59/+raS7+KdKoB3Qj IOFiKM4FvAV8uYj5bCAaZ7V/VBy2rdcOT75hr8d74hNKXyxVYwsS54BzeImpbyrjy1Wx ueD4AQ62qwwzArVnTpHPREse9TiEFwlGHC2UqN4JWg4939lt3k9WE1ce7cF1wMqP83wf 2kuJOtUzuGW3NbPcunZSHvh/Dn5Nmcgnbi9xL7JJZThC7Tw/e1HYEnylfwQWnrTn125L r7WQ== X-Gm-Message-State: AIkVDXKn48h/rOKC7sLC8dkrxqfAU47R6Oir3dIp/DrovG5ZotJo6XzEL/S9s4CjtW4sqO5BTmkLAMUDdXPEmw== X-Received: by 10.25.210.213 with SMTP id j204mr4560311lfg.65.1484193786574; Wed, 11 Jan 2017 20:03:06 -0800 (PST) MIME-Version: 1.0 References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111195402.785f27c6@mscad14> In-Reply-To: <20170111195402.785f27c6@mscad14> From: "K. Macy" Date: Thu, 12 Jan 2017 04:02:55 +0000 Message-ID: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: freebsd-virtualization@freebsd.org, soralx@cydem.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 04:09:03 -0000 I hope you keep it up or at least figure out what the driver is doing. If they haven't explicitly put in the license terms that virtualization is forbidden for consumer cards, there's nothing wrong with hot patching the driver ... assuming that they don't do things like Skype does where it repeatedly checksums the memory image. Good hunting. -M On Wed, Jan 11, 2017 at 19:54 wrote: > > > I had a bit more play with nVidia and FreeBSD guest. > > > > First, `nvidia-smi -q` output diff [0] is interesting. It suggests that > > the card may be in some incompletely initialized state: notice the > > "Unknown Error" instead of real UUID, and the P8 power state. Could it > > be that the driver doesn't put the card's BIOS in the right state? > > The command was run in both host and guest without Xorg loaded. > > > > Second, I was able to start Xorg by disabling rendering acceleration > > (Option "NoAccel"). Now nVidia's Xorg module does not fail to allocate > > DMA (I guess it does not try?), but oddly, reserves 48 GB (!?) of virtual > > memory instead. Sadly there is still no display for some reason. > > > > Relevant dmesg bits are below [1]. Of particular interest is the line > > "nvidia-modeset: Allocated GPU:0 () @ PCI:0000:00:00.0" -- the PCI > > address is obviously incorrect. > > > > Xorg log bits [2] show that X is up. But the monitor stays in sleep mode. > > With more options [3], I get this: [4]. Edit: actually, host reboot > > made it behave the same as just with "NoAccel", maybe. > > > > Clearly the driver is able to talk to the card: e.g., it attaches and > > responds to `nvidia-smi` [with the exception of UUID], reads EDID from > > the monitor. But some channel of communication is clearly missing or > > not working right. Any ideas how to go about finding out which one? > > > > [0] > > ==============NVSMI LOG============== > > > > -Timestamp : Wed Jan 11 19:40:54 2017 > > +Timestamp : Wed Jan 11 11:08:40 2017 > > Driver Version : 367.44 > > > > Attached GPUs : 1 > > -GPU 0000:01:00.0 > > +GPU 0000:00:04.0 > > Product Name : Quadro 2000 > > Product Brand : Quadro > > Display Mode : Enabled > > @@ -17,11 +17,11 @@ > > Current : N/A > > Pending : N/A > > Serial Number : N/A > > - GPU UUID : > GPU-f6c71b8e-f6c8-5a42-260d-1164720bf4f2 > > + GPU UUID : Unknown Error > > Minor Number : 0 > > VBIOS Version : 70.06.0D.00.02 > > MultiGPU Board : No > > - Board ID : 0x100 > > + Board ID : 0x4 > > GPU Part Number : N/A > > Inforom Version > > Image Version : N/A > > @@ -34,16 +34,16 @@ > > GPU Virtualization Mode > > Virtualization mode : None > > PCI > > - Bus : 0x01 > > - Device : 0x00 > > + Bus : 0x00 > > + Device : 0x04 > > Domain : 0x0000 > > Device Id : 0x0DD810DE > > - Bus Id : 0000:01:00.0 > > + Bus Id : 0000:00:04.0 > > Sub System Id : 0x084A10DE > > GPU Link Info > > PCIe Generation > > Max : 2 > > - Current : 2 > > + Current : 1 > > Link Width > > Max : 16x > > Current : 16x > > @@ -54,7 +54,7 @@ > > Tx Throughput : N/A > > Rx Throughput : N/A > > Fan Speed : 30 % > > - Performance State : P0 > > + Performance State : P8 > > Clocks Throttle Reasons : N/A > > FB Memory Usage > > Total : 963 MiB > > @@ -113,7 +113,7 @@ > > Double Bit ECC : N/A > > Pending : N/A > > Temperature > > - GPU Current Temp : 38 C > > + GPU Current Temp : 35 C > > GPU Shutdown Temp : N/A > > GPU Slowdown Temp : N/A > > Power Readings > > @@ -125,10 +125,10 @@ > > Min Power Limit : N/A > > Max Power Limit : N/A > > Clocks > > - Graphics : 625 MHz > > - SM : 1251 MHz > > - Memory : 1304 MHz > > - Video : 540 MHz > > + Graphics : 405 MHz > > + SM : 810 MHz > > + Memory : 324 MHz > > + Video : 405 MHz > > Applications Clocks > > Graphics : N/A > > Memory : N/A > > > > > > [1] > > nvidia0: on vgapci0 > > vgapci0: child nvidia0 requested pci_enable_io > > vgapci0: attempting to allocate 1 MSI vectors (1 supported) > > msi: routing MSI IRQ 269 to local APIC 3 vector 51 > > vgapci0: using IRQ 269 for MSI > > vgapci0: child nvidia0 requested pci_enable_io > > nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX > platforms 367.44 Wed Aug 17 22:05:09 PDT 2016 > > acquiring duplicate lock of same type: "os.lock_sx" > > 1st os.lock_sx @ nvidia_os.c:599 > > 2nd os.lock_sx @ nvidia_os.c:599 > > stack backtrace: > > #0 0xffffffff80aa6780 at witness_debugger+0x70 > > #1 0xffffffff80aa6683 at witness_checkorder+0xde3 > > #2 0xffffffff80a4fac2 at _sx_xlock+0x72 > > #3 0xffffffff82a515c2 at os_acquire_mutex+0x32 > > #4 0xffffffff82a21068 at _nv016673rm+0x18 > > nvidia-modeset: Allocated GPU:0 () @ PCI:0000:00:00.0 > > nvidia-modeset: WARNING: GPU:0: Lost display notification; continuing. > > NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000000 > > NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000001 > > NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000002 > > NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000003 > > NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000004 > > NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000005 > > NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000006 > > NVRM: Xid (PCI:0000:00:04): 16, Head 00000000 Count 00000007 > > > > When rebooting, I get this: > > nvidia-modeset: ERROR: GPU:0: Idling display engine timed out: > 0x0000857d:0:0:0x00000040 > > > > [2] > > [ 659.406] (--) PCI:*(0:0:4:0) 10de:0dd8:10de:084a rev 161, Mem @ > 0xc2000000/33554432, 0x3400000000/134217728, 0x3408000000/67108864, I/O @ > 0x00002080/128, BIOS @ 0x????????/65536 > > [ 659.407] (II) LoadModule: "glx" > > [ 659.412] (II) Loading /usr/local/lib/xorg/modules/extensions/libglx.so > > [ 659.594] (II) Module glx: vendor="NVIDIA Corporation" > > [ 659.594] compiled for 4.0.2, module version = 1.0.0 > > [ 659.594] Module class: X.Org Server Extension > > [ 659.594] (II) NVIDIA GLX Module 367.44 Wed Aug 17 22:01:17 PDT 2016 > > [ 659.595] (II) LoadModule: "nvidia" > > [ 659.595] (II) Loading /usr/local/lib/xorg/modules/drivers/nvidia_drv.so > > [ 659.603] (II) Module nvidia: vendor="NVIDIA Corporation" > > [ 659.603] compiled for 4.0.2, module version = 1.0.0 > > [ 659.603] Module class: X.Org Video Driver > > [ 659.604] (II) NVIDIA dlloader X Driver 367.44 Wed Aug 17 21:41:06 > PDT 2016 > > [ 659.604] (II) NVIDIA Unified Driver for all Supported NVIDIA GPUs > > [ 659.604] (--) Using syscons driver with X support (version 2.0) > > [ 659.604] (--) using VT number 9 > > [...] > > [ 659.609] (WW) VGA arbiter: cannot open kernel arbiter, no multi-card > support > > [ 659.609] (**) NVIDIA(0): Depth 24, (--) framebuffer bpp 32 > > [ 659.609] (==) NVIDIA(0): RGB weight 888 > > [ 659.609] (==) NVIDIA(0): Default visual is TrueColor > > [ 659.609] (==) NVIDIA(0): Using gamma correction (1.0, 1.0, 1.0) > > [ 659.610] (**) NVIDIA(0): Option "NoPowerConnectorCheck" "yes" > > [ 659.610] (**) NVIDIA(0): Option "ThermalConfigurationCheck" "no" > > [ 659.610] (**) NVIDIA(0): Option "Accel" "no" > > [ 659.610] (**) NVIDIA(0): Disabling 2D acceleration > > [ 668.054] (--) NVIDIA(0): Valid display device(s) on GPU-0 at PCI:0:4:0 > > [ 668.054] (--) NVIDIA(0): CRT-0 > > [ 668.054] (--) NVIDIA(0): DFP-0 (boot) > > [...] > > [ 668.125] (II) NVIDIA(GPU-0): Skipping Power Connector Check. > > [ 668.125] (II) NVIDIA(GPU-0): Skipping Thermal Configuration Check. > > [ 668.125] (II) NVIDIA(GPU-0): Acceleration disabled. > > [ 668.125] (II) NVIDIA(0): NVIDIA GPU Quadro 2000 (GF106GL) at PCI:0:4:0 > (GPU-0) > > [ 668.125] (--) NVIDIA(0): Memory: 1048576 kBytes > > [ 668.125] (--) NVIDIA(0): VideoBIOS: 70.06.0d.00.02 > > [ 668.125] (II) NVIDIA(0): Detected PCI Express Link width: 16X > > [ 668.125] (**) NVIDIA(0): Using HorizSync/VertRefresh ranges from the > EDID for display > > [ 668.125] (**) NVIDIA(0): device DELL 2007FP (DFP-0) (Using EDID > frequencies has > > [ 668.125] (**) NVIDIA(0): been enabled on all display devices.) > > [ 668.127] (==) NVIDIA(0): > > [ 668.127] (==) NVIDIA(0): No modes were requested; the default mode > "nvidia-auto-select" > > [ 668.127] (==) NVIDIA(0): will be used as the requested mode. > > [ 668.127] (==) NVIDIA(0): > > [ 668.128] (II) NVIDIA(0): Validated MetaModes: > > [ 668.128] (II) NVIDIA(0): "DFP-0:nvidia-auto-select" > > [ 668.128] (II) NVIDIA(0): Virtual screen size determined to be 1600 x > 1200 > > [ 668.139] (--) NVIDIA(0): DPI set to (99, 98); computed from > "UseEdidDpi" X config > > [ 668.139] (--) NVIDIA(0): option > > [ 668.139] (--) Depth 24 pixmap format is 32 bpp > > [ 668.144] (II) NVIDIA: Reserving 49152.00 MB of virtual memory for > indirect memory > > [ 668.144] (II) NVIDIA: access. > > [ 668.370] (II) NVIDIA(0): Setting mode "DFP-0:nvidia-auto-select" > > [ 675.574] (==) NVIDIA(0): Disabling shared memory pixmaps > > [ 675.574] (==) NVIDIA(0): Backing store enabled > > [ 675.574] (==) NVIDIA(0): Silken mouse enabled > > [ 675.575] (**) NVIDIA(0): DPMS enabled > > [ 675.575] (II) Loading sub module "dri2" > > [ 675.575] (II) LoadModule: "dri2" > > [ 675.575] (II) Module "dri2" already built-in > > [ 675.575] (II) NVIDIA(0): [DRI2] Setup complete > > [ 675.575] (II) NVIDIA(0): [DRI2] VDPAU driver: nvidia > > [ 675.576] (--) RandR disabled > > [ 675.577] (EE) Failed to initialize GLX extension (Compatible NVIDIA X > driver not found) > > [ 675.577] (II) Loading sub module "shadow" > > [ 675.577] (II) LoadModule: "shadow" > > [ 675.577] (II) Loading /usr/local/lib/xorg/modules/libshadow.so > > [ 675.577] (II) Module shadow: vendor="X.Org Foundation" > > [ 675.577] compiled for 1.17.4, module version = 1.1.0 > > [ 675.577] ABI class: X.Org ANSI C Emulation, version 0.4 > > [ 675.638] (II) config/devd: probing input devices... > > [ 675.638] (II) config/devd: adding input device (null) (/dev/kbdmux) > > [ 675.638] (II) LoadModule: "kbd" > > [ 675.639] (II) Loading /usr/local/lib/xorg/modules/input/kbd_drv.so > > [ 675.640] (II) Module kbd: vendor="X.Org Foundation" > > [ 675.640] compiled for 1.17.4, module version = 1.8.1 > > [ 675.640] Module class: X.Org XInput Driver > > [ 675.640] ABI class: X.Org XInput driver, version 21.0 > > [ 675.640] (II) Using input driver 'kbd' for 'kbdmux' > > [ 675.640] (**) kbdmux: always reports core events > > [ 675.640] (**) kbdmux: always reports core events > > [ 675.640] (**) Option "Protocol" "standard" > > [ 675.640] (**) Option "XkbRules" "base" > > [ 675.640] (**) Option "XkbModel" "pc105" > > [ 675.640] (**) Option "XkbLayout" "us" > > [ 675.640] (**) Option "config_info" "devd:kbdmux" > > [ 675.640] (II) XINPUT: Adding extended input device "kbdmux" (type: > KEYBOARD, id 6) > > [ 675.640] (II) config/devd: kbdmux is enabled, ignoring device atkbd0 > > [ 675.640] (II) config/devd: adding input device (null) (/dev/sysmouse) > > [ 675.640] (II) LoadModule: "mouse" > > [ 675.640] (II) Loading /usr/local/lib/xorg/modules/input/mouse_drv.so > > [ 675.642] (II) Module mouse: vendor="X.Org Foundation" > > [ 675.642] compiled for 1.17.4, module version = 1.9.1 > > [ 675.642] Module class: X.Org XInput Driver > > [ 675.642] ABI class: X.Org XInput driver, version 21.0 > > [ 675.642] (II) Using input driver 'mouse' for 'sysmouse' > > [ 675.642] (**) sysmouse: always reports core events > > [ 675.642] (**) Option "Device" "/dev/sysmouse" > > [ 675.642] (==) sysmouse: Protocol: "Auto" > > [ 675.642] (**) sysmouse: always reports core events > > [ 675.642] (==) sysmouse: Emulate3Buttons, Emulate3Timeout: 50 > > [ 675.642] (**) sysmouse: ZAxisMapping: buttons 4 and 5 > > [ 675.642] (**) sysmouse: Buttons: 5 > > [ 675.642] (**) Option "config_info" "devd:sysmouse" > > [ 675.642] (II) XINPUT: Adding extended input device "sysmouse" (type: > MOUSE, id 7) > > [ 675.642] (**) sysmouse: (accel) keeping acceleration scheme 1 > > [ 675.642] (**) sysmouse: (accel) acceleration profile 0 > > [ 675.642] (**) sysmouse: (accel) acceleration factor: 2.000 > > [ 675.642] (**) sysmouse: (accel) acceleration threshold: 4 > > [ 675.642] (II) sysmouse: SetupAuto: hw.iftype is 4, hw.model is 0 > > [ 675.642] (II) sysmouse: SetupAuto: protocol is SysMouse > > > > > > [3] > > Option "Accel" "no" > > Option "RenderAccel" "no" > > Option "UBB" "no" > > Option "NoFlip" "yes" > > Option "Overlay" "no" > > Option "MultisampleCompatibility" "off" > > Option "NoPowerConnectorCheck" "yes" > > Option "ThermalConfigurationCheck" "no" > > Option "TripleBuffer" "off" > > Option "ModeDebug" "yes" > > Option "IndirectMemoryAccess" "no" > > Option "UseSysmemPixmapAccel" "no" > > #Option "UseDPLib" "off" > > > > > > [4] > > Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Display > engine push buffer channel allocation failed > > Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Failed to > allocate display engine core DMA push buffer > > > > Fatal trap 12: page fault while in kernel mode > > cpuid = 2; apic id = 02 > > fault virtual address = 0x20 > > fault code = supervisor read data, page not present > > instruction pointer = 0x20:0xffffffff8243997b > > stack pointer = 0x28:0xfffffe007aec8ab8 > > frame pointer = 0x28:0xfffff80006a58808 > > code segment = base 0x0, limit 0xfffff, type 0x1b > > = DPL 0, pres 1, long 1, def32 0, gran 1 > > processor eflags = interrupt enabled, resume, IOPL = 3 > > current process = 631 (Xorg) > > [ thread pid 631 tid 100062 ] > > Stopped at _nv002035kms+0x3b: movq 0x20(%rax,%rdx,8),%rax > > db> > > > > > > -- > > [SorAlx] ridin' VN2000 Classic LT > > _______________________________________________ > > freebsd-virtualization@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > > From owner-freebsd-virtualization@freebsd.org Thu Jan 12 04:31:34 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B0CE2CAC769 for ; Thu, 12 Jan 2017 04:31:34 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 9CB451E31 for ; Thu, 12 Jan 2017 04:31:34 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id CB16FF805 for ; Wed, 11 Jan 2017 20:31:33 -0800 (PST) Date: Wed, 11 Jan 2017 20:31:33 -0800 From: To: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111203133.20c6e3d3@mscad14> In-Reply-To: <20170111195402.785f27c6@mscad14> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111195402.785f27c6@mscad14> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 04:31:34 -0000 > Xorg log bits [2] show that X is up. But the monitor stays in sleep BTW, this is what happens to Xorg: USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 638 100.0 0.8 50481584 50612 u0 R< 12:21 4:33.05 |-- /usr/local/bin/X :0 -auth /root/.serverauth.624 (Xorg) It cannot be killed, and gdb hangs when trying to attach to the process. -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Thu Jan 12 04:50:36 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8883DCACA33 for ; Thu, 12 Jan 2017 04:50:36 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 68E3B149A; Thu, 12 Jan 2017 04:50:35 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 552C5F805; Wed, 11 Jan 2017 20:50:35 -0800 (PST) Date: Wed, 11 Jan 2017 20:50:34 -0800 From: To: Cc: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111205034.22623aaa@mscad14> In-Reply-To: References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111195402.785f27c6@mscad14> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 04:50:36 -0000 > I hope you keep it up or at least figure out what the driver is doing. Not in my plans at the moment. I prefer AMD GPUs over nV for OpenCL. nVidia did (and does) serve me well for the last 10 year with their excellent FreeBSD graphics driver: I had very few problems with it; it's stable and well documented, and has the performance. Huge thanks and respect to them for that [I just bought another low-end nVidia card for my home computer]. But times a-changin'. We have things like OpenCL and virtualization these days that are of interest and available to everyone (not just the "professionals"). > If they haven't explicitly put in the license terms that virtualization > is forbidden for consumer cards, there's nothing wrong with hot > patching the driver ... Why do you care about some license? It's your card, your computer, so you're free to do what _you_ want with it. How can one "forbid" virtualization? It's impossible. Only way is to design hardware and firmware that is not amenable to VT -- but nobody would do that nowadays. > assuming that they don't do things like Skype > does where it repeatedly checksums the memory image. There are ways around things. But it is easier to buy AMD than to fight someone's stupidity or incompetence. > Good hunting. > > -M -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Thu Jan 12 05:09:07 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D152BCAC080 for ; Thu, 12 Jan 2017 05:09:07 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id BD3C91F6F; Thu, 12 Jan 2017 05:09:07 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 38ABBF805; Wed, 11 Jan 2017 21:09:07 -0800 (PST) Date: Wed, 11 Jan 2017 21:09:06 -0800 From: To: Cc: , Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111210906.5f3bcc5e@mscad14> In-Reply-To: <201701120325.v0C3PgU7047029@pdx.rh.CN85.dnsmgr.net> References: <20170111184810.5e9fb07f@mscad14> <201701120325.v0C3PgU7047029@pdx.rh.CN85.dnsmgr.net> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 05:09:07 -0000 > > Removing another signature of detecting virtualization and increasing > > compatibility would be negligible gain? Just asking... > I don't think we are going to try and defeat the NVidia virtualization > checks, and I can probably assure you that they would patch them as > fast as we bypassed them. No-no, that's not what I meant. It is not about nVidia, just general quality of the hypervisor. Ideally, running in the VM should to be indistinguishable from running on bare metal, except things like CPUID, peripheral ID strings, etc (that can be easily changed by recompiling). For instance, suppose someone is doing security research -- studying clever malware for example; you'd want to run in VM, but want to have as few ways as possible for the subject to find out that it is running in a VM. Or Internet security... Just some examples. FreeBSD was always (as far as I remember, anyway) in the lead in research (as a platform) and quality, so I figured it'd be nice to keep it this way. > This is officially an item on my plate. Functional hardware that works > doing VGA passthrough under ESXi was brought up about 2 weeks ago so I > am past that stage. I have Quadro FX3800 on a Supermicro X9DAi. Great! I'm glad to see there is interest in GPU passthrough on FreeBSD. > You have helped some in uncovering the next set of issues, but my > plate is very large, and seems to have grown appendages that are > holding all sorts of things :-) Yeah, it's a dream to have a nice, flat plates for projects :) > I am still coming up to speed on the bhyve code, so it wont be > a half hour fix by me, but it well get fixed. Count me in for help. I haven't got lots of time to play with this (or at least shouldn't be spending so much time), but can aid with testing, and also can provide access to a machine with an AMD card (or both nVidia & AMD). > Rod Grimes -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Thu Jan 12 05:39:42 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59440CAC653 for ; Thu, 12 Jan 2017 05:39:42 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 462A2107A; Thu, 12 Jan 2017 05:39:42 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id C0614F805; Wed, 11 Jan 2017 21:39:41 -0800 (PST) Date: Wed, 11 Jan 2017 21:39:41 -0800 From: To: Cc: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111213941.0789c8ce@mscad14> In-Reply-To: <839b4e54-fd7a-20fd-630f-f8aaee0c79ee@freebsd.org> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111184810.5e9fb07f@mscad14> <839b4e54-fd7a-20fd-630f-f8aaee0c79ee@freebsd.org> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 05:39:42 -0000 > That's a different issue - it's unlikely, if not impossible, to > configure bhyve with enough RAM to hit 37 bits worth where that would > become a problem. No need to worry about that. Well, there may be peripheral cards that have less bits... Anyway, I see what you mean: memory manager can always remap the DMA regions to lower virtual addresses. Good to know it's not an issue. > There are lots of BIOS/UEFI implementations out there that have the > same restriction. In general, there should be no need for a guest to > reprogram device BARs. There are all sorts of situations... But anyway, if it is too much work, then we can forget about this until someone needs it again. > After changing the 64-bit BAR base address, did you still need the > pci=nocrs option for Linux ? I'd hope this would be no longer necessary. No. As expected, the option is no longer needed. BTW, is it [generally] safe to decrease the BAR base address further? My workstation has a CPU with just 36 address bits... > The problem is the knowledge set of graphics/GPU knowhow and > equipment access, and bhyve/PCI programming, are disjoint. The time > I've spent on it has been the inverse, where I feel that I've spent a > half-day doing things that anyone who knew about graphics could get > done in a half-hour :) > > For these type of issues, joint work is best to leverage the > knowledge of both sides. From my point-of-view, the work you've done > has been very helpful. Yeah, we could benefit from more information exchange, I agree. I am trying to do my part to share what I learned; I started with no knowledge of bhyve, PCI, or GPU passthrough a week ago -- so for now my efforts appear useful, but soon the progress will stop, and we will need someone with real knowledge to step in. Incidentally, could someone put a note about that hardcoded BAR base on the bhyve PCI passthrough page [0] if it won't be fixed soon, so many others can play with VGA passthrough meanwhile? [0] https://wiki.freebsd.org/bhyve/pci_passthru > later, > Peter. -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Thu Jan 12 06:44:13 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A5A0CAC472 for ; Thu, 12 Jan 2017 06:44:13 +0000 (UTC) (envelope-from yhdeng@comcast.net) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5F8AC10A5 for ; Thu, 12 Jan 2017 06:44:13 +0000 (UTC) (envelope-from yhdeng@comcast.net) Received: by mailman.ysv.freebsd.org (Postfix) id 5C1B0CAC471; Thu, 12 Jan 2017 06:44:13 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A001CAC46F for ; Thu, 12 Jan 2017 06:44:13 +0000 (UTC) (envelope-from yhdeng@comcast.net) Received: from resqmta-ch2-01v.sys.comcast.net (resqmta-ch2-01v.sys.comcast.net [IPv6:2001:558:fe21:29:69:252:207:33]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "resqmta-po-01v.sys.comcast.net", Issuer "COMODO RSA Organization Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C74210A4 for ; Thu, 12 Jan 2017 06:44:13 +0000 (UTC) (envelope-from yhdeng@comcast.net) Received: from resomta-ch2-12v.sys.comcast.net ([69.252.207.108]) by resqmta-ch2-01v.sys.comcast.net with SMTP id RZ7LcVn8FRNZDRZ7McT7Bi; Thu, 12 Jan 2017 06:44:12 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=comcast.net; s=q20161114; t=1484203452; bh=RhlbjO/0GTaUmpacdSK4AEnl69tUxnKh165J/0BCOuw=; h=Received:Received:From:To:Subject:Date:Message-ID:Content-Type: MIME-Version; b=cXmsNmaYSZvAJK04nZvztEGWK9meOcURZkT+x8Po+XGuF3nKmOI2HqHMifOZHKHRW Vy60ElSWzaLmNBlQOr+4V65WFzqJIxq5Vle0ANxupzwm4Rl/81KWbwdjna/CnazgK3 jlVM0rPCtRSptQBVC92N79akSL0j7OVWO6aR2SqHn6Ut3ZY0N6lr8H7mNLq8xqzcbL ib7zG+aaBVZCtdKlO3wS+SdBqBi5ZhI1ZwHxesQo9yb/m6Gd4ifoiHozPqO/3+P2kZ 0RGN9V8lHCtbciDKdBKKDr+kIjHHZqbwL4aZliBPYmgL3j8aWw4FZNYWo4wg3DiDhI mb7AXiY8TWcqA== Received: from nscdt-PC ([61.6.0.203]) by resomta-ch2-12v.sys.comcast.net with SMTP id RZ5tc1Yy4WEg6RZ7EcHAHm; Thu, 12 Jan 2017 06:44:09 +0000 From: "bunkertor" To: "virtualization" Subject: =?utf-8?B?Y2FzdCB5b3VyIG1pbmQgYmFjaw==?= Date: Thu, 12 Jan 2017 10:44:03 +0400 Message-ID: <1008684778.20170112094403@comcast.net> Content-Language: en-ca MIME-Version: 1.0 X-CMAE-Envelope: MS4wfFViAcVBodnJrvCehTQIMnjVLsa8siWDy0z/HFDbMKBgNkHA+t2zkz+cVs/Sb0SXVqNs0r+jLu5WIbfra1WCxevylpIhDcsApOBXbtI1WxH2DkaGOhL6 uMvy9ttCuBZnmZWjYV5eOGfRJAbyvyZpz1bnkpTTH8LDmRMSnhcIpUxt Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 06:44:13 -0000 SGV5LCANCg0KSSB3YW50IHlvdSAgdG8gY2FzdCB5b3VyIG1pbmQgIGJhY2sgIHRvIHRoZSBmaXJz dCB0aW1lIHdlIG1ldCwgdGFrZSBhIGxvb2sgYXQgdGhlIGl0IGhlcmUgaHR0cDovL29saW1waWEu cmVkd2luZWVudGVycHJpc2UtbGxjLmNvbS8wYzBkDQoNCldhcm0gcmVnYXJkcywgYnVua2VydG9y DQoNCg== From owner-freebsd-virtualization@freebsd.org Thu Jan 12 07:10:18 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1A1BCCAC939 for ; Thu, 12 Jan 2017 07:10:18 +0000 (UTC) (envelope-from internationils@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.20]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 73A181C85 for ; Thu, 12 Jan 2017 07:10:16 +0000 (UTC) (envelope-from internationils@gmx.net) Received: from [10.0.2.15] ([87.174.158.148]) by mail.gmx.com (mrgmx103 [212.227.17.168]) with ESMTPSA (Nemesis) id 0LeeNW-1coEZC3jH4-00qQ8e for ; Thu, 12 Jan 2017 08:10:08 +0100 Subject: Re: Help getting vbox .vdi imares to run under Freenas To: freebsd-virtualization@freebsd.org References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> <4b65f427-07dc-8b0c-1ac9-59c3c1533336@freebsd.org> <201dbbc3-53de-ba98-6a88-16dd06444bc5@gmx.net> <13af4cda-44d7-5b2c-7dbe-862b3fc8be40@freebsd.org> From: Nils Message-ID: <9ec1e04b-28f7-5a09-bda3-0a5b1507c652@gmx.net> Date: Thu, 12 Jan 2017 08:10:08 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:vci0hRvi4l+gOYzvMVWcg6IKLnvyXYtFFOjXjvXSy6NhwV3ae5c G5e87MEi9XC1nQeeWDggm1H8hTKVk97kxSPflhNPJ7ODIBjREjQt8CvnSnOgjaMaVVFukyx kbVJeHdh40J0pyB1fQYnSoG4TQ86h/aqT0gs/2wK+wgB7zvz+wNTOLae0o6zwqZ0cvTG8+N YuxRgb+7/fqBBR3DkN1ug== X-UI-Out-Filterresults: notjunk:1;V01:K0:wSegJ3O3Fvc=:qMeWBI4dJ+pDOXsKu/iV5E /4Qpr9xP/bevGzLTCbODly4CLURSlkCbWEUcG/B5fxgGtKfVlSo85RAbXSYE8PcDRXjvR0jPa SJf8AnVBi+0HkVQSHEExMtSlmLUbfavqQk1wKHXEid43lLniPthLSrByehHYWy5gPv8PHa5S1 ds2KolhhvkTeJ9a0RCdNbojXlIEsPHxvbVsDlZQoTlsTpDEqz68g4Bs7ofsnMYyBaaYCwSpvF IPtpG/mvyz1sx1Xa4JwOja+djHTzrqgZYLr7PCqeT+Kb8s1kC6vjiThdTCn6an4ED2Blu9gZe MpYquUrR8dN1n76rPAyC1PzlbiNJWbrQ5QUw/JMpe7BUHgz3Kxwv3Ew4J089Nh3QR29RrkPEe T0/zYMK9JVyITDJhGwvFWFC4rU5rZkFNmySO5JgKuK0EWypUk32u7BFSb7WgrrSkySFvb8elf z0FOJpfUSycDeLrbY0+HCgWcNduL3A/hBHXxwcy9REhMtwp/pL3lLScVOh+XTP3XlseqEHi8p zQ7kPsbwIwQ/KHzKXVQQDdRfAyRx7U/g6DnQw+C6f1nc9yiSV9GbF/ll8tYiVdtJo724czs8M ELtBPiNyydvDq8xfwvvN7CqxEKuENI91jZJ3KPc9PgJfgNyc57nU4VvR2UeE+yBdHi5JFtKlW NrFVY/RYyioj3sCelwNsV5hjU/flNUUAEbEafy1gjxIIwHhdUDgZIiNozFL2+kIdMK9cmYeyV hJsHIgivE8eXse1WG+9xz/QmUsmf91Jv0FDbZ9MrkZLc+a8nNIvHeoqvAA2BbvojsE/vkVWiw rSEB7Ri X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 07:10:18 -0000 On 2017-01-08 19:04, Peter Grehan wrote: > There's some additional info at > https://github.com/pr1ntf/iohyve/issues/228 - >> grub> ls >> (hd0) (cd0) (host) >> grub> ls (hd0) >> Device hd0: No known filesystem detected - Total size 16777216 sectors > ... so it looks like grub isn't able to auto-detect the partitions. Anything that I can do to debug that? https://github.com/grehan-freebsd/grub2-bhyve doesn't appear very active... would runningit under gdb make sense, or are there any other diagnostics that you can think of? All the diagnostics I've done have said that the image looks good (extracting MBR, looking at it with fdisk -l under Linux, etc.) Thanks Nils From owner-freebsd-virtualization@freebsd.org Thu Jan 12 07:29:44 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DCC05CACFB4 for ; Thu, 12 Jan 2017 07:29:44 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id CAB6D1D5F for ; Thu, 12 Jan 2017 07:29:44 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 5F8C9F805 for ; Wed, 11 Jan 2017 23:29:43 -0800 (PST) Date: Wed, 11 Jan 2017 23:29:42 -0800 From: To: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170111232942.1e3cc7cc@mscad14> In-Reply-To: <20170111183729.6bd82398@mscad14> References: <20170111014544.70670784@mscad14> <201701111041.v0BAfOoV043390@pdx.rh.CN85.dnsmgr.net> <20170111183729.6bd82398@mscad14> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 07:29:45 -0000 > This gives me the idea to try a different driver version in Linux... Tried the same driver version in Linux as in FreeBSD. The driver seems to talk to the card now, but not sure whether I can call this progress: [ 0.536988] PCI host bridge to bus 0000:00 [ 0.537291] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.537776] pci_bus 0000:00: root bus resource [io 0x0d00-0x1fff window] [ 0.538248] pci_bus 0000:00: root bus resource [io 0x2000-0x211f window] [ 0.538722] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xc40fffff window] [ 0.539244] pci_bus 0000:00: root bus resource [mem 0x3400000000-0x340c0fffff window] [ 0.539791] pci_bus 0000:00: root bus resource [bus 00] [ 0.540204] pci 0000:00:00.0: [1275:1275] type 00 class 0x060000 [ 0.540402] pci 0000:00:00.0: reg 0x30: [mem 0x00000000-0x000007ff pref] [ 0.540557] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.540826] pci 0000:00:01.0: reg 0x30: [mem 0x00000000-0x000007ff pref] [ 0.540923] pci 0000:00:02.0: [1af4:1001] type 00 class 0x010000 [ 0.541052] pci 0000:00:02.0: reg 0x10: [io 0x2000-0x203f] [ 0.541090] pci 0000:00:02.0: reg 0x14: [mem 0xc0000000-0xc0001fff] [ 0.541273] pci 0000:00:02.0: reg 0x30: [mem 0x00000000-0x000007ff pref] [ 0.541442] pci 0000:00:03.0: [1af4:1000] type 00 class 0x020000 [ 0.541568] pci 0000:00:03.0: reg 0x10: [io 0x2040-0x205f] [ 0.541605] pci 0000:00:03.0: reg 0x14: [mem 0xc0002000-0xc0003fff] [ 0.541786] pci 0000:00:03.0: reg 0x30: [mem 0x00000000-0x000007ff pref] [ 0.541992] pci 0000:00:04.0: [10de:0dd8] type 00 class 0x030000 [ 0.542136] pci 0000:00:04.0: reg 0x10: [mem 0xc2000000-0xc3ffffff] [ 0.542198] pci 0000:00:04.0: reg 0x14: [mem 0x3400000000-0x3407ffffff 64bit pref] [ 0.542259] pci 0000:00:04.0: reg 0x1c: [mem 0x3408000000-0x340bffffff 64bit pref] [ 0.542302] pci 0000:00:04.0: reg 0x24: [io 0x2080-0x20ff] [ 0.542346] pci 0000:00:04.0: reg 0x30: [mem 0xf6000000-0xf607ffff pref] [ 0.549031] vgaarb: setting as boot device: PCI:0000:00:04.0 [ 0.549430] vgaarb: device added: PCI:0000:00:04.0,decodes=io+mem,owns=io+mem,locks=none [ 0.549995] vgaarb: loaded [ 0.550190] vgaarb: bridge control possible 0000:00:04.0 [ 0.616082] pci 0000:00:04.0: can't claim BAR 6 [mem 0xf6000000-0xf607ffff pref]: no compatible bridge window [ 0.616775] pci 0000:00:04.0: BAR 6: assigned [mem 0xc0080000-0xc00fffff pref] [ 0.617281] pci 0000:00:01.0: BAR 6: assigned [mem 0xc0004000-0xc00047ff pref] [ 0.617789] pci 0000:00:02.0: BAR 6: assigned [mem 0xc0004800-0xc0004fff pref] [ 0.618303] pci 0000:00:03.0: BAR 6: assigned [mem 0xc0005000-0xc00057ff pref] [ 0.618807] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.618808] pci_bus 0000:00: resource 5 [io 0x0d00-0x1fff window] [ 0.618809] pci_bus 0000:00: resource 6 [io 0x2000-0x211f window] [ 0.618810] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xc40fffff window] [ 0.618811] pci_bus 0000:00: resource 8 [mem 0x3400000000-0x340c0fffff window] [ 1.669308] nvidia: module verification failed: signature and/or required key missing - tainting kernel [ 1.676499] AVX2 version of gcm_enc/dec engaged. [ 1.676844] nvidia 0000:00:04.0: can't derive routing for PCI INT A [ 1.676845] nvidia 0000:00:04.0: PCI INT A: no GSI [ 1.676904] vgaarb: device changed decodes: PCI:0000:00:04.0,olddecodes=io+mem,decodes=none:owns=io+mem [ 1.676983] nvidia-nvlink: Nvlink Core is being initialized, major device number 248 [ 1.676991] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 367.57 Mon Oct 3 20:37:01 PDT 2016 [ 1.683125] AES CTR mode by8 optimization enabled [ 1.687576] [drm] Initialized drm 1.1.0 20060810 [ 1.706991] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 367.57 Mon Oct 3 20:32:57 PDT 2016 [ 1.708732] [drm] [nvidia-drm] [GPU ID 0x00000004] Loading driver After starting Xorg: [ 23.762260] divide error: 0000 [#1] SMP [ 23.762271] Modules linked in: nvidia_uvm(POE) mac_hid 8250_fintek ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear nvidia_drm(POE) nvidia_modeset(POE) drm_kms_helper crct10dif_pclmul syscopyarea sysfillrect crc32_pclmul ghash_clmulni_intel sysimgblt fb_sys_fops drm aesni_intel aes_x86_64 lrw nvidia(POE) gf128mul glue_helper ablk_helper cryptd fjes [ 23.762273] CPU: 2 PID: 1423 Comm: Xorg Tainted: P OE 4.4.0-59-generic #80-Ubuntu [ 23.762273] Hardware name: BHYVE, BIOS 1.00 03/14/2014 [ 23.762274] task: ffff880005129c00 ti: ffff880006b08000 task.ti: ffff880006b08000 [ 23.762373] RIP: 0010:[] [] _nv008359rm+0xdb/0x150 [nvidia] [ 23.762374] RSP: 0018:ffff880006b0b990 EFLAGS: 00010246 [ 23.762374] RAX: 00000000ffffffff RBX: 0000000000000000 RCX: 00000000ffffffff [ 23.762375] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000001 [ 23.762375] RBP: ffff880006062e78 R08: 0000000000000020 R09: ffff880006062e60 [ 23.762375] R10: 0000000058772da3 R11: 0000000000000000 R12: ffff880039484008 [ 23.762375] R13: 0000000000000000 R14: ffff880006046008 R15: 0000000000000200 [ 23.762376] FS: 00007fa89f280a00(0000) GS:ffff88003fd00000(0000) knlGS:0000000000000000 [ 23.762377] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 23.762377] CR2: 00007f7151f3a760 CR3: 000000003875c000 CR4: 00000000000406e0 [ 23.762378] Stack: [ 23.762379] 0000000000010000 ffff880039484008 ffff880035538008 ffff880006046008 [ 23.762380] ffff88003a170408 ffffffffc0256d39 0000000000000009 ffff88000618a008 [ 23.762380] ffff880035538008 ffff880039484008 ffff88000670e808 ffffffffc024e74d [ 23.762380] Call Trace: [ 23.762460] [] ? _nv009265rm+0xa69/0xeb0 [nvidia] [ 23.762539] [] ? _nv009268rm+0x7d/0x300 [nvidia] [ 23.762633] [] ? _nv011447rm+0xa7/0x280 [nvidia] [ 23.762700] [] ? _nv011662rm+0x3b/0x60 [nvidia] [ 23.762765] [] ? _nv016842rm+0x9b8/0xad0 [nvidia] [ 23.762831] [] ? _nv000833rm+0x32c/0x6a0 [nvidia] [ 23.762896] [] ? rm_init_adapter+0x6a/0x100 [nvidia] [ 23.762929] [] ? nv_open_device+0x186/0x620 [nvidia] [ 23.762962] [] ? nvidia_open+0x14d/0x2f0 [nvidia] [ 23.762995] [] ? nvidia_frontend_open+0x58/0xa0 [nvidia] [ 23.762997] [] ? chrdev_open+0xbf/0x1b0 [ 23.762998] [] ? do_dentry_open+0x1ff/0x310 [ 23.762999] [] ? cdev_put+0x30/0x30 [ 23.763000] [] ? vfs_open+0x54/0x80 [ 23.763002] [] ? may_open+0x5b/0xf0 [ 23.763003] [] ? path_openat+0x1b7/0x1330 [ 23.763004] [] ? simple_xattr_get+0x2f/0xb0 [ 23.763005] [] ? do_filp_open+0x91/0x100 [ 23.763006] [] ? __alloc_fd+0x46/0x190 [ 23.763007] [] ? do_sys_open+0x138/0x2a0 [ 23.763008] [] ? mntput+0x24/0x40 [ 23.763009] [] ? path_put+0x1e/0x30 [ 23.763010] [] ? SyS_open+0x1e/0x20 [ 23.763011] [] ? entry_SYSCALL_64_fastpath+0x16/0x71 [ 23.763019] Code: 14 48 89 d8 48 c1 e8 1d 48 85 c0 75 73 8d 14 dd 00 00 00 00 3b 55 04 76 03 8b 55 04 41 89 56 2c 8b 75 08 8d 4c 32 ff 31 d2 89 c8 f6 41 89 46 78 89 c1 44 89 f8 0f af 45 14 41 0f af c5 41 c1 [ 23.763100] RIP [] _nv008359rm+0xdb/0x150 [nvidia] [ 23.763100] RSP [ 23.763102] ---[ end trace ceee1eeaca570dad ]--- -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Thu Jan 12 10:49:34 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EAC83CACAFB for ; Thu, 12 Jan 2017 10:49:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DA30D1BCD for ; Thu, 12 Jan 2017 10:49:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v0CAnY1N049083 for ; Thu, 12 Jan 2017 10:49:34 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 215972] Bhyve crash more then 1 cpu AMD Date: Thu, 12 Jan 2017 10:49:34 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Jan 2017 10:49:35 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215972 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-standards@FreeBSD.o |freebsd-virtualization@Free |rg |BSD.org CC|freebsd-amd64@FreeBSD.org | --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Jan 13 00:08:56 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7451CAD676 for ; Fri, 13 Jan 2017 00:08:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D663B1A84 for ; Fri, 13 Jan 2017 00:08:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v0D08uf9077658 for ; Fri, 13 Jan 2017 00:08:56 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 213333] FreeBSD 11-RELEASE fails to boot under KVM/Qemu Date: Fri, 13 Jan 2017 00:08:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: miguelcastellanos@hotmail.es X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 00:08:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D213333 Miguel Castellanos changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |miguelcastellanos@hotmail.e | |s --- Comment #7 from Miguel Castellanos --- I confirm the same problem with Opteron 6272 Processors, under Centos 7.2/K= VM Hypervisor. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Jan 13 00:11:16 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A293CCAD89B for ; Fri, 13 Jan 2017 00:11:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9222D1BCA for ; Fri, 13 Jan 2017 00:11:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v0D0BFYs084859 for ; Fri, 13 Jan 2017 00:11:16 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 213333] FreeBSD 11-RELEASE fails to boot under KVM/Qemu Date: Fri, 13 Jan 2017 00:11:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cam@neo-zeon.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 00:11:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D213333 --- Comment #8 from Cameron --- Can anyone try booting FreeBSD 11.0 directly on the hardware? Unfortunately, I'm unable to. This may not be KVM related. I also tried 11-STABLE around a week back, the issue is still present there. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Jan 13 01:45:04 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0C3DCAD331 for ; Fri, 13 Jan 2017 01:45:04 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1CE411C95; Fri, 13 Jan 2017 01:45:03 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id v0D1irjP051209; Thu, 12 Jan 2017 17:44:53 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id v0D1ifxJ051207; Thu, 12 Jan 2017 17:44:41 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201701130144.v0D1ifxJ051207@pdx.rh.CN85.dnsmgr.net> Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) In-Reply-To: <20170111213941.0789c8ce@mscad14> To: soralx@cydem.org Date: Thu, 12 Jan 2017 17:44:41 -0800 (PST) CC: grehan@freebsd.org, freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 01:45:04 -0000 ... > > Incidentally, could someone put a note about that hardcoded BAR base > on the bhyve PCI passthrough page [0] if it won't be fixed soon, so > many others can play with VGA passthrough meanwhile? I am working with Michael Dexter to get changes made to this wiki page to reflect your work here in getting a step closer to working VGA passthrough, along with a note that we know it does not work at the present time. > [0] https://wiki.freebsd.org/bhyve/pci_passthru > > -- > [SorAlx] ridin' VN2000 Classic LT -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Fri Jan 13 02:42:51 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D1B59CACCE6 for ; Fri, 13 Jan 2017 02:42:51 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 642CE1EFA for ; Fri, 13 Jan 2017 02:42:50 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 1081A2052DED for ; Fri, 13 Jan 2017 12:42:29 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 096822809D3 for ; Fri, 13 Jan 2017 12:42:29 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id yH_-fsWcsHvf for ; Fri, 13 Jan 2017 12:42:28 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (96-82-80-65-static.hfc.comcastbusiness.net [96.82.80.65]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id E03CA280992; Fri, 13 Jan 2017 12:42:26 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111195402.785f27c6@mscad14> From: Peter Grehan Cc: freebsd-virtualization@freebsd.org Message-ID: <7cfaedbd-df48-53a8-2510-5f180ce1f2f6@freebsd.org> Date: Thu, 12 Jan 2017 18:42:44 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170111195402.785f27c6@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=okpfQa3PAAAA:8 a=aU_aj2REWSNhthCTgJkA:9 a=pILNOxqGKmIA:10 a=ajisY8KShb3P6wK432_8:22 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 02:42:51 -0000 Hi, > First, `nvidia-smi -q` output diff [0] is interesting. It suggests that > the card may be in some incompletely initialized state: notice the > "Unknown Error" instead of real UUID, and the P8 power state. Could it > be that the driver doesn't put the card's BIOS in the right state? That is extremely likely. bhyve itself doesn't have a BIOS, though bhyve/UEFI could be modified to handle options ROMs (see http://awilliam.github.io/presentations/KVM-Forum-2014/#/) > The command was run in both host and guest without Xorg loaded. Thanks for the diff; this is very useful. > - GPU UUID : GPU-f6c71b8e-f6c8-5a42-260d-1164720bf4f2 > + GPU UUID : Unknown Error That implies some type of h/w access isn't working, either MMIO registers or response from a DMA command. > - Board ID : 0x100 > + Board ID : 0x4 The same ? > PCIe Generation > Max : 2 > - Current : 2 > + Current : 1 bhyve's emulated PCI hostbridge only advertises gen-1 - that could be easily changed to gen2. That could make a difference for some of the clock issues below (source is pci_emul.c:pci_emul_add_pciecap()) > Link Width > Max : 16x > Current : 16x That's a bit unexpected since the hostbridge only advertises 1x, but the driver is probably exporting the host value here. > - Performance State : P0 > + Performance State : P8 Note sure what's happening here. > Clocks > - Graphics : 625 MHz > - SM : 1251 MHz > - Memory : 1304 MHz > - Video : 540 MHz > + Graphics : 405 MHz > + SM : 810 MHz > + Memory : 324 MHz > + Video : 405 MHz This may be related to the gen1 vs gen2 issue above. > When rebooting, I get this: > nvidia-modeset: ERROR: GPU:0: Idling display engine timed out: 0x0000857d:0:0:0x00000040 This may be DMA not working. A general issue with PCI passthrough is that often MMIO from the guest works, since that is just VT-x remapping, but DMA doesn't work due to issues with IOMMU programming (or incorrect mappings being used). This gives a device that partially works in that registers can be read, but data transfer doesn't work. > Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Display engine push buffer channel allocation failed > Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Failed to allocate display engine core DMA push buffer Not sure what's happening with those. Would it be possible to try the nouveau driver ? At least the source is available, so it may be easier to determine what is broken. later, Peter. From owner-freebsd-virtualization@freebsd.org Fri Jan 13 02:44:41 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 543D7CACDB9 for ; Fri, 13 Jan 2017 02:44:41 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 13F751F82 for ; Fri, 13 Jan 2017 02:44:40 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 9D62420AE98B for ; Fri, 13 Jan 2017 12:44:20 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 964302809D3 for ; Fri, 13 Jan 2017 12:44:20 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id erpKkND-lzOT for ; Fri, 13 Jan 2017 12:44:20 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (96-82-80-65-static.hfc.comcastbusiness.net [96.82.80.65]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id D39592809CC; Fri, 13 Jan 2017 12:44:18 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111184810.5e9fb07f@mscad14> <839b4e54-fd7a-20fd-630f-f8aaee0c79ee@freebsd.org> <20170111213941.0789c8ce@mscad14> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: <90ef4d8a-f30a-222e-3b87-a707532b2d65@freebsd.org> Date: Thu, 12 Jan 2017 18:44:36 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170111213941.0789c8ce@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=hVrhctQQT5__G7nRQyAA:9 a=o1VQUq3WoUO33KEi:21 a=SIqT9dhQQ1kWD3A9:21 a=pILNOxqGKmIA:10 a=nhxybO9QC6wA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 02:44:41 -0000 Hi, > BTW, is it [generally] safe to decrease the BAR base address further? > My workstation has a CPU with just 36 address bits... Yes. The only potential conflict is with the top of guest RAM, and 36 bits is a lot of RAM :) later, Peter. From owner-freebsd-virtualization@freebsd.org Fri Jan 13 02:48:47 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 52649CACEBE for ; Fri, 13 Jan 2017 02:48:47 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 0F60E102E for ; Fri, 13 Jan 2017 02:48:46 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 41EE820AE98B for ; Fri, 13 Jan 2017 12:48:25 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 3B7EF2809D5 for ; Fri, 13 Jan 2017 12:48:25 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id kvwa8ZbBZ4-H for ; Fri, 13 Jan 2017 12:48:25 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (96-82-80-65-static.hfc.comcastbusiness.net [96.82.80.65]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id D10082809CC; Fri, 13 Jan 2017 12:48:23 +1000 (AEST) Subject: Re: Help getting vbox .vdi imares to run under Freenas To: Nils References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> <4b65f427-07dc-8b0c-1ac9-59c3c1533336@freebsd.org> <201dbbc3-53de-ba98-6a88-16dd06444bc5@gmx.net> <13af4cda-44d7-5b2c-7dbe-862b3fc8be40@freebsd.org> <9ec1e04b-28f7-5a09-bda3-0a5b1507c652@gmx.net> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: Date: Thu, 12 Jan 2017 18:48:39 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <9ec1e04b-28f7-5a09-bda3-0a5b1507c652@gmx.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=NEAV23lmAAAA:8 a=of-J_LLlAAAA:8 a=79d3EzZwm303N1QeqIwA:9 a=pILNOxqGKmIA:10 a=7wg_fZrAzKgA:10 a=Bn2pgwyD2vrAyMmN8A2t:22 a=b_RZu896Y6R6hc4tNgcq:22 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 02:48:47 -0000 Hi Nils, > On 2017-01-08 19:04, Peter Grehan wrote: > >> There's some additional info at >> https://github.com/pr1ntf/iohyve/issues/228 - >>> grub> ls (hd0) (cd0) (host) grub> ls (hd0) Device hd0: No known >>> filesystem detected - Total size 16777216 sectors >> ... so it looks like grub isn't able to auto-detect the >> partitions. > > Anything that I can do to debug that? > https://github.com/grehan-freebsd/grub2-bhyve doesn't appear very > active... would runningit under gdb make sense, or are there any > other diagnostics that you can think of? All the diagnostics I've > done have said that the image looks good (extracting MBR, looking at > it with fdisk -l under Linux, etc.) Thanks Nils Running it under gdb is probably the best (since it's a curses app, so you will have to use the techniques in http://www.dirac.org/linux/gdb/07-Debugging_Ncurses_Programs.php) later, Peter. From owner-freebsd-virtualization@freebsd.org Fri Jan 13 02:49:59 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0071FCACF82 for ; Fri, 13 Jan 2017 02:49:59 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id B5D4210C0 for ; Fri, 13 Jan 2017 02:49:58 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 5DCD420A10C0 for ; Fri, 13 Jan 2017 12:49:38 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 567D52809D5 for ; Fri, 13 Jan 2017 12:49:38 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 836EVH1IXd5l for ; Fri, 13 Jan 2017 12:49:38 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (96-82-80-65-static.hfc.comcastbusiness.net [96.82.80.65]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id D4F112809CC; Fri, 13 Jan 2017 12:49:36 +1000 (AEST) Subject: Re: [Bug 215740] [bhyve] utilizing passthru breaks raw device usage with virtio-blk | ahci-hd To: Harry Schmalzbauer References: <58737A50.30506@omnilan.de> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: Date: Thu, 12 Jan 2017 18:49:54 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <58737A50.30506@omnilan.de> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: quoted-printable X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=6I5d2MoRAAAA:8 a=PiYCDd7JiSVUa8zsN9cA:9 a=pILNOxqGKmIA:10 a=IjZwj45LgO3ly-622nXo:22 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 02:49:59 -0000 Hi Harry, > Bez=FCglich bugzilla-noreply@freebsd.org's Nachricht vom 04.01.2017 21= :37 > (localtime): >> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215740 >> > =85 >> --- Comment #1 from Peter Grehan --- >> Would you be able to post a verbose dmesg (boot -v) ? > > Due to lack of knowledge I experimented with BIOS VT-d settings "Remap > Interrupt enable" and "Remap DMA enable". > No difference in any combination. > Is there anything else I can provide to help narrowing down the problem= ? > Has anybody else ever tried such a VM (using character special file as > virtio-blk backend together with a passthru device)? With success? I had a look at the dmesg logs you posted in the bug report -=20 everything looked fine there. I'll give it a try and see if I can repro. later, Peter. From owner-freebsd-virtualization@freebsd.org Fri Jan 13 07:48:20 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D7492CAD940 for ; Fri, 13 Jan 2017 07:48:20 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id AE9141205 for ; Fri, 13 Jan 2017 07:48:20 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 04616F805 for ; Thu, 12 Jan 2017 23:48:14 -0800 (PST) Date: Thu, 12 Jan 2017 23:48:10 -0800 From: To: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170112234810.2cb83671@mscad14> In-Reply-To: <201701130144.v0D1ifxJ051207@pdx.rh.CN85.dnsmgr.net> References: <20170111213941.0789c8ce@mscad14> <201701130144.v0D1ifxJ051207@pdx.rh.CN85.dnsmgr.net> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 07:48:20 -0000 Good news, everyone! I tried an AMD card, and it is almost working. I have a lot of logs and info, but I will try to restrain the length of this message. There was no need to do anything special to get the card to work, other than figuring out how to deal with Linux, setting up drivers and OpenCL SDK & linking libraries to the right places, compiling soft, etc. First, PCI info [0] and some dmesg bits [1]. AMD drivers: amdgpu-pro-16.50-362463.tar.xz AMD-APP-SDKInstaller-v3.0.130.136-GA-linux64.tar.bz2 Next, the good news: Xorg starts, and display works!.. Kind of: -- `glxgears` window is flickery, has parts of gears missing, and does not look good in general; -- xterm window has the rectangular cursor shapes plastered all over, in random places; -- full-screen (1600x1200) `glxgears` is slower than expected, and the performance varies suddenly [2]; -- VDPAU works, but I suspect it's not using the GPU [3][4]; I haven't figure a way how to force the use of GPU. Also, the main window with text looks OK most of the time (when doing the video test and in the end, in particular), but show a smaller black rectangle in top left corner of the screen instead of the video samples; -- it almost feels like the DMA and framebuffers aren't always correctly configured, but still are transferring data [from somewhere to somewhere sometimes]. I'm getting lots of messages like [5][6], among others, in various cases. Of some 3 OpenCL applications I tested, one appeared to complete successfully [7]. Running it also produces messages as in [6]. But the numbers make sense, comparing to e.g. tests of R9 Nano (~/mixbench/results/OpenCL/alt_R9-Nano_d1912.5.log) and expectations of the GPU chip. This is exciting! Dunno if the benchmark check whether the computations are correct, though. `clinfo` result [8] is a bit mixed... e.g., "Max clock frequency: 555Mhz" is wrong. Some more tests [9]. [0] 00:04.0 0300: 1002:67df (rev c7) (prog-if 00 [VGA controller]) Subsystem: 1682:9480 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s <64ns, L1 <1us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+ Address: 00000000fee03000 Data: 4022 Kernel driver in use: amdgpu Kernel modules: amdgpu 00:05.0 0300: 10de:0dd8 (rev a1) (prog-if 00 [VGA controller]) Subsystem: 10de:084a Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Kernel driver in use: nvidia Kernel modules: nvidiafb, nouveau, nvidia_367, nvidia_367_drm [1] [ 0.617109] pci 0000:00:04.0: can't claim BAR 6 [mem 0xf6840000-0xf685ffff pref]: no compatible bridge window [ 0.617806] pci 0000:00:05.0: can't claim BAR 6 [mem 0xf6000000-0xf607ffff pref]: no compatible bridge window [ 0.618496] pci 0000:00:05.0: BAR 6: assigned [mem 0xc0080000-0xc00fffff pref] [ 0.619000] pci 0000:00:04.0: BAR 6: assigned [mem 0xc0020000-0xc003ffff pref] [ 0.619508] pci 0000:00:01.0: BAR 6: assigned [mem 0xc0004000-0xc00047ff pref] [ 0.620011] pci 0000:00:02.0: BAR 6: assigned [mem 0xc0004800-0xc0004fff pref] [ 0.620513] pci 0000:00:03.0: BAR 6: assigned [mem 0xc0005000-0xc00057ff pref] [ 0.621033] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.621034] pci_bus 0000:00: resource 5 [io 0x0d00-0x1fff window] [ 0.621035] pci_bus 0000:00: resource 6 [io 0x2000-0x229f window] [ 0.621036] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xc40fffff window] [ 0.621037] pci_bus 0000:00: resource 8 [mem 0x3400000000-0x341c0fffff window] [ 1.692950] [drm] Initialized drm 1.1.0 20060810 [ 1.701483] AVX2 version of gcm_enc/dec engaged. [ 1.702477] AES CTR mode by8 optimization enabled [ 1.705010] nvidia: module license 'NVIDIA' taints kernel. [ 1.705982] Disabling lock debugging due to kernel taint [ 1.706716] amdgpu: module verification failed: signature and/or required key missing - tainting kernel [ 1.714692] Could not create tracefs 'fence_wait_end' directory [ 1.715984] Could not create tracefs 'fence_wait_start' directory [ 1.717618] Could not create tracefs 'fence_signaled' directory [ 1.718110] Could not create tracefs 'fence_enable_signal' directory [ 1.718593] Could not create tracefs 'fence_destroy' directory [ 1.719214] Could not create tracefs 'fence_init' directory [ 1.719616] Could not create tracefs 'fence_emit' directory [ 1.720027] Could not create tracefs 'fence_annotate_wait_on' directory [ 1.724756] nvidia 0000:00:05.0: can't derive routing for PCI INT A [ 1.726058] nvidia 0000:00:05.0: PCI INT A: no GSI [ 1.727090] vgaarb: device changed decodes: PCI:0000:00:05.0,olddecodes=io+mem,decodes=none:owns=io+mem [ 1.729188] nvidia-nvlink: Nvlink Core is being initialized, major device number 247 [ 1.730133] NVRM: loading NVIDIA UNIX x86_64 Kernel Module 367.57 Mon Oct 3 20:37:01 PDT 2016 [ 1.741012] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 367.57 Mon Oct 3 20:32:57 PDT 2016 [ 1.743758] [drm] [nvidia-drm] [GPU ID 0x00000005] Loading driver [ 1.808253] [drm] amdgpu kernel modesetting enabled. [ 1.810043] AMD IOMMUv2 driver by Joerg Roedel [ 1.810481] AMD IOMMUv2 functionality not available on this system [ 1.812928] CRAT table not found [ 1.813165] Finished initializing topology ret=0 [ 1.813570] kfd kfd: Initialized module [ 1.814088] amdgpu 0000:00:04.0: can't derive routing for PCI INT A [ 1.814527] amdgpu 0000:00:04.0: PCI INT A: no GSI [ 1.815077] [drm] initializing kernel modesetting (POLARIS10 0x1002:0x67DF 0x1682:0x9480 0xC7). [ 1.815690] [drm] register mmio base: 0xC0400000 [ 1.816012] [drm] register mmio size: 262144 [ 1.816332] [drm] doorbell mmio base: 0xC0200000 [ 1.816659] [drm] doorbell mmio size: 2097152 [ 1.816985] [drm] UVD is enabled in VM mode [ 1.817279] [drm] VCE enabled in VM mode [ 1.817567] amdgpu 0000:00:04.0: Invalid ROM contents [ 1.990245] ATOM BIOS: D00901 [ 1.990462] [drm] GPU posting now... [ 2.043833] tsc: Refined TSC clocksource calibration: 3489.546 MHz [ 2.044265] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x324cbd595a2, max_idle_ns: 440795315348 ns [ 2.121186] amdgpu 0000:00:04.0: VRAM: 8192M 0x0000000000000000 - 0x00000001FFFFFFFF (8192M used) [ 2.121808] amdgpu 0000:00:04.0: GTT: 8192M 0x0000000200000000 - 0x00000003FFFFFFFF [ 2.122342] [drm] Detected VRAM RAM=8192M, BAR=256M [ 2.122686] [drm] RAM width 256bits GDDR5 [ 2.123063] [TTM] Zone kernel: Available graphics memory: 2023196 kiB [ 2.123521] [TTM] Initializing pool allocator [ 2.123872] [TTM] Initializing DMA pool allocator [ 2.124214] [drm] amdgpu: 8192M of VRAM memory ready [ 2.124563] [drm] amdgpu: 8192M of GTT memory ready. [ 2.124914] [drm] GART: num cpu pages 2097152, num gpu pages 2097152 [ 2.126509] [drm] PCIE GART of 8192M enabled (table at 0x0000000000040000). [ 2.128085] amdgpu 0000:00:04.0: amdgpu: using MSI. [ 2.136333] [drm] amdgpu: irq initialized. [ 2.136631] Can't find requested voltage id in vdd_dep_on_sclk table! [ 2.137190] amdgpu: powerplay initialized [ 2.137738] amdgpu 0000:00:04.0: fence driver on ring 0 use gpu addr 0x0000000200000008, cpu addr 0xffff880006650008 [ 2.138528] amdgpu 0000:00:04.0: fence driver on ring 1 use gpu addr 0x0000000200000018, cpu addr 0xffff880006650018 [ 2.139317] amdgpu 0000:00:04.0: fence driver on ring 2 use gpu addr 0x0000000200000028, cpu addr 0xffff880006650028 [ 2.140119] amdgpu 0000:00:04.0: fence driver on ring 3 use gpu addr 0x0000000200000038, cpu addr 0xffff880006650038 [ 2.140912] amdgpu 0000:00:04.0: fence driver on ring 4 use gpu addr 0x0000000200000048, cpu addr 0xffff880006650048 [ 2.141689] amdgpu 0000:00:04.0: fence driver on ring 5 use gpu addr 0x0000000200000058, cpu addr 0xffff880006650058 [ 2.142490] amdgpu 0000:00:04.0: fence driver on ring 6 use gpu addr 0x0000000200000068, cpu addr 0xffff880006650068 [ 2.143283] amdgpu 0000:00:04.0: fence driver on ring 7 use gpu addr 0x0000000200000078, cpu addr 0xffff880006650078 [ 2.144055] amdgpu 0000:00:04.0: fence driver on ring 8 use gpu addr 0x0000000200000088, cpu addr 0xffff880006650088 [ 2.144888] amdgpu 0000:00:04.0: fence driver on ring 9 use gpu addr 0x0000000200000098, cpu addr 0xffff880006650098 [ 2.145663] amdgpu 0000:00:04.0: fence driver on ring 10 use gpu addr 0x00000002000000a8, cpu addr 0xffff8800066500a8 [ 2.146571] [drm] Found UVD firmware Version: 1.79 Family ID: 16 [ 2.214736] amdgpu 0000:00:04.0: fence driver on ring 11 use gpu addr 0x000000000109c420, cpu addr 0xffffc90000c5a420 [ 2.215510] [drm] Found VCE firmware Version: 52.4 Binary ID: 3 [ 2.224329] amdgpu 0000:00:04.0: fence driver on ring 12 use gpu addr 0x00000002000000c8, cpu addr 0xffff8800066500c8 [ 2.225113] amdgpu 0000:00:04.0: fence driver on ring 13 use gpu addr 0x00000002000000d8, cpu addr 0xffff8800066500d8 [ 2.225904] amdgpu 0000:00:04.0: fence driver on ring 14 use gpu addr 0x00000002000000e8, cpu addr 0xffff8800066500e8 [ 2.553099] failed to send message 150 ret is 0 [ 2.875033] failed to send pre message 136 ret is 0 [ 2.876529] [drm] DAL is enabled [ 2.876840] [drm] [FeatureOverride_overriden feature] Overridden FEATURE_ENABLE_GPU_SCALING is enabled now [ 2.877528] [drm] [Backlight_Caps] This method is not supported [ 2.878140] [drm] DM_PPLIB: values for Engine clock [ 2.878488] [drm] DM_PPLIB: 30000 [ 2.878739] [drm] DM_PPLIB: 60800 [ 2.879011] [drm] DM_PPLIB: 91000 [ 2.879264] [drm] DM_PPLIB: 107700 [ 2.879514] [drm] DM_PPLIB: 114500 [ 2.879763] [drm] DM_PPLIB: 119100 [ 2.880013] [drm] DM_PPLIB: 123600 [ 2.880259] [drm] DM_PPLIB: 128800 [ 2.880507] [drm] DM_PPLIB: Warning: using default validation clocks! [ 2.880961] [drm] DM_PPLIB: Validation clocks: [ 2.881273] [drm] DM_PPLIB: engine_max_clock: 72000 [ 2.881635] [drm] DM_PPLIB: memory_max_clock: 80000 [ 2.881996] [drm] DM_PPLIB: level : 0 [ 2.882338] [drm] DM_PPLIB: reducing engine clock level from 8 to 3 [ 2.882777] [drm] DM_PPLIB: values for Memory clock [ 2.883140] [drm] DM_PPLIB: 30000 [ 2.883381] [drm] DM_PPLIB: 200000 [ 2.883631] [drm] DM_PPLIB: Warning: using default validation clocks! [ 2.884080] [drm] DM_PPLIB: Validation clocks: [ 2.884391] [drm] DM_PPLIB: engine_max_clock: 72000 [ 2.884752] [drm] DM_PPLIB: memory_max_clock: 80000 [ 2.885115] [drm] DM_PPLIB: level : 0 [ 2.885458] [drm] DM_PPLIB: reducing memory clock level from 2 to 2 [ 2.885903] [drm] DC: create_links: connectors_num: physical:5, virtual:0 [ 2.886384] [drm] [HwTrace_Hotplug] Connector[0] description:signal 32 [ 2.886846] [drm] [I2cAux_Config] Using channel: CHANNEL_ID_DDC1 [1] [ 2.887325] [drm] [HwTrace_Hotplug] Connector[1] description:signal 32 [ 2.887781] [drm] [I2cAux_Config] Using channel: CHANNEL_ID_DDC3 [3] [ 2.888240] [drm] [HwTrace_Hotplug] Connector[2] description:signal 32 [ 2.888700] [drm] [I2cAux_Config] Using channel: CHANNEL_ID_DDC2 [2] [ 2.889153] [drm] [HwTrace_Hotplug] Connector[3] description:signal 4 [ 2.889607] [drm] [I2cAux_Config] Using channel: CHANNEL_ID_DDC4 [4] [ 2.890093] [drm] [HwTrace_Hotplug] Connector[4] description:signal 2 [ 2.890554] [drm] [I2cAux_Config] Using channel: CHANNEL_ID_DDC6 [6] [ 2.893310] [drm] [IfTrace_DC] Display Core initialized [ 2.893687] [drm] amdgpu: freesync_module init done ffff880005005700. [ 2.894391] [drm] [HwTrace_Hotplug] link=0, dc_sink_in= (null) is now Disconnected [ 2.895019] [drm] DCHPD: connector_id=0: dc_sink didn't change. [ 2.895618] [drm] [HwTrace_Hotplug] link=1, dc_sink_in= (null) is now Disconnected [ 2.896204] [drm] DCHPD: connector_id=1: dc_sink didn't change. [ 2.896791] [drm] [HwTrace_Hotplug] link=2, dc_sink_in= (null) is now Disconnected [ 2.897366] [drm] DCHPD: connector_id=2: dc_sink didn't change. [ 2.897878] [drm] [HwTrace_Hotplug] link=3, dc_sink_in= (null) is now Disconnected [ 2.898455] [drm] DCHPD: connector_id=3: dc_sink didn't change. [ 2.911776] [drm] SADs count is: -2, don't need to read it [ 2.912168] [drm] [Conn_Detect] [DVI-D-1] DELL 2007FP: [Block 0] 00 FF FF FF FF FF FF 00 10 AC 21 A0 4C 38 56 32 05 11 01 03 80 29 1F 78 EE EE 91 A3 54 4C 99 26 0F 50 54 A5 4B 00 81 80 A9 40 71 4F 01 01 01 01 01 01 01 01 01 01 48 3F 40 30 62 B0 32 40 40 C0 13 00 6F 13 11 00 00 1E 00 00 00 FF 00 43 39 35 33 36 37 31 55 32 56 38 4C 0A 00 00 00 FC 00 44 45 4C 4C 20 32 30 30 37 46 50 0A 20 00 00 00 FD 00 38 4C 1E 53 11 00 0A 20 20 20 20 20 20 00 9F ^ [ 2.914871] [drm] [Detection_EDID Parser] dc_link_detect: manufacturer_id = AC10, product_id = A021, serial_number = 3256384C, manufacture_week = 5, manufacture_year = 17, display_name = DELL 2007FP, speaker_flag = 0, audio_mode_count = 0 [ 2.916301] [drm] [HwTrace_Hotplug] link=4, dc_sink_in=ffff880137b03000 is now Connected [ 2.916867] [drm] DCHPD: connector_id=4: Old sink= (null) New sink=ffff880137b03000 [ 2.917458] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013). [ 2.917923] [drm] Driver supports precise vblank timestamp query. [ 2.918351] [drm] KMS initialized. [ 2.920363] [drm] ring test on 0 succeeded in 7 usecs [ 3.262416] clocksource: Switched to clocksource tsc [ 5.469348] [drm] ring test on 1 succeeded in 89 usecs [ 5.469725] [drm] ring test on 2 succeeded in 2 usecs [ 5.470090] [drm] ring test on 3 succeeded in 2 usecs [ 5.470455] [drm] ring test on 4 succeeded in 2 usecs [ 5.470827] [drm] ring test on 5 succeeded in 2 usecs [ 5.471193] [drm] ring test on 6 succeeded in 2 usecs [ 5.471563] [drm] ring test on 7 succeeded in 2 usecs [ 5.471931] [drm] ring test on 8 succeeded in 2 usecs [ 5.472576] [drm] ring test on 9 succeeded in 109 usecs [ 5.473104] [drm] ring test on 10 succeeded in 65 usecs [ 5.499798] [drm] ring test on 11 succeeded in 1 usecs [ 5.500163] [drm] UVD initialized successfully. [ 5.599850] [drm] ring test on 12 succeeded in 11 usecs [ 5.600239] [drm] ring test on 13 succeeded in 5 usecs [ 5.600613] [drm] ring test on 14 succeeded in 4 usecs [ 5.600975] [drm] VCE initialized successfully. [ 5.992789] [drm] fb mappable at 0x340136F000 [ 5.993113] [drm] vram apper at 0x3400000000 [ 5.993451] [drm] size 7680000 [ 5.993670] [drm] fb depth is 24 [ 5.993900] [drm] pitch is 6400 [ 5.994226] fbcon: amdgpudrmfb (fb0) is primary device [ 5.994500] [drm] Atomic commit: SET crtc id 0: [ffff880137c7a000] [ 5.994502] [drm] [IfTrace_DC] dc_commit_targets: 1 targets [ 5.994503] [drm] [IfTrace_DC] core_target 0x69b6860: stream_count=1 [ 5.994504] [drm] [IfTrace_DC] core_stream 0x381f8400: src: 0, 0, 1600, 1200; dst: 0, 0, 1600, 1200; [ 5.994505] [drm] [IfTrace_DC] pix_clk_khz: 162000, h_total: 2160, v_total: 1250 [ 5.994505] [drm] [IfTrace_DC] sink name: DELL 2007FP, serial: 844511308 [ 5.994506] [drm] [IfTrace_DC] link: 4 [ 5.995044] [drm] [Conn_Mode] [DVI-D-1] {1600x1200, 2160x1250@162000Khz}^ [ 5.995045] [drm] DM_NOT_IMPL: dm_bandwidth_update [ 5.998135] [drm] [IfTrace_DC] dc_pre_update_surfaces_to_target: commit 1 surfaces to target 0x69b6860 [ 5.998566] [drm] DM_NOT_IMPL: dm_bandwidth_update [ 5.999384] [drm] DM_NOT_IMPL: dm_bandwidth_update [ 6.000156] [drm] DM_NOT_IMPL: dm_bandwidth_update [ 6.097314] Console: switching to colour frame buffer device 200x75 [ 6.097731] [drm] DM_NOT_IMPL: dm_bandwidth_update [ 6.098073] [drm] DM_NOT_IMPL: dm_bandwidth_update [ 6.202405] amdgpu 0000:00:04.0: fb0: amdgpudrmfb frame buffer device [ 6.213420] [drm] ib test on ring 0 succeeded [ 6.213888] [drm] ib test on ring 1 succeeded [ 6.214246] [drm] ib test on ring 2 succeeded [ 6.214605] [drm] ib test on ring 3 succeeded [ 6.214962] [drm] ib test on ring 4 succeeded [ 6.215319] [drm] ib test on ring 5 succeeded [ 6.215679] [drm] ib test on ring 6 succeeded [ 6.216037] [drm] ib test on ring 7 succeeded [ 6.216395] [drm] ib test on ring 8 succeeded [ 6.216738] [drm] ib test on ring 9 succeeded [ 6.217081] [drm] ib test on ring 10 succeeded [ 6.218797] [drm] ib test on ring 11 succeeded [ 6.219554] [drm] ib test on ring 12 succeeded [ 6.221227] [drm] Initialized amdgpu 3.9.0 20150101 for 0000:00:04.0 on minor 1 [ 6.226666] [drm] DM_NOT_IMPL: dm_bandwidth_update [ 6.227220] [drm] DM_NOT_IMPL: dm_bandwidth_update [2] GL_RENDERER = AMD Radeon (TM) RX 480 Graphics GL_VERSION = 4.5.13462 Compatibility Profile Context 16.50.5 GL_VENDOR = ATI Technologies Inc. GL_EXTENSIONS = GL_AMDX_debug_output GL_AMD_blend_minmax_factor[...] VisualID 258, 0x102 878 frames in 394.5 seconds = 2.226 FPS 14175 frames in 5.0 seconds = 2834.942 FPS 14269 frames in 5.0 seconds = 2853.633 FPS 14191 frames in 5.0 seconds = 2838.192 FPS 14231 frames in 5.0 seconds = 2846.003 FPS 14376 frames in 5.0 seconds = 2875.148 FPS 14346 frames in 5.0 seconds = 2868.773 FPS 12828 frames in 5.0 seconds = 2565.346 FPS 6007 frames in 5.0 seconds = 1201.194 FPS 6006 frames in 5.0 seconds = 1201.018 FPS 6004 frames in 5.0 seconds = 1200.700 FPS 6004 frames in 5.0 seconds = 1200.657 FPS [...] 6004 frames in 5.0 seconds = 1200.800 FPS 6004 frames in 5.0 seconds = 1200.704 FPS 5983 frames in 5.0 seconds = 1196.414 FPS 6177 frames in 5.0 seconds = 1235.288 FPS 6177 frames in 5.0 seconds = 1235.267 FPS 6182 frames in 5.0 seconds = 1236.311 FPS 6183 frames in 5.0 seconds = 1236.526 FPS 6181 frames in 5.0 seconds = 1236.176 FPS 6181 frames in 5.0 seconds = 1236.167 FPS [3] # vdpauinfo display: :0 screen: 0 API version: 1 Information string: G3DVL VDPAU Driver Shared Library version 1.0 Video surface: name width height types ------------------------------------------- 420 16384 16384 NV12 YV12 422 16384 16384 UYVY YUYV 444 16384 16384 Y8U8V8A8 V8U8Y8A8 Decoder capabilities: name level macbs width height ---------------------------------------------------- MPEG1 --- not supported --- MPEG2_SIMPLE 3 65536 4096 4096 MPEG2_MAIN 3 65536 4096 4096 H264_BASELINE 52 65536 4096 4096 H264_MAIN 52 65536 4096 4096 H264_HIGH 52 65536 4096 4096 VC1_SIMPLE 1 65536 4096 4096 VC1_MAIN 2 65536 4096 4096 VC1_ADVANCED 4 65536 4096 4096 MPEG4_PART2_SP 3 65536 4096 4096 MPEG4_PART2_ASP 5 65536 4096 4096 DIVX4_QMOBILE --- not supported --- DIVX4_MOBILE --- not supported --- DIVX4_HOME_THEATER --- not supported --- DIVX4_HD_1080P --- not supported --- DIVX5_QMOBILE --- not supported --- DIVX5_MOBILE --- not supported --- DIVX5_HOME_THEATER --- not supported --- DIVX5_HD_1080P --- not supported --- H264_CONSTRAINED_BASELINE --- not supported --- H264_EXTENDED --- not supported --- H264_PROGRESSIVE_HIGH --- not supported --- H264_CONSTRAINED_HIGH --- not supported --- H264_HIGH_444_PREDICTIVE --- not supported --- HEVC_MAIN 186 65536 4096 4096 HEVC_MAIN_10 186 65536 4096 4096 HEVC_MAIN_STILL --- not supported --- HEVC_MAIN_12 --- not supported --- HEVC_MAIN_444 --- not supported --- Output surface: name width height nat types ---------------------------------------------------- B8G8R8A8 16384 16384 y NV12 YV12 UYVY YUYV Y8U8V8A8 V8U8Y8A8 A8I8 I8A8 R8G8B8A8 16384 16384 y NV12 YV12 UYVY YUYV Y8U8V8A8 V8U8Y8A8 A8I8 I8A8 R10G10B10A2 16384 16384 y NV12 YV12 UYVY YUYV Y8U8V8A8 V8U8Y8A8 A8I8 I8A8 B10G10R10A2 16384 16384 y NV12 YV12 UYVY YUYV Y8U8V8A8 V8U8Y8A8 A8I8 I8A8 Bitmap surface: name width height ------------------------------ B8G8R8A8 16384 16384 R8G8B8A8 16384 16384 R10G10B10A2 16384 16384 B10G10R10A2 16384 16384 A8 16384 16384 Video mixer: feature name sup ------------------------------------ DEINTERLACE_TEMPORAL y DEINTERLACE_TEMPORAL_SPATIAL - INVERSE_TELECINE - NOISE_REDUCTION y SHARPNESS y LUMA_KEY - HIGH QUALITY SCALING - L1 - HIGH QUALITY SCALING - L2 - HIGH QUALITY SCALING - L3 - HIGH QUALITY SCALING - L4 - HIGH QUALITY SCALING - L5 - HIGH QUALITY SCALING - L6 - HIGH QUALITY SCALING - L7 - HIGH QUALITY SCALING - L8 - HIGH QUALITY SCALING - L9 - parameter name sup min max ----------------------------------------------------- VIDEO_SURFACE_WIDTH y 48 4096 VIDEO_SURFACE_HEIGHT y 48 4096 CHROMA_TYPE y LAYERS y 0 4 attribute name sup min max ----------------------------------------------------- BACKGROUND_COLOR y CSC_MATRIX y NOISE_REDUCTION_LEVEL y 0.00 1.00 SHARPNESS_LEVEL y -1.00 1.00 LUMA_KEY_MIN_LUMA y LUMA_KEY_MAX_LUMA y [4] # ./qvdpautest Warning: LLVM emitted unknown config register: 0x4 qvdpautest 0.5.1 Intel(R) Core(TM) i5-4690 CPU @ 3.50GHz Unknown GPU VDPAU API version : 1 VDPAU implementation : G3DVL VDPAU Driver Shared Library version 1.0 FATAL: get_bits failed : No backend implementation could be loaded.!! MPEG DECODING (1920x1080): 118 frames/s MPEG DECODING (1280x720): 254 frames/s H264 DECODING (1920x1080): 277 frames/s H264 DECODING (1280x720): 476 frames/s VC1 DECODING (1440x1080): 60 frames/s MPEG4 DECODING (1920x1080): 52 frames/s MIXER WEAVE (1920x1080): 1328 frames/s MIXER BOB (1920x1080): 2730 fields/s MIXER TEMPORAL (1920x1080): 237 fields/s MIXER TEMPORAL + IVTC (1920x1080): 880 fields/s MIXER TEMPORAL + SKIP_CHROMA (1920x1080): 994 fields/s MIXER TEMPORAL_SPATIAL (1920x1080): 1021 fields/s MIXER TEMPORAL_SPATIAL + IVTC (1920x1080): 947 fields/s MIXER TEMPORAL_SPATIAL + SKIP_CHROMA (1920x1080): 918 fields/s MIXER TEMPORAL_SPATIAL (720x576 video to 1920x1080 display): 1689 fields/s MULTITHREADED MPEG DECODING (1920x1080): 43 frames/s MULTITHREADED MIXER TEMPORAL (1920x1080): 885 fields/s [5] [ 6614.076946] amdgpu 0000:00:04.0: GPU fault detected: 146 0x0750080c [ 6614.078367] amdgpu 0000:00:04.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x0010B8EA [ 6614.080023] amdgpu 0000:00:04.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0C00800C [ 6614.081672] amdgpu 0000:00:04.0: GPU fault detected: 146 0x0750880c [ 6614.083202] amdgpu 0000:00:04.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x0010B8EA [ 6614.084816] amdgpu 0000:00:04.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0608800C [6] [13757.971926] VM fault (0x0c, vmid 4) at page 1095939, read from 'TC4' (0x54433400) (72) [13757.971931] amdgpu 0000:00:04.0: GPU fault detected: 146 0x0818480c [13757.973101] amdgpu 0000:00:04.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x0010B903 [13757.974489] amdgpu 0000:00:04.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0804800C [13757.975875] VM fault (0x0c, vmid 4) at page 1095939, read from 'TC4' (0x54433400) (72) [13764.909712] amdgpu 0000:00:04.0: GPU fault detected: 146 0x07e0480c [13764.911848] amdgpu 0000:00:04.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x0010B8FC [13764.914273] amdgpu 0000:00:04.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0E04800C [13764.916700] VM fault (0x0c, vmid 7) at page 1095932, read from 'TC4' (0x54433400) (72) [7] root@lunixtst:~/mixbench# ./mixbench-ocl-alt mixbench-ocl/alternating (v0.02-25-g7587376) Use "-h" argument to see available options ------------------------ Device specifications ------------------------ Platform: AMD Accelerated Parallel Processing Device: Ellesmere/Advanced Micro Devices, Inc. Driver version: 2236.5 Address bits: 64 GPU clock rate: 555 MHz Total global mem: 8093 MB Max allowed buffer: 4048 MB OpenCL version: OpenCL 1.2 AMD-APP (2236.5) Total CUs: 14 ----------------------------------------------------------------------- Buffer size: 64MB Workgroup size: 256 Workitem stride: NDRange Buffer allocation: Device allocated Loading kernel source file... Precompilation of kernels... [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] ---------------------------------------------------------- CSV data ---------------------------------------------------------- Experiment ID, Single Precision ops,,,, Double precision ops,,,, Integer operations,,, Compute iters, Flops/byte, ex.time, GFLOPS, GB/sec, Flops/byte, ex.time, GFLOPS, GB/sec, Iops/byte, ex.time, GIOPS, GB/sec 0, 0.000, 14.36, 0.00, 598.08, 0.000, 19.61, 0.00, 876.04, 0.000, 9.88, 0.00, 869.55 1, 0.129, 10.18, 105.50, 817.59, 0.065, 18.89, 56.85, 881.23, 0.129, 10.83, 99.13, 768.29 2, 0.267, 10.33, 207.89, 779.57, 0.133, 17.37, 123.64, 927.27, 0.267, 11.78, 182.27, 683.51 3, 0.414, 10.16, 317.15, 766.46, 0.207, 19.52, 165.02, 797.57, 0.414, 13.67, 235.65, 569.49 4, 0.571, 10.71, 400.96, 701.68, 0.286, 23.51, 182.71, 639.50, 0.571, 13.32, 322.40, 564.20 5, 0.741, 10.64, 504.52, 681.10, 0.370, 26.50, 202.55, 546.90, 0.741, 13.96, 384.45, 519.01 6, 0.923, 10.07, 639.89, 693.22, 0.462, 28.05, 229.66, 497.59, 0.923, 13.72, 469.41, 508.53 7, 1.120, 9.80, 767.00, 684.82, 0.560, 30.77, 244.27, 436.19, 1.120, 16.14, 465.59, 415.71 8, 1.333, 10.49, 818.70, 614.03, 0.667, 33.39, 257.24, 385.87, 1.333, 15.47, 555.22, 416.41 9, 1.565, 9.69, 997.21, 637.11, 0.783, 36.19, 266.99, 341.16, 1.565, 16.76, 576.74, 368.47 10, 1.818, 9.60, 1117.98, 614.89, 0.909, 38.92, 275.89, 303.48, 1.818, 16.16, 664.30, 365.36 11, 2.095, 9.49, 1245.15, 594.27, 1.048, 41.58, 284.05, 271.14, 2.095, 16.84, 701.37, 334.74 12, 2.400, 8.96, 1438.38, 599.33, 1.200, 44.18, 291.62, 243.02, 2.400, 17.44, 738.98, 307.91 13, 2.737, 8.88, 1571.18, 574.09, 1.368, 51.88, 269.06, 196.62, 2.737, 18.10, 771.01, 281.72 14, 3.111, 9.15, 1643.19, 528.17, 1.556, 54.66, 275.01, 176.79, 3.111, 19.47, 772.22, 248.21 15, 3.529, 8.67, 1857.80, 526.38, 1.765, 52.21, 308.49, 174.81, 3.529, 20.13, 800.24, 226.74 16, 4.000, 7.80, 2201.42, 550.35, 2.000, 54.71, 314.03, 157.02, 4.000, 20.58, 834.89, 208.72 17, 4.533, 7.96, 2292.38, 505.67, 2.267, 56.13, 325.19, 143.46, 4.533, 20.47, 891.76, 196.71 18, 5.143, 8.07, 2394.13, 465.53, 2.571, 60.17, 321.19, 124.91, 5.143, 22.17, 871.61, 169.48 19, 5.846, 8.25, 2473.12, 423.03, 2.923, 62.81, 324.81, 111.12, 5.846, 22.82, 894.11, 152.94 20, 6.667, 8.14, 2636.84, 395.53, 3.333, 65.43, 328.20, 98.46, 6.667, 22.59, 950.54, 142.58 21, 7.636, 8.03, 2806.95, 367.58, 3.818, 68.10, 331.12, 86.72, 7.636, 23.24, 970.12, 127.04 22, 8.800, 8.23, 2870.13, 326.15, 4.400, 70.75, 333.88, 75.88, 8.800, 24.72, 955.73, 108.61 23, 10.222, 7.81, 3160.13, 309.14, 5.111, 73.54, 335.82, 65.70, 10.222, 24.52, 1007.29, 98.54 24, 12.000, 7.77, 3316.47, 276.37, 6.000, 77.60, 332.10, 55.35, 12.000, 24.97, 1031.90, 85.99 25, 14.286, 7.35, 3651.59, 255.61, 7.143, 80.30, 334.30, 46.80, 14.286, 25.61, 1048.37, 73.39 26, 17.333, 8.23, 3391.18, 195.65, 8.667, 83.22, 335.45, 38.71, 17.333, 27.26, 1024.18, 59.09 27, 21.600, 7.34, 3951.54, 182.94, 10.800, 85.91, 337.47, 31.25, 21.600, 27.10, 1069.93, 49.53 28, 28.000, 7.01, 4287.91, 153.14, 14.000, 86.43, 347.85, 24.85, 28.000, 28.36, 1060.17, 37.86 29, 38.667, 7.13, 4366.03, 112.91, 19.333, 88.14, 353.30, 18.27, 38.667, 29.04, 1072.42, 27.73 30, 60.000, 6.78, 4754.44, 79.24, 30.000, 91.74, 351.14, 11.70, 60.000, 28.78, 1119.19, 18.65 31, 124.000, 6.64, 5016.70, 40.46, 62.000, 94.45, 352.43, 5.68, 124.000, 30.30, 1098.50, 8.86 32, inf, 6.41, 5362.94, 0.00, inf, 95.21, 360.90, 0.00, inf, 29.98, 1146.16, 0.00 ------------------------------------------------------------------------------------------------------------------------------ Others tried: root@lunixtst:~/shoc# ./shoc/src/opencl/level1/md5hash/MD5Hash bash: ./shoc/src/opencl/level1/md5hash/MD5Hash: No such file or directory root@lunixtst:~/shoc# ./shoc/src/opencl/level1/md5hash/MD5Hash^C root@lunixtst:~/shoc# ./src/opencl/level1/md5hash/MD5Hash Chose device: name='Ellesmere' index=0 id=0x1005150 ERROR: mismatch in key index found. ERROR: mismatch in key index found. ERROR: mismatch in key index found. ERROR: mismatch in key index found. ERROR: mismatch in key index found. ERROR: mismatch in key index found. ERROR: mismatch in key index found. ERROR: mismatch in key index found. ERROR: mismatch in key index found. ERROR: mismatch in key index found. test atts units median mean stddev min max trial0 trial1 trial2 trial3 trial4 trial5 trial6 trial7 trial8 trial9 MD5Hash 7x10 GHash/s N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A Note: Any results marked with (*) had missing values. This can occur on systems with a mixture of device types or architectural capabilities. # ethminer -G -M #(after starting and stopping X) [OPENCL]:Found suitable OpenCL device [Ellesmere] with 8533725184 bytes of GPU memory â—‡ 23:39:16.061|ethminer #00004000… Benchmarking on platform: { "platform": "AMD Accelerated Parallel Processing", "device": "Ellesmere", "version": "OpenCL 1.2 AMD-APP (2236.5)" } Preparing DAG... â„ą 23:39:16.062|gpuminer0 workLoop 0 #00000000… #00000000… â„ą 23:39:16.063|gpuminer0 Initialising miner... Warming up... Trial 1... [OPENCL]:Using platform: AMD Accelerated Parallel Processing [OPENCL]:Using device: Ellesmere(OpenCL 1.2 AMD-APP (2236.5)) [OPENCL]:Printing program log [OPENCL]: [OPENCL]:Creating one big buffer for the DAG [OPENCL]:Loading single big chunk kernels [OPENCL]:Mapping one big chunk. 0 Trial 2... [OPENCL]:Creating buffer for header. [OPENCL]:Creating mining buffer 0 [OPENCL]:Creating mining buffer 1 [ 46.474707] amdgpu 0000:00:04.0: GPU fault detected: 147 0x00004802 [ 46.475150] amdgpu 0000:00:04.0: VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x00000000 [ 46.475672] amdgpu 0000:00:04.0: VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x0C048002 0 Trial 3... 0 Trial 4... 0 Trial 5... 0 [***HANG HERE, HAVE TO REBOOT HOST TO RESET CARD***] [8] # clinfo Number of platforms 1 Platform Name AMD Accelerated Parallel Processing Platform Vendor Advanced Micro Devices, Inc. Platform Version OpenCL 2.0 AMD-APP (2236.5) Platform Profile FULL_PROFILE Platform Extensions cl_khr_icd cl_amd_event_callback cl_amd_offline_devices Platform Extensions function suffix AMD Platform Name AMD Accelerated Parallel Processing Number of devices 2 Device Name Ellesmere Device Vendor Advanced Micro Devices, Inc. Device Vendor ID 0x1002 Device Version OpenCL 1.2 AMD-APP (2236.5) Driver Version 2236.5 Device OpenCL C Version OpenCL C 1.2 Device Type GPU Device Profile FULL_PROFILE Device Board Name (AMD) AMD Radeon (TM) RX 480 Graphics Device Topology (AMD) PCI-E, 00:04.0 Max compute units 14 SIMD per compute unit (AMD) 4 SIMD width (AMD) 16 SIMD instruction width (AMD) 1 Max clock frequency 555MHz Graphics IP (AMD) 8.0 Device Partition (core) Max number of sub-devices 14 Supported partition types none specified Max work item dimensions 3 Max work item sizes 256x256x256 Max work group size 256 Preferred work group size multiple 64 Wavefront width (AMD) 64 Preferred / native vector sizes char 4 / 4 short 2 / 2 int 1 / 1 long 1 / 1 half 1 / 1 (cl_khr_fp16) float 1 / 1 double 1 / 1 (cl_khr_fp64) Half-precision Floating-point support (cl_khr_fp16) Denormals No Infinity and NANs No Round to nearest No Round to zero No Round to infinity No IEEE754-2008 fused multiply-add No Support is emulated in software No Correctly-rounded divide and sqrt operations No Single-precision Floating-point support (core) Denormals No Infinity and NANs Yes Round to nearest Yes Round to zero Yes Round to infinity Yes IEEE754-2008 fused multiply-add Yes Support is emulated in software No Correctly-rounded divide and sqrt operations Yes Double-precision Floating-point support (cl_khr_fp64) Denormals Yes Infinity and NANs Yes Round to nearest Yes Round to zero Yes Round to infinity Yes IEEE754-2008 fused multiply-add Yes Support is emulated in software No Correctly-rounded divide and sqrt operations No Address bits 64, Little-Endian Global memory size 8495808512 (7.912GiB) Global free memory (AMD) 8278188 (7.895GiB) Global memory channels (AMD) 8 Global memory banks per channel (AMD) 16 Global memory bank width (AMD) 256 bytes Error Correction support No Max memory allocation 4244635648 (3.953GiB) Unified memory for Host and Device No Minimum alignment for any data type 128 bytes Alignment of base address 2048 bits (256 bytes) Global Memory cache type Read/Write Global Memory cache size 16384 Global Memory cache line 64 bytes Image support Yes Max number of samplers per kernel 16 Max size for 1D images from buffer 134217728 pixels Max 1D or 2D image array size 2048 images Base address alignment for 2D image buffers 256 bytes Pitch alignment for 2D image buffers 256 bytes Max 2D image size 16384x16384 pixels Max 3D image size 2048x2048x2048 pixels Max number of read image args 128 Max number of write image args 8 Local memory type Local Local memory size 32768 (32KiB) Local memory syze per CU (AMD) 65536 (64KiB) Local memory banks (AMD) 32 Max constant buffer size 65536 (64KiB) Max number of constant args 8 Max size of kernel argument 1024 Queue properties Out-of-order execution No Profiling Yes Prefer user sync for interop Yes Profiling timer resolution 1ns Profiling timer offset since Epoch (AMD) 1484275845220815516ns (Thu Jan 12 18:50:45 2017) Execution capabilities Run OpenCL kernels Yes Run native kernels No Thread trace supported (AMD) Yes SPIR versions 1.2 printf() buffer size 1048576 (1024KiB) Built-in kernels Device Available Yes Compiler Available Yes Linker Available Yes Device Extensions cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_image2d_from_buffer cl_khr_spir cl_khr_gl_event [...skip Intel CPU...] NULL platform behavior clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...) No platform clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...) No platform clCreateContext(NULL, ...) [default] No platform clCreateContext(NULL, ...) [other] Success [AMD] clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU) No platform clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU) No platform clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR) No platform clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM) No platform clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL) No platform # /opt/AMDAPPSDK-3.0/bin/x86_64/clinfo Number of platforms: 1 Platform Profile: FULL_PROFILE Platform Version: OpenCL 2.0 AMD-APP (2236.5) Platform Name: AMD Accelerated Parallel Processing Platform Vendor: Advanced Micro Devices, Inc. Platform Extensions: cl_khr_icd cl_amd_event_callback cl_amd_offline_devices Platform Name: AMD Accelerated Parallel Processing Number of devices: 2 Device Type: CL_DEVICE_TYPE_GPU Vendor ID: 1002h Board name: AMD Radeon (TM) RX 480 Graphics Device Topology: PCI[ B#0, D#4, F#0 ] Max compute units: 14 Max work items dimensions: 3 Max work items[0]: 256 Max work items[1]: 256 Max work items[2]: 256 Max work group size: 256 Preferred vector width char: 4 Preferred vector width short: 2 Preferred vector width int: 1 Preferred vector width long: 1 Preferred vector width float: 1 Preferred vector width double: 1 Native vector width char: 4 Native vector width short: 2 Native vector width int: 1 Native vector width long: 1 Native vector width float: 1 Native vector width double: 1 Max clock frequency: 555Mhz Address bits: 64 Max memory allocation: 4244635648 Image support: Yes Max number of images read arguments: 128 Max number of images write arguments: 8 Max image 2D width: 16384 Max image 2D height: 16384 Max image 3D width: 2048 Max image 3D height: 2048 Max image 3D depth: 2048 Max samplers within kernel: 16 Max size of kernel argument: 1024 Alignment (bits) of base address: 2048 Minimum alignment (bytes) for any datatype: 128 Single precision floating point capability Denorms: No Quiet NaNs: Yes Round to nearest even: Yes Round to zero: Yes Round to +ve and infinity: Yes IEEE754-2008 fused multiply-add: Yes Cache type: Read/Write Cache line size: 64 Cache size: 16384 Global memory size: 8495808512 Constant buffer size: 65536 Max number of constant args: 8 Local memory type: Scratchpad Local memory size: 32768 Max pipe arguments: 0 Max pipe active reservations: 0 Max pipe packet size: 0 Max global variable size: 0 Max global variable preferred total size: 0 Max read/write image args: 0 Max on device events: 0 Queue on device max size: 0 Max on device queues: 0 Queue on device preferred size: 0 SVM capabilities: Coarse grain buffer: No Fine grain buffer: No Fine grain system: No Atomics: No Preferred platform atomic alignment: 0 Preferred global atomic alignment: 0 Preferred local atomic alignment: 0 Kernel Preferred work group size multiple: 64 Error correction support: 0 Unified memory for Host and Device: 0 Profiling timer resolution: 1 Device endianess: Little Available: Yes Compiler available: Yes Execution capabilities: Execute OpenCL kernels: Yes Execute native function: No Queue on Host properties: Out-of-Order: No Profiling : Yes Queue on Device properties: Out-of-Order: No Profiling : No Platform ID: 0x7faab8eecdb8 Name: Ellesmere Vendor: Advanced Micro Devices, Inc. Device OpenCL C version: OpenCL C 1.2 Driver version: 2236.5 Profile: FULL_PROFILE Version: OpenCL 1.2 AMD-APP (2236.5) Extensions: cl_khr_fp64 cl_amd_fp64 cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_3d_image_writes cl_khr_byte_addressable_store cl_khr_fp16 cl_khr_gl_sharing cl_amd_device_attribute_query cl_amd_vec3 cl_amd_printf cl_amd_media_ops cl_amd_media_ops2 cl_amd_popcnt cl_khr_image2d_from_buffer cl_khr_spir cl_khr_gl_event [9] # /opt/amdgpu-pro/bin/kmstest trying to open device 'i915'...failed trying to open device 'amdgpu'...done main: All ok! # /opt/amdgpu-pro/bin/modetest trying to open device 'i915'...failed trying to open device 'amdgpu'...done Encoders: id crtc type possible crtcs possible clones 45 0 TMDS 0x0000003f 0x00000000 47 0 TMDS 0x0000003f 0x00000000 49 0 TMDS 0x0000003f 0x00000000 51 0 TMDS 0x0000003f 0x00000000 53 34 TMDS 0x0000003f 0x00000000 [...and so on...] # /opt/amdgpu-pro/bin/proptest [works fine, output similar to subset of `modetest`] # /opt/amdgpu-pro/bin/vbltest trying to open device 'i915'...failed trying to open device 'amdgpu'...done starting count: 276079 freq: 60.87Hz freq: 60.02Hz freq: 60.02Hz freq: 60.02Hz freq: 60.02Hz freq: 60.02Hz freq: 60.02Hz ^C # /opt/amdgpu-pro/bin/amdgpu_test DRM Driver: Name: [amdgpu] : Date [20150101] : Description [AMD GPU] CUnit - A unit testing framework for C - Version 2.1-3 http://cunit.sourceforge.net/ Suite: Basic Tests Test: Query Info Test ...passed Test: Memory alloc Test ...passed Test: Userptr Test ...passed Test: Command submission Test (GFX) ... [HANGS HERE] OK, I failed to restrain the length of this message. -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Fri Jan 13 08:07:49 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9EF3CAD521 for ; Fri, 13 Jan 2017 08:07:49 +0000 (UTC) (envelope-from internationils@gmx.net) Received: from mout.gmx.net (mout.gmx.net [212.227.17.21]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mout.gmx.net", Issuer "TeleSec ServerPass DE-2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 199611EAD for ; Fri, 13 Jan 2017 08:07:48 +0000 (UTC) (envelope-from internationils@gmx.net) Received: from [10.0.2.15] ([89.14.249.222]) by mail.gmx.com (mrgmx102 [212.227.17.168]) with ESMTPSA (Nemesis) id 0M6874-1cdR2b3VAA-00y9ea for ; Fri, 13 Jan 2017 09:07:43 +0100 Subject: Re: Help getting vbox .vdi imares to run under Freenas To: freebsd-virtualization@freebsd.org References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> <4b65f427-07dc-8b0c-1ac9-59c3c1533336@freebsd.org> <201dbbc3-53de-ba98-6a88-16dd06444bc5@gmx.net> <13af4cda-44d7-5b2c-7dbe-862b3fc8be40@freebsd.org> <9ec1e04b-28f7-5a09-bda3-0a5b1507c652@gmx.net> From: Nils Message-ID: <7562cdfa-2d4f-b39c-469c-dc3536b4bae6@gmx.net> Date: Fri, 13 Jan 2017 09:07:42 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Icedove/45.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:xMfxsc2vWB8DfL8D70Ryj1admKU+ZO06IjJIuT1y4r56g5cstus oGJ6iGVi4lVMWgJ+b3lAIBJ1w7v8EuJmthJX9j7sUw3ZLzTMUpMKuMdRiEJXFH425xGgSyb dLXDgyN/ChCO8UGYYePrxIfm3P+zRESlMobMEqI35/NPzaUvLOJW3YImEjlY4dSkvnumRse HFz/LGD3eej8/9XbH+ipw== X-UI-Out-Filterresults: notjunk:1;V01:K0:ATjTBk113SY=:ePD7NuR9orr2R8USFJhqDN +FXkO9Odo6dj3Z6pOFV0FRp4glC87gjf2pRituGae8T7RRcE3Gy5WjQFsmqEd2RlON1VfdSeF bDem6ci9SciN1aYBAtJpFtlFYaP1cVlbapv1NMlblGkWjefAggjTWes1imWQAGJVjYR9CoVR6 x9cPjru6z5OifODHIB0v24OsJ0CU5yu2KjNWHjQpH9tSHx1E92DBStipDkDl1mGCt2OE20hfN JEsZOABA4XiieoEcdl2Ntkv0V9R2HeO0F6xk1uBT9LHvXKsP6grPMjXW3jDG2Ul49SqdrmM1c QOo8r08dVEqdwoTllXgnc/6VZzY6mgfcW/l1d8c+fyKECO369aFpJRUQHQUnC9N/rEGsYwR36 aIFvCk/jKE6/a5UUWjk4+T085Q8lFLv9ojWWHA1pYpZrCuGPVg2olJtvgJEIVzK+XgnCW07tc BIfRUukw/pvkMwqQ65W3SR+Q0OMSxiLAFtQw64STxKbkNuxqu5yFddyvpuoZ8e2sn6dCbQvC7 0SPUMwV2je+uD+JprtWJcHT1q0vnDuYb53O/dAbdgQvIv9qEYngT/tBwjloZUzW2lwkriLEaL fLL81z5/oMXojlMkeBZTInnVjlOr5MftCTNVk+Nfk4X8TJKbKm2zjrUK/JY4zLqDRtwM9d5X+ sQbzub2iVYqNhRcSOhTTyCDOjATVT0PDIRZomrbdxoF9mld6dG1TWTsqxpHrLpLG1fcTAClSc Xo7lTsnr03c8+G5SMndZ/2PoykkYFUR7ohoMpMbcauL5k9DMQy8Pl8pO3vNEGgvRGmYiZeGWM +ziKHAwdl++yknt78VIBhxqbd/nBfiRcC5QGJEjlSQYGbqpiU6k8VjOH/W8Zm1e9ZEa/E6j/r ZpK9RgdBdEc3F7kwZwc+VvYxofvPWMGu0zIs4ZDoSGqU9BHtzy2DTUxGCymMmCxDidVY3kLk0 dEwSVGxFdaw/7heNfLwFXuqGp8wn87uqG1E16H2xRZSFHQGhOzEU6TX5W6xG9xMLqeN9yuB50 Vp0d+mlaQXXZshkaAyF3Ea1nwYZTIkWJYJ36H6KWffCqU5w9UsOqVbKhUkUXqJxamA== X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 08:07:49 -0000 On 2017-01-13 03:48, Peter Grehan wrote: > Hi Nils, >> Anything that I can do to debug that? >> https://github.com/grehan-freebsd/grub2-bhyve doesn't appear very >> active... would runningit under gdb make sense, or are there any >> other diagnostics that you can think of? All the diagnostics I've >> done have said that the image looks good (extracting MBR, looking at >> it with fdisk -l under Linux, etc.) Thanks Nils > > Running it under gdb is probably the best (since it's a curses app, so > you will have to use the techniques in > http://www.dirac.org/linux/gdb/07-Debugging_Ncurses_Programs.php) Hi Peter, can I do this inside of a jail? I'm on FreeNAS, so I don't want to (and can't) install and compile a lot of stuff in the main FN host. From what I've managed to read up on, jails don't get the full networking, but at the grub stage I should be OK, correct? Thanks Nils From owner-freebsd-virtualization@freebsd.org Fri Jan 13 08:17:38 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E74DACAD9FA for ; Fri, 13 Jan 2017 08:17:38 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id D313C14C3; Fri, 13 Jan 2017 08:17:38 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id DBD8BF805; Fri, 13 Jan 2017 00:17:37 -0800 (PST) Date: Fri, 13 Jan 2017 00:17:37 -0800 From: To: Cc: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170113001737.5fe3001b@mscad14> In-Reply-To: <7cfaedbd-df48-53a8-2510-5f180ce1f2f6@freebsd.org> References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111195402.785f27c6@mscad14> <7cfaedbd-df48-53a8-2510-5f180ce1f2f6@freebsd.org> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 08:17:39 -0000 > > First, `nvidia-smi -q` output diff [0] is interesting. It suggests > > that the card may be in some incompletely initialized state: notice > > the "Unknown Error" instead of real UUID, and the P8 power state. > > Could it be that the driver doesn't put the card's BIOS in the right > > state? > > That is extremely likely. bhyve itself doesn't have a BIOS, though > bhyve/UEFI could be modified to handle options ROMs (see > http://awilliam.github.io/presentations/KVM-Forum-2014/#/) Hm, interesting. I wonder if a card that's not designed for use with UEFI is destined not to work well/at_all with bhyve... I'll read the presentation later. > > - GPU UUID : > > GPU-f6c71b8e-f6c8-5a42-260d-1164720bf4f2 > > + GPU UUID : Unknown Error > > That implies some type of h/w access isn't working, either MMIO > registers or response from a DMA command. I have a feeling it's something to do with DMA that's not getting configured correctly for data transfers, and returns wrong data (or good data to wrong location). > > - Board ID : 0x100 > > + Board ID : 0x4 > > The same ? I'm quite sure it was the same card. > > PCIe Generation > > Max : 2 > > - Current : 2 > > + Current : 1 > > bhyve's emulated PCI hostbridge only advertises gen-1 - that could be > easily changed to gen2. That could make a difference for some of the > clock issues below > (source is pci_emul.c:pci_emul_add_pciecap()) I doubt the generation number matters. But yeah, wouldn't hurt to change it to '2'. > > Link Width > > Max : 16x > > Current : 16x > That's a bit unexpected since the hostbridge only advertises 1x, but > the driver is probably exporting the host value here. Yeah, nVidia is known to like talking directly to the card in its own, proprietary way. > > - Performance State : P0 > > + Performance State : P8 > > Note sure what's happening here. Driver not kicking the card's BIOS into the right mode to switch to dynamic power state selection? > > Clocks > > - Graphics : 625 MHz > > - SM : 1251 MHz > > - Memory : 1304 MHz > > - Video : 540 MHz > > + Graphics : 405 MHz > > + SM : 810 MHz > > + Memory : 324 MHz > > + Video : 405 MHz > > This may be related to the gen1 vs gen2 issue above. I doubt it's related to PCIe gen. Most likely because the card seems to remain in P8 (low power) mode, according to the same SMI tool. But the frequencies don't look right anyway; well, I didn't bother to look up what P8 is supposed to run at. > > When rebooting, I get this: > > nvidia-modeset: ERROR: GPU:0: Idling display engine timed out: > > 0x0000857d:0:0:0x00000040 > > This may be DMA not working. Yes, I strongly suspect DMA too, especially when it comes to DRI stuff. > A general issue with PCI passthrough is that often MMIO from the > guest works, since that is just VT-x remapping, but DMA doesn't work > due to issues with IOMMU programming (or incorrect mappings being > used). This gives a device that partially works in that registers can > be read, but data transfer doesn't work. Didn't we verify that the BARs are programmed correctly? So you're saying that bhyve has a bug in that it doesn't program the IOMMU right to match guest's memory-mapped address regions to host's addresses? > > Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: > > Display engine push buffer channel allocation failed Jan 11 11:34:49 > > fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Failed to allocate > > display engine core DMA push buffer > > Not sure what's happening with those. > > Would it be possible to try the nouveau driver ? At least the source > is available, so it may be easier to determine what is broken. I could, but for now I'd like to focus more on AMD card (which also has an open-source driver). > > BTW, is it [generally] safe to decrease the BAR base address further? > > My workstation has a CPU with just 36 address bits... > Yes. The only potential conflict is with the top of guest RAM, and 36 > bits is a lot of RAM :) 64G of RAM isn't that much these days, how incredible is that :) But you're saying there's nothing else inbetween the top of guest's RAM and the BAR base? In that case it's nothing to worry about at all, as a guest will always have less RAM that the host's CPU can address. > later, > Peter. -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Fri Jan 13 08:46:22 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC36CCAB00D for ; Fri, 13 Jan 2017 08:46:22 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id 9B1511252 for ; Fri, 13 Jan 2017 08:46:22 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id 04815F805 for ; Fri, 13 Jan 2017 00:46:22 -0800 (PST) Date: Fri, 13 Jan 2017 00:46:21 -0800 From: To: Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170113004621.16be4e0a@mscad14> In-Reply-To: <20170112234810.2cb83671@mscad14> References: <20170111213941.0789c8ce@mscad14> <201701130144.v0D1ifxJ051207@pdx.rh.CN85.dnsmgr.net> <20170112234810.2cb83671@mscad14> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 08:46:22 -0000 > -- VDPAU works, but I suspect it's not using the GPU [3][4]; > I haven't figure a way how to force the use of GPU. Also, > the main window with text looks OK most of the time (when > doing the video test and in the end, in particular), but > show a smaller black rectangle in top left corner of the > screen instead of the video samples; > -- it almost feels like the DMA and framebuffers aren't always > correctly configured, but still are transferring data [from > somewhere to somewhere sometimes]. It is possible run a VNC server, vino, on top of the Xorg session. I can send screenshots of how the display looks like to anyone interested. -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Fri Jan 13 11:41:17 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D8462CAD589 for ; Fri, 13 Jan 2017 11:41:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C7AA91C43 for ; Fri, 13 Jan 2017 11:41:17 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v0DBfG5I007728 for ; Fri, 13 Jan 2017 11:41:17 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 213333] FreeBSD 11-RELEASE fails to boot under KVM/Qemu Date: Fri, 13 Jan 2017 11:41:16 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: luciano@vespaperitivo.it X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 11:41:17 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D213333 Luciano Mannucci changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |luciano@vespaperitivo.it --- Comment #9 from Luciano Mannucci --- (In reply to Cameron from comment #8) I tried on Intel I3 Vaio and on an dual Intel(R) Xeon(R) CPU E5410 @ 2.33G= Hz and it doesn't boot. I guess it may be a bios issue linked to UEFI beeing advertized but not properly implemented (Just a guess). 10.3-RELEASE boots flawlessly on the same hardware. Hope this helps, Luciano. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Jan 13 13:06:21 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92DCACAE033 for ; Fri, 13 Jan 2017 13:06:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 823E218A9 for ; Fri, 13 Jan 2017 13:06:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v0DD6KAc013751 for ; Fri, 13 Jan 2017 13:06:21 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 213333] FreeBSD 11-RELEASE fails to boot under KVM/Qemu Date: Fri, 13 Jan 2017 13:06:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: miguelcastellanos@hotmail.es X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 13:06:21 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D213333 --- Comment #10 from Miguel Castellanos --- I tried directly on the hardware as requested. FreeBSD-11.RELEASE booted correctly under the physical hardware, but fails = to boot as guest on a qemu virtual machine under KVM (Centos 7.2) on the same hardware. I tested on a SuperMicro server with dual AMD Opteron 6272 processors, with= a non UEFI BIOS. If you need more details please let me know. Thanks. Miguel --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Jan 13 15:47:24 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2BCC7CAD378 for ; Fri, 13 Jan 2017 15:47:24 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E40061335; Fri, 13 Jan 2017 15:47:23 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id v0DFlFVp054298; Fri, 13 Jan 2017 07:47:15 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id v0DFlEjt054297; Fri, 13 Jan 2017 07:47:14 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201701131547.v0DFlEjt054297@pdx.rh.CN85.dnsmgr.net> Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) In-Reply-To: <20170113001737.5fe3001b@mscad14> To: soralx@cydem.org Date: Fri, 13 Jan 2017 07:47:14 -0800 (PST) CC: grehan@freebsd.org, freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 15:47:24 -0000 ... > > > - Performance State : P0 > > > + Performance State : P8 > > > > Note sure what's happening here. > > Driver not kicking the card's BIOS into the right mode > to switch to dynamic power state selection? I suspect this is a failure to run the BIOS code that enables the secondary power connector so you can come out of slot only power mode. > > > Jan 11 11:34:49 fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: > > > Display engine push buffer channel allocation failed Jan 11 11:34:49 > > > fbsd12tst kernel: nvidia-modeset: ERROR: GPU:0: Failed to allocate > > > display engine core DMA push buffer > > > > Not sure what's happening with those. > > > > Would it be possible to try the nouveau driver ? At least the source > > is available, so it may be easier to determine what is broken. > > I could, but for now I'd like to focus more on AMD card > (which also has an open-source driver). The general rule on other platforms is that ATI/AMD cards tend to just work, where as the NVidia cards are very picky and unless official listed as known to work when passed through you well fight problems. Very few cards are listed as known to work, most populars being the Quadro 2000, and the Quadro FX3800. Many cards are listed as known to NOT work. ... GOOD WORK on getting as far as you have as quickly as you have! Note that the https://wiki.freebsd.org/bhyve/pci_passthru has had a small update to reflect we know that VGA passthrough is not working at this time. Also a note about AMD IOmmu/AMD-Vi was added, hopefully saving someone from duplicate work. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Fri Jan 13 17:37:21 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 07D3ACAEB08 for ; Fri, 13 Jan 2017 17:37:21 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EBB761A81 for ; Fri, 13 Jan 2017 17:37:20 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v0DHbKZx080944 for ; Fri, 13 Jan 2017 17:37:20 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 216009] Floppy does not work under KVM Date: Fri, 13 Jan 2017 17:37:21 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: linimon@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 17:37:21 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D216009 Mark Linimon changed: What |Removed |Added ---------------------------------------------------------------------------- CC|freebsd-amd64@FreeBSD.org | Assignee|freebsd-bugs@FreeBSD.org |freebsd-virtualization@Free | |BSD.org --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Jan 13 19:04:38 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 636CACAC72B for ; Fri, 13 Jan 2017 19:04:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 523F41007 for ; Fri, 13 Jan 2017 19:04:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v0DJ4c9G058416 for ; Fri, 13 Jan 2017 19:04:38 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 215471] Using bsnmpd with the snmp_hostres module on a vmware ESXi guest with a disconnected CD drive uses 100% CPU Date: Fri, 13 Jan 2017 19:04:38 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: stl@ossuary.net X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 19:04:38 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215471 Scott Larson changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |stl@ossuary.net --- Comment #3 from Scott Larson --- I've run into this myself. In the case of ESXi VMs my workaround has been to remove the cd device from the VM config. However I've noticed this is not limited to just that scope. This issue of relentlessly polling cd0 when the hostres module is enabled has also popped up on some purely physical instal= ls as well. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Jan 13 22:23:22 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EBF77CAEA3F for ; Fri, 13 Jan 2017 22:23:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D76AC1AD4 for ; Fri, 13 Jan 2017 22:23:22 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v0DMNLfI043745 for ; Fri, 13 Jan 2017 22:23:22 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 213333] FreeBSD 11-RELEASE fails to boot under KVM/Qemu Date: Fri, 13 Jan 2017 22:23:22 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cam@neo-zeon.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 22:23:23 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D213333 --- Comment #11 from Cameron --- I have a non-UEFI BIOS with a Supermicro H8DG6-F motherboard and dual 6348 CPU's. This issue doesn't appear to be UEFI related. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Jan 13 23:48:16 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 481FACAE987 for ; Fri, 13 Jan 2017 23:48:16 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 00E4E1B3D for ; Fri, 13 Jan 2017 23:48:15 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 3701620B4A16 for ; Sat, 14 Jan 2017 09:47:48 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 2ABA52809EF for ; Sat, 14 Jan 2017 09:47:48 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id Tr-XsKkVytTh for ; Sat, 14 Jan 2017 09:47:48 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (96-82-80-65-static.hfc.comcastbusiness.net [96.82.80.65]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 0D5B92809E2; Sat, 14 Jan 2017 09:47:44 +1000 (AEST) Subject: Re: Help getting vbox .vdi imares to run under Freenas To: Nils References: <9bcff734-5ed5-a228-df5e-5ff35cf9e3ff@gmx.net> <4b65f427-07dc-8b0c-1ac9-59c3c1533336@freebsd.org> <201dbbc3-53de-ba98-6a88-16dd06444bc5@gmx.net> <13af4cda-44d7-5b2c-7dbe-862b3fc8be40@freebsd.org> <9ec1e04b-28f7-5a09-bda3-0a5b1507c652@gmx.net> <7562cdfa-2d4f-b39c-469c-dc3536b4bae6@gmx.net> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: <1e2fdfce-4ead-73d3-557a-0f8b81b85fb3@freebsd.org> Date: Fri, 13 Jan 2017 15:48:00 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <7562cdfa-2d4f-b39c-469c-dc3536b4bae6@gmx.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=PUXyLj3wagtSbv8cjS0A:9 a=pILNOxqGKmIA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Jan 2017 23:48:16 -0000 Hi Nils, > can I do this inside of a jail? I'm on FreeNAS, so I don't want to (and > can't) install and compile a lot of stuff in the main FN host. From what > I've managed to read up on, jails don't get the full networking, but at > the grub stage I should be OK, correct? I've not tried it but I believe so. The /dev/vmm device file would have to be made available in the jail. later, Peter. From owner-freebsd-virtualization@freebsd.org Sat Jan 14 05:54:40 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 278E1CAFE0F for ; Sat, 14 Jan 2017 05:54:40 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from smtp.triumf.ca (smtp.triumf.ca [142.90.100.188]) by mx1.freebsd.org (Postfix) with ESMTP id EB4F21CF0; Sat, 14 Jan 2017 05:54:39 +0000 (UTC) (envelope-from soralx@cydem.org) Received: from mscad14 (mscad14.triumf.ca [142.90.115.36]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by smtp.triumf.ca (Postfix) with ESMTP id D3253F805; Fri, 13 Jan 2017 21:54:38 -0800 (PST) Date: Fri, 13 Jan 2017 21:54:38 -0800 From: To: Cc: , Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) Message-ID: <20170113215438.01c3b674@mscad14> In-Reply-To: <201701131547.v0DFlEjt054297@pdx.rh.CN85.dnsmgr.net> References: <20170113001737.5fe3001b@mscad14> <201701131547.v0DFlEjt054297@pdx.rh.CN85.dnsmgr.net> X-Mailer: Claws Mail 3.14.1 (GTK+ 2.24.29; amd64-portbld-freebsd9.3) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jan 2017 05:54:40 -0000 > I suspect this is a failure to run the BIOS code that > enables the secondary power connector so you can come > out of slot only power mode. Well, that Quadro does not have a power connector, but I imagine card BIOS routines would be similar between all cards in a family, including those that require the extra power. Does bhyve not execute peripheral cards' option ROMs? I guess it doesn't. This could explain a lot of strange behaviour seen resulting from running in a VM. How does UEFI work in this regard? My guess is that cards have to explicitly support the new boot method (UEFI)? So passthrough with newer cards may be easier? This could explain why the newer RX 480 worked right away, and the older Quadro 2000 (and a lot of other nVidia cards without manufacturer's support for VMs) had no chance -- UEFI cards are somehow more "autonomous". It all is just speculation on my side, I know nothing about this UEFI stuff. Could you summarize in couple sentences what's the deal between bhyve and UEFI (if there is any), or future plans? > The general rule on other platforms is that ATI/AMD cards tend to just > work, where as the NVidia cards are very picky and unless official > listed as known to work when passed through you well fight problems. > Very few cards are listed as known to work, most populars being the > Quadro 2000, and the Quadro FX3800. Many cards are listed as known to > NOT work. Yeah, messing with nVidia for this reason (and because of the closed driver) seems to me like a huge time sink. I don't have the time, so decided to try with AMD for now. I am only interested in nVidia because it's best choice for a Windows VM I hope to run (SolidWorks, Altium, etc). OpenCL on AMD is a priority for me personally right now anyway. > GOOD WORK on getting as far as you have as quickly as you have! > Note that the https://wiki.freebsd.org/bhyve/pci_passthru has > had a small update to reflect we know that VGA passthrough is > not working at this time. Also a note about AMD IOmmu/AMD-Vi > was added, hopefully saving someone from duplicate work. Perhaps a note could be added about ATI/AMD cards partially working, to encourage others to play? -- [SorAlx] ridin' VN2000 Classic LT From owner-freebsd-virtualization@freebsd.org Sat Jan 14 06:54:10 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 26682CAFFC2 for ; Sat, 14 Jan 2017 06:54:10 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id AF8DB1EF7 for ; Sat, 14 Jan 2017 06:54:09 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id C44A520B4B42 for ; Sat, 14 Jan 2017 16:53:47 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id BCD492809CC for ; Sat, 14 Jan 2017 16:53:47 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id nsesBwNRLpKR for ; Sat, 14 Jan 2017 16:53:47 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 98E4C280996; Sat, 14 Jan 2017 16:53:45 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org References: <20170110003332.7cf8ba15@mscad14> <0de7e0fe-5680-b1be-bd57-6bf446c2fd38@talk2dom.com> <0c927784-3e3f-7946-fba9-c25001f4156c@talk2dom.com> <20170110180117.7f246b5a@mscad14> <20170111014544.70670784@mscad14> <93196ea2-5439-49ff-54fd-7b7273bdec85@freebsd.org> <20170111195402.785f27c6@mscad14> <7cfaedbd-df48-53a8-2510-5f180ce1f2f6@freebsd.org> <20170113001737.5fe3001b@mscad14> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: Date: Fri, 13 Jan 2017 22:54:01 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170113001737.5fe3001b@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=okpfQa3PAAAA:8 a=T6pNkRvUAAAA:8 a=HsyY3lUf_bTxMnHSjM4A:9 a=TgtqQ1FgE2jwKowV:21 a=SnqjUP613dYyXqCm:21 a=pILNOxqGKmIA:10 a=kPJpQtqUOtsA:10 a=gA6IeH5FQcgA:10 a=NWVoK91CQyQA:10 a=ajisY8KShb3P6wK432_8:22 a=hgMNUkviuzi0Hbumswsj:22 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jan 2017 06:54:10 -0000 Hi, >> That is extremely likely. bhyve itself doesn't have a BIOS, though >> bhyve/UEFI could be modified to handle options ROMs (see >> http://awilliam.github.io/presentations/KVM-Forum-2014/#/) > > Hm, interesting. I wonder if a card that's not designed for use > with UEFI is destined not to work well/at_all with bhyve... > I'll read the presentation later. I think in general almost all cards have UEFI ROM support these days since it has been mandated by Microsoft. However, as Rod mentioned, the bhyve UEFI implementation does not run PCI device option ROMs. (see http://vfio.blogspot.com/2014/08/does-my-graphics-card-rom-support-efi.html) >>> - GPU UUID : >>> GPU-f6c71b8e-f6c8-5a42-260d-1164720bf4f2 >>> + GPU UUID : Unknown Error >> >> That implies some type of h/w access isn't working, either MMIO >> registers or response from a DMA command. > > I have a feeling it's something to do with DMA that's > not getting configured correctly for data transfers, > and returns wrong data (or good data to wrong location). Yes. >> A general issue with PCI passthrough is that often MMIO from the >> guest works, since that is just VT-x remapping, but DMA doesn't work >> due to issues with IOMMU programming (or incorrect mappings being >> used). This gives a device that partially works in that registers can >> be read, but data transfer doesn't work. > > Didn't we verify that the BARs are programmed correctly? The BAR values you see are fictional and are created by bhyve. The actual physical BAR values are those set up by the host BIOS. bhyve uses EPT mappings to translate between the 'fake' value and the real value. > So you're saying that bhyve has a bug in that it doesn't > program the IOMMU right to match guest's memory-mapped > address regions to host's addresses? There isn't a known bug, but the 64-bit BAR region hasn't been tested for a long time so it's possible there is an issue with it. >>> BTW, is it [generally] safe to decrease the BAR base address further? >>> My workstation has a CPU with just 36 address bits... >> Yes. The only potential conflict is with the top of guest RAM, and 36 >> bits is a lot of RAM :) > > 64G of RAM isn't that much these days, how incredible is that :) > But you're saying there's nothing else inbetween the top of > guest's RAM and the BAR base? In that case it's nothing to > worry about at all, as a guest will always have less RAM that > the host's CPU can address. Right - the 64-bit PCI decode region would be set dynamically based on the phys address bits, rather than being a hard-coded value. later, Peter. From owner-freebsd-virtualization@freebsd.org Sat Jan 14 07:13:14 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 74E3CCAF36E for ; Sat, 14 Jan 2017 07:13:14 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 2838915DE for ; Sat, 14 Jan 2017 07:13:13 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 174AC20B4B42 for ; Sat, 14 Jan 2017 17:12:52 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 0D8732809D3 for ; Sat, 14 Jan 2017 17:12:52 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id QUcncdWKJhQY for ; Sat, 14 Jan 2017 17:12:51 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id A8521280996; Sat, 14 Jan 2017 17:12:46 +1000 (AEST) Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD 11-RC2) To: soralx@cydem.org, freebsd-rwg@pdx.rh.CN85.dnsmgr.net References: <20170113001737.5fe3001b@mscad14> <201701131547.v0DFlEjt054297@pdx.rh.CN85.dnsmgr.net> <20170113215438.01c3b674@mscad14> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: <2f0df251-9ce0-93c7-ad23-81479cef4f03@freebsd.org> Date: Fri, 13 Jan 2017 23:13:04 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <20170113215438.01c3b674@mscad14> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=N659UExz7-8A:10 a=IgFoBzBjUZAA:10 a=NEAV23lmAAAA:8 a=QwR64u4WghamZgzYezwA:9 a=pILNOxqGKmIA:10 a=pj9Md4Rj8o0A:10 a=Bn2pgwyD2vrAyMmN8A2t:22 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jan 2017 07:13:14 -0000 Hi, > Does bhyve not execute peripheral cards' option ROMs? Not yet. > I guess it doesn't. This could explain a lot of strange > behaviour seen resulting from running in a VM. Yes. > How does UEFI work in this regard? My guess is that cards > have to explicitly support the new boot method (UEFI)? Yes - an additional section in the option ROM is needed, but as mentioned in an earlier email, that support is now widespread thanks to Windows. > So passthrough with newer cards may be easier? This could > explain why the newer RX 480 worked right away, and the > older Quadro 2000 (and a lot of other nVidia cards without > manufacturer's support for VMs) had no chance -- UEFI cards > are somehow more "autonomous". Possibly, though it might also be the card itself not requiring as much initialization from the option ROM. > It all is just speculation on my side, I know nothing about > this UEFI stuff. > > Could you summarize in couple sentences what's the deal between > bhyve and UEFI (if there is any), or future plans? UEFI is the ROM firmware for bhyve (and most modern PCs). bhyve has a custom build of the standard Intel EDK2 distribution: https://github.com/freebsd/uefi-edk2/tree/bhyve/UDK2014.SP1 The changes are to support running as a hypervisor guest, where a lot of what is in a normal boot ROM isn't required (e.g. DRAM controller setup, CPU microcode update), and it also contains drivers for device emulations supported by bhyve. Currently, the ability to process an option ROM has been disabled. later, Peter. From owner-freebsd-virtualization@freebsd.org Sat Jan 14 10:38:16 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1458ACA7831 for ; Sat, 14 Jan 2017 10:38:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 03CF216BE for ; Sat, 14 Jan 2017 10:38:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id v0EAcFJa054784 for ; Sat, 14 Jan 2017 10:38:15 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 215235] Can't use iSCSI targets on AWS storage gateway Date: Sat, 14 Jan 2017 10:38:15 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 10.3-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: trasz@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: trasz@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc bug_status assigned_to Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jan 2017 10:38:16 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215235 Edward Tomasz Napierala changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |trasz@FreeBSD.org Status|New |Open Assignee|freebsd-virtualization@Free |trasz@FreeBSD.org |BSD.org | --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Sat Jan 14 17:59:02 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BFD92CAFC61 for ; Sat, 14 Jan 2017 17:59:02 +0000 (UTC) (envelope-from doctor@doctor.nl2k.ab.ca) Received: from doctor.nl2k.ab.ca (doctor.nl2k.ab.ca [204.209.81.1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AAE74187B for ; Sat, 14 Jan 2017 17:59:02 +0000 (UTC) (envelope-from doctor@doctor.nl2k.ab.ca) Received: from doctor by doctor.nl2k.ab.ca with local (Exim 4.88) (envelope-from ) id 1cSSbV-000Gvm-TO for freebsd-virtualization@freebsd.org; Sat, 14 Jan 2017 10:59:01 -0700 Date: Sat, 14 Jan 2017 10:59:01 -0700 From: The Doctor To: freebsd-virtualization@freebsd.org Subject: bhyve /Xen Message-ID: <20170114175901.GB64860@doctor.nl2k.ab.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.7.2 (2016-11-26) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Jan 2017 17:59:02 -0000 1) What do I need to send a bhyve crashing report? Seems that yes 6 Vms are running and are crashing at the same time. 2) Windows 2016 and bhyve Why can Widnows 2016 rpc not work with bhyve? 3) Bhyve to Xen . Is there an easy way to migrate from bhyve to Xen on FreeBSD 11 ? -- Member - Liberal International This is doctor@@nl2k.ab.ca Ici doctor@@nl2k.ab.ca God,Queen and country!Never Satan President Republic!Beware AntiChrist rising! http://www.fullyfollow.me/rootnl2k Look at Psalms 14 and 53 on Atheism Birthdate 29 Jan 1969 Redhill Surrey England