From owner-freebsd-virtualization@freebsd.org  Fri Sep  9 10:49:29 2016
Return-Path: <owner-freebsd-virtualization@freebsd.org>
Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D3DFBD4AEC
 for <freebsd-virtualization@mailman.ysv.freebsd.org>;
 Fri,  9 Sep 2016 10:49:29 +0000 (UTC)
 (envelope-from misc-freebsd@talk2dom.com)
Received: from mail.shmtech.biz (mail.shmtech.biz [IPv6:2001:41c9:1:37::e])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client CN "mail.shmtech.biz",
 Issuer "Go Daddy Secure Certification Authority" (verified OK))
 by mx1.freebsd.org (Postfix) with ESMTPS id AB6CCBC4
 for <freebsd-virtualization@freebsd.org>; Fri,  9 Sep 2016 10:49:28 +0000 (UTC)
 (envelope-from misc-freebsd@talk2dom.com)
Received: from [10.0.1.100] (originc9729.pndsl.co.uk [84.92.225.78])
 (authenticated bits=0)
 by mail.shmtech.biz (8.15.2/8.14.9) with ESMTPSA id u89AnPhS013143
 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO)
 for <freebsd-virtualization@freebsd.org>; Fri, 9 Sep 2016 11:49:25 +0100 (BST)
 (envelope-from misc-freebsd@talk2dom.com)
DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=talk2dom.com;
 s=shmtech4; t=1473418165; x=1474022965;
 bh=ALBXmBsxqQ9lP1rWLpDlk3lEedpi1Ec8C64mQkxP4K8=;
 h=Subject:To:References:From:Date:In-Reply-To;
 b=BXoqjHfdw99/IqeGDJTRjQ6Ajc3H5cSvhyvJElMf0JWk0eEUiyQp9HFFdLT6lZwNf
 v5J3haq68nANxR55pv8d/tWn7TWMZujxLINIluVLgUtnnf4j1PDGdpzoFepA9X4EmH
 XOS13Q6nocn6wwzpQj1o8DxxgfBjeuT33KfcBM/0=
X-Authentication-Warning: sendmail: Host originc9729.pndsl.co.uk
 [84.92.225.78] claimed to be [10.0.1.100]
Subject: Re: Issues with GTX960 on CentOS7 using bhyve PCI passthru (FreeBSD
 11-RC2)
To: freebsd-virtualization@freebsd.org
References: <727839e2-4be1-5661-8c19-6d13f73ad36f@talk2dom.com>
 <62b2a72a-3662-5b3e-5029-f284a625d05b@freebsd.org>
 <22df7de9-1dfb-561c-125d-99af2b3da70b@talk2dom.com>
From: Dom <misc-freebsd@talk2dom.com>
Message-ID: <2a0d0094-3263-de91-b3f8-38dda4a523ac@talk2dom.com>
Date: Fri, 9 Sep 2016 11:49:25 +0100
User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101
 Thunderbird/45.2.0
MIME-Version: 1.0
In-Reply-To: <22df7de9-1dfb-561c-125d-99af2b3da70b@talk2dom.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
Content-Transfer-Encoding: 7bit
X-BeenThere: freebsd-virtualization@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: "Discussion of various virtualization techniques FreeBSD supports."
 <freebsd-virtualization.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-virtualization>, 
 <mailto:freebsd-virtualization-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-virtualization/>
List-Post: <mailto:freebsd-virtualization@freebsd.org>
List-Help: <mailto:freebsd-virtualization-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization>, 
 <mailto:freebsd-virtualization-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Fri, 09 Sep 2016 10:49:29 -0000

It looks like there may not be an issue with MSI after all.

The nvidia driver is issued an IRQ when first used, not at boot time.
If I run the CUDA "deviceQuery" sample then this appears in dmesg:

[   67.207929] nvidia 0000:00:06.0: irq 29 for MSI/MSI-X
[   67.646207] NVRM: RmInitAdapter failed! (0x24:0x1f:1356)
[   67.646570] NVRM: rm_init_adapter failed for device bearing minor 
number 0
[   67.647214] NVRM: nvidia_frontend_open: minor 0, module->open() 
failed, error -5

Maybe the IRQ is deallocated immediately so doesn't appear in the output 
of /proc/interrupts?

I guess I'll need to research the NVRM error above now but at least this 
thread might be useful regarding the BAR allocation.