Date: Fri, 3 Feb 2023 14:07:42 +0100 From: Mario Marietto <marietto2008@gmail.com> To: "Daniel O'Connor" <darius@dons.net.au> Cc: freebsd-hackers <freebsd-hackers@freebsd.org> Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy Message-ID: <CA%2B1FSigVQr=B0FNvgWXHgTgSRWnc8cfo2zJf6azt2P1uLT%2BasQ@mail.gmail.com> In-Reply-To: <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au> References: <CA%2B1FSigT376-VQoKM-D96fpUZi5MWDbdNjjeAgdhnWs0vVzN-A@mail.gmail.com> <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> <CA%2B1FSiiA3vn7saY4Xnhh1AwfO54f_pKxpPzB1KoECB_SG7=p6A@mail.gmail.com> <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Hello Daniel and everyone. ---> If you want to pass through the video card (as evidenced by devctl and pptdevs) then why have the nvidia driver installed? Why did you say this ? Are you aware of some bug that prevents a pci device from being passed if its driver is installed ? Maybe there is. Infact I made some progress. I have commented this line on /etc/rc.conf : #kld_list="nvidia nvidia-modeset" and I've been able to detach and reattach the 4 slots of the 2080 ti. I think that if I load the nvidia modules on the rc.conf,they interfere with a later attaching / detaching of 1 or 2 slots of the gpu (hdac and vgapci). Do you think that it could be caused by a bug on the nvidia driver or in the bhyve source code ? What do you suggest I do ? Il giorno mer 1 feb 2023 alle ore 13:49 Daniel O'Connor <darius@dons.net.au> ha scritto: > > > > On 1 Feb 2023, at 21:41, Mario Marietto <marietto2008@gmail.com> wrote: > > ---> Do you have 'vmm_load="YES"' in loader.conf ? > > > > yes. > > > > ---> If you want to pass through the video card (as evidenced by devctl > and pptdevs) then why have the nvidia driver installed? > > > > because I have 3 gpus on my PC. I use the intel gpu on the host ; I > would like to use the nvidia gtx 1060 within a bhyve Linux vm and the 2080 > ti with the Linuxulator to run stable diffusion,or vice versa. Stable > diffusion needs pytorch + cuda and they need the nVidia driver installed on > FreeBSD and the modules loaded. > > > > My goal is to run stable diffusion and a bhyve / linux vm +my nvidia > gpus attached. > > I note that https://wiki.freebsd.org/bhyve/pci_passthru says "Note: VGA / > GPU pass-through devices are not currently supported" so perhaps that is > why it is ignoring your pptdevs line. > > Although that said some googling suggests that is not the case so I don't > know, unfortunately I haven't tried it myself. > > There are some threads about it on the forums, eg: > https://forums.freebsd.org/threads/bhyve-gpu-pass-through.83152/ > > Those suggest that modifications to bhyve are required before it will work > properly but none of them are for vmm that I can see. > > -- > Daniel O'Connor > "The nice thing about standards is that there > are so many of them to choose from." > -- Andrew Tanenbaum > > -- Mario. [-- Attachment #2 --] <div dir="ltr"><div>Hello Daniel and everyone.</div><div><span class="gmail-im"> <br></span> ---> If you want to pass through the video card (as evidenced by devctl and pptdevs) then why have the nvidia driver installed?</div><div><br></div><div>Why did you say this ? Are you aware of some bug that prevents a pci device from being passed if its driver is installed ? <br></div><div><br></div><div>Maybe there is. Infact I made some progress. I have commented this line on /etc/rc.conf :</div><div><br></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">#kld_list="nvidia nvidia-modeset"</span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)"><br></span></span></div><div><span style="font-family:arial,sans-serif"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">and I've been able to detach and reattach the 4 slots of the 2080 ti. I think that if I load the nvidia modules on the rc.conf,they interfere with a later attaching / detaching of 1 or 2 slots of the gpu (hdac and vgapci). Do you think that it could be caused by a bug on the nvidia driver or in the bhyve source code ? What do you suggest I do ?<br></span></span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mer 1 feb 2023 alle ore 13:49 Daniel O'Connor <<a href="mailto:darius@dons.net.au">darius@dons.net.au</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> <br> > On 1 Feb 2023, at 21:41, Mario Marietto <<a href="mailto:marietto2008@gmail.com" target="_blank">marietto2008@gmail.com</a>> wrote:<br> > ---> Do you have 'vmm_load="YES"' in loader.conf ?<br> > <br> > yes.<br> > <br> > ---> If you want to pass through the video card (as evidenced by devctl and pptdevs) then why have the nvidia driver installed?<br> > <br> > because I have 3 gpus on my PC. I use the intel gpu on the host ; I would like to use the nvidia gtx 1060 within a bhyve Linux vm and the 2080 ti with the Linuxulator to run stable diffusion,or vice versa. Stable diffusion needs pytorch + cuda and they need the nVidia driver installed on FreeBSD and the modules loaded.<br> > <br> > My goal is to run stable diffusion and a bhyve / linux vm +my nvidia gpus attached.<br> <br> I note that <a href="https://wiki.freebsd.org/bhyve/pci_passthru" rel="noreferrer" target="_blank">https://wiki.freebsd.org/bhyve/pci_passthru</a>; says "Note: VGA / GPU pass-through devices are not currently supported" so perhaps that is why it is ignoring your pptdevs line.<br> <br> Although that said some googling suggests that is not the case so I don't know, unfortunately I haven't tried it myself.<br> <br> There are some threads about it on the forums, eg:<br> <a href="https://forums.freebsd.org/threads/bhyve-gpu-pass-through.83152/" rel="noreferrer" target="_blank">https://forums.freebsd.org/threads/bhyve-gpu-pass-through.83152/</a><br>; <br> Those suggest that modifications to bhyve are required before it will work properly but none of them are for vmm that I can see.<br> <br> --<br> Daniel O'Connor<br> "The nice thing about standards is that there<br> are so many of them to choose from."<br> -- Andrew Tanenbaum<br> <br> </blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Mario.<br></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSigVQr=B0FNvgWXHgTgSRWnc8cfo2zJf6azt2P1uLT%2BasQ>