From nobody Wed Feb 1 11:11:18 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P6K4b547xz3bw6H for ; Wed, 1 Feb 2023 11:11:59 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P6K4b1qv3z3tK1 for ; Wed, 1 Feb 2023 11:11:59 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x530.google.com with SMTP id q19so7355888edd.2 for ; Wed, 01 Feb 2023 03:11:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Hw2eJcAi3oFxvz60/qtOq6sBARuTok37Vqg/ILRkt/Y=; b=IE2j2MCIKPv2SEZzoyB5adTSSSkuIDqL7471ViT+Sn7IOzlIBlmVSV3EyQ/dVstfTm eideHbWiriBeyympubDdnL8w+KSTNvsB7IzlKPd9wjaWDKjCGo45wxlyqAuUmY0qi+LB TXc6vZb7HpAIqDFk7DVfmkFMduc0TIEyYg9jlf/yE3pJQpdTrr8IWHpurnLHySG3VFrn hh+jtrpUSengFwE64PPH4B0MNQ0otv2HiMb9ABXuQlLqI6kykfHaza4jGSiX/K6a8hPr hq98ayUU8lfR9vDQkOnXUE7KQXtX29HiBbK7qFL0GakxWyejKwi8i33EHTjrZZ7WTtwZ qq/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Hw2eJcAi3oFxvz60/qtOq6sBARuTok37Vqg/ILRkt/Y=; b=yUoGOqA7yQzwO0YbiRksKadOAcGuipTePJfPVjcrj47oWirgRGvH1GtazzHbxmKpwS Jk63jZlzHKQfrp65DYeKYujACDc0ChrKb7RAUPuyi5QcFvvdZ6Aqjyc3sTz3yQL9Smr/ lnRc2OANOEM9DTQ5LB/LGoyOBrkjeYVbP6rO8w/nglttH87gclWxlUT+tDnMdgfEdf+t A6JyPyznCTiRpuDK6FsoR2N+iUVL3X70UPzaDjJSgwAjDHQ87lKCYpHhOqOrmCikwoLa 1ka3CruuBRzRk+LpjQ43ggfPAuDHhwYUN00fN00g+dqcFwHXHqgB0qNvl29tgFkacZfv GDRg== X-Gm-Message-State: AO0yUKVDAP4DM4nlXTwzgbiyINefRqKjUOBJVKOnT0mVh0w6Lf+pjVn3 hqU+WAtF7MMGo6fw9vVaP7PpnspKLjndG3KQkbyO4xY7aMQ= X-Google-Smtp-Source: AK7set+tbCOlNQBavdo2a0dGPP+ouDFVceHo+L8e/tML8uubszai8VksjMuoNHaF9a4YIC6BpRccf4fNWpeC8nEjBL8= X-Received: by 2002:a05:6402:12ce:b0:49c:48ad:3d17 with SMTP id k14-20020a05640212ce00b0049c48ad3d17mr498564edx.17.1675249915199; Wed, 01 Feb 2023 03:11:55 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> In-Reply-To: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> From: Mario Marietto Date: Wed, 1 Feb 2023 12:11:18 +0100 Message-ID: Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy To: "Daniel O'Connor" Cc: freebsd-hackers Content-Type: multipart/alternative; boundary="0000000000001b03b505f3a184aa" X-Rspamd-Queue-Id: 4P6K4b1qv3z3tK1 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000001b03b505f3a184aa Content-Type: text/plain; charset="UTF-8" ---> Do you have 'vmm_load="YES"' in loader.conf ? yes. ---> If you want to pass through the video card (as evidenced by devctl and pptdevs) then why have the nvidia driver installed? because I have 3 gpus on my PC. I use the intel gpu on the host ; I would like to use the nvidia gtx 1060 within a bhyve Linux vm and the 2080 ti with the Linuxulator to run stable diffusion,or vice versa. Stable diffusion needs pytorch + cuda and they need the nVidia driver installed on FreeBSD and the modules loaded. My goal is to run stable diffusion and a bhyve / linux vm +my nvidia gpus attached. Thanks. Il giorno mer 1 feb 2023 alle ore 07:24 Daniel O'Connor ha scritto: > > > > On 30 Jan 2023, at 05:37, Mario Marietto wrote: > > In some FreeBSD 13.1 machines I have the problem below,in some others I > don't have it. I would like to know what the causes could be and how to fix > it. > > > > root@marietto:/usr/home/marietto # devctl detach pci0:1:0:0 > > > > devctl: Failed to detach pci0:1:0:0: Device busy > > > > root@marietto:/usr/home/marietto # devctl detach pci0:2:0:0 > > > > devctl: Failed to detach pci0:2:0:0: Device busy > > > > Not even it works if instead of detach them, I try to attach them > directly to the ppt driver : > > > > root@marietto:/usr/home/marietto/bhyve/Files # devctl set driver > pci0:2:0:0 ppt > > devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy > > > > root@marietto:/usr/home/marietto/bhyve/Files # devctl detach pci0:2:0:0 > > devctl: Failed to detach pci0:2:0:0: Device busy > > > > Pci addresses 1:0:0 and 2:0:0 belong to the two GPUs that I have on my > PC : > > > > vgapci0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1c02 subvendor=0x19da subdevice=0x2438 > > vgapci1@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1e04 subvendor=0x19da subdevice=0x2503 > > > > Actually I have commented this line on /boot/loader.conf,because it > makes no difference if I keep it uncommented or not. It is totally ignored. > > > > #pptdevs="1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3" > > Do you have 'vmm_load="YES"' in loader.conf? > > If not I am not sure it will have an effect - vmm must be loaded early > otherwise the card will be grabbed by the vgapci driver rather than be put > under ppt control. > > > In every FreeBSD machine I load the nvidia kernel modules adding the > line below to /etc/rc.conf : > > > > kld_list="nvidia nvidia-modeset" > > > > and I have installed the nvidia-driver package on every machine. But as > I said,in some of them I see the error above,in some others I don't see it. > I'm not able to isolate the dysfunctional pattern. The xorg.conf file is > the same for each machine. Can someone help me to troubleshoot the error ? > thanks. > > If you want to pass through the video card (as evidenced by devctl and > pptdevs) then why have the nvidia driver installed? > > Perhaps explain what you are trying to achieve rather than asking for help > on how to sold what you think the problem is. > > PS I have trimmed the CCs as that seems impolite. > > -- > Daniel O'Connor > "The nice thing about standards is that there > are so many of them to choose from." > -- Andrew Tanenbaum > > -- Mario. --0000000000001b03b505f3a184aa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
---> Do you have 'vmm_load=3D= "YES"' in loader.conf ?

yes.

---> If you want to pass through the video card (as evidenced by = devctl and pptdevs) then why have the nvidia driver installed?
because I have 3 gpus on my PC. I use the intel gpu on the hos= t ; I would like to use the nvidia gtx 1060 within a bhyve Linux vm and the= 2080 ti with the Linuxulator to run stable diffusion,or vice versa. Stable= diffusion needs pytorch + cuda and they need the nVidia driver installed o= n FreeBSD and the modules loaded.

My goal is t= o run stable diffusion and a bhyve / linux vm +my nvidia gpus attached.
=

Thanks.

Il giorno mer 1 feb 2023 alle o= re 07:24 Daniel O'Connor <darius@dons.net.au> ha scritto:


> On 30 Jan 2023, at 05:37, Mario Marietto <marietto2008@gmail.com> wrote: > In some FreeBSD 13.1 machines I have the problem below,in some others = I don't have it. I would like to know what the causes could be and how = to fix it.
>
> root@marietto:/usr/home/marietto # devctl detach pci0:1:0:0
>
> devctl: Failed to detach pci0:1:0:0: Device busy
>
> root@marietto:/usr/home/marietto # devctl detach pci0:2:0:0
>
> devctl: Failed to detach pci0:2:0:0: Device busy
>
> Not even it works if instead of detach them, I try to attach them dire= ctly to the ppt driver :
>
> root@marietto:/usr/home/marietto/bhyve/Files # devctl set driver pci0:= 2:0:0 ppt
> devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy
>=C2=A0
> root@marietto:/usr/home/marietto/bhyve/Files # devctl detach pci0:2:0:= 0
> devctl: Failed to detach pci0:2:0:0: Device busy
>
> Pci addresses 1:0:0 and 2:0:0 belong to the two GPUs that I have on my= PC :
>
> vgapci0@pci0:1:0:0:=C2=A0 =C2=A0 =C2=A0class=3D0x030000 rev=3D0xa1 hdr= =3D0x00 vendor=3D0x10de device=3D0x1c02 subvendor=3D0x19da subdevice=3D0x24= 38
> vgapci1@pci0:2:0:0:=C2=A0 =C2=A0 =C2=A0class=3D0x030000 rev=3D0xa1 hdr= =3D0x00 vendor=3D0x10de device=3D0x1e04 subvendor=3D0x19da subdevice=3D0x25= 03
>
> Actually I have commented this line on /boot/loader.conf,because it ma= kes no difference if I keep it uncommented or not. It is totally ignored. >
> #pptdevs=3D"1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3"

Do you have 'vmm_load=3D"YES"' in loader.conf?

If not I am not sure it will have an effect - vmm must be loaded early othe= rwise the card will be grabbed by the vgapci driver rather than be put unde= r ppt control.

> In every FreeBSD machine I load the nvidia kernel modules adding the l= ine below to /etc/rc.conf :
>
> kld_list=3D"nvidia nvidia-modeset"
>
> and I have installed the nvidia-driver package on every machine. But a= s I said,in some of them I see the error above,in some others I don't s= ee it. I'm not able to isolate the dysfunctional pattern. The xorg.conf= file is the same for each machine. Can someone help me to troubleshoot the= error ? thanks.

If you want to pass through the video card (as evidenced by devctl and pptd= evs) then why have the nvidia driver installed?

Perhaps explain what you are trying to achieve rather than asking for help = on how to sold what you think the problem is.

PS I have trimmed the CCs as that seems impolite.

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
-- Andrew Tanenbaum



--
Mario.
=
--0000000000001b03b505f3a184aa--