From nobody Sun Jan 29 19:07:53 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P4gns3vTGz3bq0c; Sun, 29 Jan 2023 19:08:33 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x531.google.com (mail-ed1-x531.google.com [IPv6:2a00:1450:4864:20::531]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P4gnr6Kxjz3D10; Sun, 29 Jan 2023 19:08:32 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=ZLYkTJ3Z; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::531 as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x531.google.com with SMTP id f7so1832165edw.5; Sun, 29 Jan 2023 11:08:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=KcqQ8Jt4znegl3Pgn4KcLIBxsBjh//Jdt6b7lDvi3dg=; b=ZLYkTJ3ZTbRDzA6dzYIDZHqYYrMSKiBATedmtgTRBG+6h8bFCTujzBss52sJB3VMb/ YFkJOv2N/b6YxoxWUWK0DTERODkWRVdFHcQwxZZmYZWeY+p1ae3W4+ZOBg8koWAYt6V0 NFUkfuJlNzjIuXNwUtTPUSw2/n6Yilq1F8WrmKLSky4jKVAwEwok9kEQ18m4+v2Jdnx5 +olNk8+VT0ZaHagBAbpiOzRuVgTZ12uA2YGiyXaQNQyPcoPtz7ULZ/VERWvpEmPU/ePu HQC0owykzn8NoOgSXxOQP5alWgxouW6OwR52M7aCFU/zvVRDLV9VWFIjJcVldeyL4byE nBIQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=KcqQ8Jt4znegl3Pgn4KcLIBxsBjh//Jdt6b7lDvi3dg=; b=TOMJFZ4IYmnmaRnuu5ol06lF3Dyu4fpYgvQuMl8AaD3pT1bjmgmxgxKYNN0/BXNQYl CWTWY2GUURoMwpN88gpUC2NQUGczb0OR1oLbaAPh7KZmc11pgVxSUqHZbD2qkNqqLNpd r7aDjhiniUUWiAt68iVQgqrj7/CRv0Vkx28TTSYbZfxud1BxBjytdc3wDT94bs3/OOFF O/7s8AZDJLpPKQfpax1FJcJ7svB7gSB3hvz2Bdb9yALicty8OHpiutfMJsiQUwd6zpAD sazS8revMlXgMvMPgs+XIexv4+kP7kOv8//2qgpQxEEA0fECH4nTDwUb88aUwEmUwqBo Y+Xw== X-Gm-Message-State: AO0yUKWvpf0HrWxItPPX9fXSOyUxHNfSfiPLkUKHdE/AfBORpqvPOMfi 0i2hAhv5WlY6YJgxEHSwYQc12YfbepLck469HLxI1Vh75lTBZL7D X-Google-Smtp-Source: AK7set/rznT/hPeJS7FZQg1zfdfSvark7DJ+80rVLtwmf4IbpvPnKvXe7gIpD5VfeRJKWEAfWF17LAgdDztXAWWHdmY= X-Received: by 2002:aa7:cd85:0:b0:4a2:ea2:4f38 with SMTP id x5-20020aa7cd85000000b004a20ea24f38mr2341407edv.20.1675019310471; Sun, 29 Jan 2023 11:08:30 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 From: Mario Marietto Date: Sun, 29 Jan 2023 20:07:53 +0100 Message-ID: Subject: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy To: FreeBSD virtualization , freebsd-hackers@freebsd.org, =?UTF-8?Q?Corvin_K=C3=B6hne?= , Dustin Marquess , Alexander Eichner , theraven@freebsd.org Content-Type: multipart/alternative; boundary="000000000000fe1faf05f36bd226" X-Spamd-Result: default: False [-3.99 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; NEURAL_HAM_MEDIUM(-0.99)[-0.986]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::531:from]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-hackers@freebsd.org]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_TO(0.00)[freebsd.org,beckhoff.com,gmail.com,aeichner.de]; FROM_EQ_ENVFROM(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; FROM_HAS_DN(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_FIVE(0.00)[6]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4P4gnr6Kxjz3D10 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --000000000000fe1faf05f36bd226 Content-Type: text/plain; charset="UTF-8" Hello to everyone. In some FreeBSD 13.1 machines I have the problem below,in some others I don't have it. I would like to know what the causes could be and how to fix it. root@marietto:/usr/home/marietto # devctl detach pci0:1:0:0 devctl: Failed to detach pci0:1:0:0: Device busy root@marietto:/usr/home/marietto # devctl detach pci0:2:0:0 devctl: Failed to detach pci0:2:0:0: Device busy Not even it works if instead of detach them, I try to attach them directly to the ppt driver : root@marietto:/usr/home/marietto/bhyve/Files # devctl set driver pci0:2:0:0 ppt devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy root@marietto:/usr/home/marietto/bhyve/Files # devctl detach pci0:2:0:0 devctl: Failed to detach pci0:2:0:0: Device busy Pci addresses 1:0:0 and 2:0:0 belong to the two GPUs that I have on my PC : vgapci0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1c02 subvendor=0x19da subdevice=0x2438 vgapci1@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e04 subvendor=0x19da subdevice=0x2503 Actually I have commented this line on /boot/loader.conf,because it makes no difference if I keep it uncommented or not. It is totally ignored. #pptdevs="1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3" In every FreeBSD machine I load the nvidia kernel modules adding the line below to /etc/rc.conf : kld_list="nvidia nvidia-modeset" and I have installed the nvidia-driver package on every machine. But as I said,in some of them I see the error above,in some others I don't see it. I'm not able to isolate the dysfunctional pattern. The xorg.conf file is the same for each machine. Can someone help me to troubleshoot the error ? thanks. -- Mario. --000000000000fe1faf05f36bd226 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello to ev= eryone.
<= br>
In some Fr= eeBSD 13.1 machines I have the problem below,in some others I don't hav= e it. I would like to know what the causes could be and how to fix it.
=

<= /div>
root@marietto:/usr/home/mari= etto # devctl detach pci0:1:0:0

devctl: Failed = to detach pci0:1:0:0: Device busy

root@marietto:= /usr/home/marietto # devctl detach pci0:2:0:0

<= /span>
devctl: Failed to detach pci0:2:0:0: Devi= ce busy
=
<= span style=3D"font-family:monospace">Not even it works if instead of detach them, I try to attach them di= rectly to the ppt driver :

root@marietto:/usr/home/marietto/b= hyve/Files # devctl set driver pci0:2:0:0 ppt
devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy
=C2=A0
root@marietto:/usr/home/marietto/bhyve/Files # devct= l detach pci0:2:0:0
devctl: Failed to detach pci0:2:0:0: Device busy

Pci addresses 1:0:0 a= nd 2:0:0 belong to the two GPUs that I have on my PC :

vgapci0@pci0:1:0:0: =C2=A0=C2=A0=C2=A0=C2=A0class=3D0x030000 rev= =3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1c02 subvendor=3D0x19da subde= vice=3D0x2438
vgapci1@pci0:2:0:= 0: =C2=A0=C2=A0=C2=A0=C2=A0class=3D0x030000 rev=3D0xa1 hdr=3D0x00 vendor=3D= 0x10de device=3D0x1e04 subvendor=3D0x19da subdevice=3D0x2503

Actually I have commented this line on /boo= t/loader.conf,because it makes no difference if I keep it uncommented or no= t. It is totally ignored.

#pptdevs=3D"1/0/0 1/0/1 2/0/0 2/0/1 2/= 0/2 2/0/3"

In every FreeBSD machine I load the nvidia kernel modules addi= ng the line below to /etc/rc.conf :

=
kld_list=3D"nvidia nvidia-modeset"= ;

and I have installed the nvidia-driver package on every = machine. But as I said,in some of them I see the error above,in some others= I don't see it. I'm not able to isolate the dysfunctional pattern. The xorg.con= f file is the same for each machine. Can someone help me to troubleshoot th= e error ? thanks.

--
Mario.
--000000000000fe1faf05f36bd226-- From nobody Wed Feb 1 06:23:29 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P6BhT53Z1z3cHQK for ; Wed, 1 Feb 2023 06:24:09 +0000 (UTC) (envelope-from darius@dons.net.au) Received: from midget.dons.net.au (2403-5800-5200-4700-225-90ff-fe47-39b4.ip6.aussiebb.net [IPv6:2403:5800:5200:4700:225:90ff:fe47:39b4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "dons.net.au", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P6BhR38BXz4kq8 for ; Wed, 1 Feb 2023 06:24:06 +0000 (UTC) (envelope-from darius@dons.net.au) Authentication-Results: mx1.freebsd.org; dkim=fail ("headers rsa verify failed") header.d=dons.net.au header.s=default header.b=JnUekN8G; spf=pass (mx1.freebsd.org: domain of darius@dons.net.au designates 2403:5800:5200:4700:225:90ff:fe47:39b4 as permitted sender) smtp.mailfrom=darius@dons.net.au; dmarc=pass (policy=quarantine) header.from=dons.net.au Received: from smtpclient.apple ([IPv6:2001:44b8:1d2:8900:dc56:cd7b:feba:3c2b]) (authenticated bits=0) by midget.dons.net.au (8.17.1/8.15.2) with ESMTPSA id 3116NjL6036853 (version=TLSv1.2 cipher=ECDHE-ECDSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 1 Feb 2023 16:53:51 +1030 (ACDT) (envelope-from darius@dons.net.au) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=dons.net.au; s=default; t=1675232631; bh=fSIjpz9CpV1AAcZNAbOiWCpZilGviztX/Gx+OUPNx6c=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=JnUekN8GJpT4ta5NfHV4hgVFB1X4YQnTsFX3cm8xMUv75YOIuibw46syWOAvFTU15 xfVilz0LKK+NRxNZmMR1joyEncEAI3OlVeynajAS/2IJtw1CRiPJ5J1tTt2NCsulEe CkLjmNeVgqj41NH3pfKOpCj/T4aYK+Jm9hgqf03E= X-Authentication-Warning: midget.dons.net.au: Host [IPv6:2001:44b8:1d2:8900:dc56:cd7b:feba:3c2b] claimed to be smtpclient.apple Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy From: "Daniel O'Connor" In-Reply-To: Date: Wed, 1 Feb 2023 16:53:29 +1030 Cc: freebsd-hackers Content-Transfer-Encoding: quoted-printable Message-Id: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> References: To: Mario Marietto X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spam-Status: No, score=0.40 X-Rspamd-Server: midget.dons.net.au X-Spamd-Result: default: False [-3.30 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.998]; MV_CASE(0.50)[]; DMARC_POLICY_ALLOW_WITH_FAILURES(-0.50)[]; R_SPF_ALLOW(-0.20)[+mx]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_VIA_SMTP_AUTH(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; DMARC_POLICY_ALLOW(0.00)[dons.net.au,quarantine]; ASN(0.00)[asn:4764, ipnet:2403:5800::/32, country:AU]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_TWO(0.00)[2]; TO_DN_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; DKIM_TRACE(0.00)[dons.net.au:-]; FROM_HAS_DN(0.00)[]; R_DKIM_REJECT(0.00)[dons.net.au:s=default]; HAS_XAW(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[] X-Rspamd-Queue-Id: 4P6BhR38BXz4kq8 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N > On 30 Jan 2023, at 05:37, Mario Marietto = wrote: > In some FreeBSD 13.1 machines I have the problem below,in some others = I don't have it. I would like to know what the causes could be and how = to fix it.=20 >=20 > root@marietto:/usr/home/marietto # devctl detach pci0:1:0:0=20 >=20 > devctl: Failed to detach pci0:1:0:0: Device busy >=20 > root@marietto:/usr/home/marietto # devctl detach pci0:2:0:0=20 >=20 > devctl: Failed to detach pci0:2:0:0: Device busy=20 >=20 > Not even it works if instead of detach them, I try to attach them = directly to the ppt driver : >=20 > root@marietto:/usr/home/marietto/bhyve/Files # devctl set driver = pci0:2:0:0 ppt=20 > devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy > =20 > root@marietto:/usr/home/marietto/bhyve/Files # devctl detach = pci0:2:0:0=20 > devctl: Failed to detach pci0:2:0:0: Device busy >=20 > Pci addresses 1:0:0 and 2:0:0 belong to the two GPUs that I have on my = PC : >=20 > vgapci0@pci0:1:0:0: class=3D0x030000 rev=3D0xa1 hdr=3D0x00 = vendor=3D0x10de device=3D0x1c02 subvendor=3D0x19da subdevice=3D0x2438=20 > vgapci1@pci0:2:0:0: class=3D0x030000 rev=3D0xa1 hdr=3D0x00 = vendor=3D0x10de device=3D0x1e04 subvendor=3D0x19da subdevice=3D0x2503 >=20 > Actually I have commented this line on /boot/loader.conf,because it = makes no difference if I keep it uncommented or not. It is totally = ignored. >=20 > #pptdevs=3D"1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3"=20 Do you have 'vmm_load=3D"YES"' in loader.conf? If not I am not sure it will have an effect - vmm must be loaded early = otherwise the card will be grabbed by the vgapci driver rather than be = put under ppt control. > In every FreeBSD machine I load the nvidia kernel modules adding the = line below to /etc/rc.conf : >=20 > kld_list=3D"nvidia nvidia-modeset" >=20 > and I have installed the nvidia-driver package on every machine. But = as I said,in some of them I see the error above,in some others I don't = see it. I'm not able to isolate the dysfunctional pattern. The xorg.conf = file is the same for each machine. Can someone help me to troubleshoot = the error ? thanks.=20 If you want to pass through the video card (as evidenced by devctl and = pptdevs) then why have the nvidia driver installed? Perhaps explain what you are trying to achieve rather than asking for = help on how to sold what you think the problem is. PS I have trimmed the CCs as that seems impolite. -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum From nobody Wed Feb 1 11:11:18 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P6K4b547xz3bw6H for ; Wed, 1 Feb 2023 11:11:59 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x530.google.com (mail-ed1-x530.google.com [IPv6:2a00:1450:4864:20::530]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P6K4b1qv3z3tK1 for ; Wed, 1 Feb 2023 11:11:59 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ed1-x530.google.com with SMTP id q19so7355888edd.2 for ; Wed, 01 Feb 2023 03:11:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Hw2eJcAi3oFxvz60/qtOq6sBARuTok37Vqg/ILRkt/Y=; b=IE2j2MCIKPv2SEZzoyB5adTSSSkuIDqL7471ViT+Sn7IOzlIBlmVSV3EyQ/dVstfTm eideHbWiriBeyympubDdnL8w+KSTNvsB7IzlKPd9wjaWDKjCGo45wxlyqAuUmY0qi+LB TXc6vZb7HpAIqDFk7DVfmkFMduc0TIEyYg9jlf/yE3pJQpdTrr8IWHpurnLHySG3VFrn hh+jtrpUSengFwE64PPH4B0MNQ0otv2HiMb9ABXuQlLqI6kykfHaza4jGSiX/K6a8hPr hq98ayUU8lfR9vDQkOnXUE7KQXtX29HiBbK7qFL0GakxWyejKwi8i33EHTjrZZ7WTtwZ qq/A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=Hw2eJcAi3oFxvz60/qtOq6sBARuTok37Vqg/ILRkt/Y=; b=yUoGOqA7yQzwO0YbiRksKadOAcGuipTePJfPVjcrj47oWirgRGvH1GtazzHbxmKpwS Jk63jZlzHKQfrp65DYeKYujACDc0ChrKb7RAUPuyi5QcFvvdZ6Aqjyc3sTz3yQL9Smr/ lnRc2OANOEM9DTQ5LB/LGoyOBrkjeYVbP6rO8w/nglttH87gclWxlUT+tDnMdgfEdf+t A6JyPyznCTiRpuDK6FsoR2N+iUVL3X70UPzaDjJSgwAjDHQ87lKCYpHhOqOrmCikwoLa 1ka3CruuBRzRk+LpjQ43ggfPAuDHhwYUN00fN00g+dqcFwHXHqgB0qNvl29tgFkacZfv GDRg== X-Gm-Message-State: AO0yUKVDAP4DM4nlXTwzgbiyINefRqKjUOBJVKOnT0mVh0w6Lf+pjVn3 hqU+WAtF7MMGo6fw9vVaP7PpnspKLjndG3KQkbyO4xY7aMQ= X-Google-Smtp-Source: AK7set+tbCOlNQBavdo2a0dGPP+ouDFVceHo+L8e/tML8uubszai8VksjMuoNHaF9a4YIC6BpRccf4fNWpeC8nEjBL8= X-Received: by 2002:a05:6402:12ce:b0:49c:48ad:3d17 with SMTP id k14-20020a05640212ce00b0049c48ad3d17mr498564edx.17.1675249915199; Wed, 01 Feb 2023 03:11:55 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> In-Reply-To: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> From: Mario Marietto Date: Wed, 1 Feb 2023 12:11:18 +0100 Message-ID: Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy To: "Daniel O'Connor" Cc: freebsd-hackers Content-Type: multipart/alternative; boundary="0000000000001b03b505f3a184aa" X-Rspamd-Queue-Id: 4P6K4b1qv3z3tK1 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000001b03b505f3a184aa Content-Type: text/plain; charset="UTF-8" ---> Do you have 'vmm_load="YES"' in loader.conf ? yes. ---> If you want to pass through the video card (as evidenced by devctl and pptdevs) then why have the nvidia driver installed? because I have 3 gpus on my PC. I use the intel gpu on the host ; I would like to use the nvidia gtx 1060 within a bhyve Linux vm and the 2080 ti with the Linuxulator to run stable diffusion,or vice versa. Stable diffusion needs pytorch + cuda and they need the nVidia driver installed on FreeBSD and the modules loaded. My goal is to run stable diffusion and a bhyve / linux vm +my nvidia gpus attached. Thanks. Il giorno mer 1 feb 2023 alle ore 07:24 Daniel O'Connor ha scritto: > > > > On 30 Jan 2023, at 05:37, Mario Marietto wrote: > > In some FreeBSD 13.1 machines I have the problem below,in some others I > don't have it. I would like to know what the causes could be and how to fix > it. > > > > root@marietto:/usr/home/marietto # devctl detach pci0:1:0:0 > > > > devctl: Failed to detach pci0:1:0:0: Device busy > > > > root@marietto:/usr/home/marietto # devctl detach pci0:2:0:0 > > > > devctl: Failed to detach pci0:2:0:0: Device busy > > > > Not even it works if instead of detach them, I try to attach them > directly to the ppt driver : > > > > root@marietto:/usr/home/marietto/bhyve/Files # devctl set driver > pci0:2:0:0 ppt > > devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy > > > > root@marietto:/usr/home/marietto/bhyve/Files # devctl detach pci0:2:0:0 > > devctl: Failed to detach pci0:2:0:0: Device busy > > > > Pci addresses 1:0:0 and 2:0:0 belong to the two GPUs that I have on my > PC : > > > > vgapci0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1c02 subvendor=0x19da subdevice=0x2438 > > vgapci1@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1e04 subvendor=0x19da subdevice=0x2503 > > > > Actually I have commented this line on /boot/loader.conf,because it > makes no difference if I keep it uncommented or not. It is totally ignored. > > > > #pptdevs="1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3" > > Do you have 'vmm_load="YES"' in loader.conf? > > If not I am not sure it will have an effect - vmm must be loaded early > otherwise the card will be grabbed by the vgapci driver rather than be put > under ppt control. > > > In every FreeBSD machine I load the nvidia kernel modules adding the > line below to /etc/rc.conf : > > > > kld_list="nvidia nvidia-modeset" > > > > and I have installed the nvidia-driver package on every machine. But as > I said,in some of them I see the error above,in some others I don't see it. > I'm not able to isolate the dysfunctional pattern. The xorg.conf file is > the same for each machine. Can someone help me to troubleshoot the error ? > thanks. > > If you want to pass through the video card (as evidenced by devctl and > pptdevs) then why have the nvidia driver installed? > > Perhaps explain what you are trying to achieve rather than asking for help > on how to sold what you think the problem is. > > PS I have trimmed the CCs as that seems impolite. > > -- > Daniel O'Connor > "The nice thing about standards is that there > are so many of them to choose from." > -- Andrew Tanenbaum > > -- Mario. --0000000000001b03b505f3a184aa Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
---> Do you have 'vmm_load=3D= "YES"' in loader.conf ?

yes.

---> If you want to pass through the video card (as evidenced by = devctl and pptdevs) then why have the nvidia driver installed?
because I have 3 gpus on my PC. I use the intel gpu on the hos= t ; I would like to use the nvidia gtx 1060 within a bhyve Linux vm and the= 2080 ti with the Linuxulator to run stable diffusion,or vice versa. Stable= diffusion needs pytorch + cuda and they need the nVidia driver installed o= n FreeBSD and the modules loaded.

My goal is t= o run stable diffusion and a bhyve / linux vm +my nvidia gpus attached.
=

Thanks.

Il giorno mer 1 feb 2023 alle o= re 07:24 Daniel O'Connor <darius@dons.net.au> ha scritto:


> On 30 Jan 2023, at 05:37, Mario Marietto <marietto2008@gmail.com> wrote: > In some FreeBSD 13.1 machines I have the problem below,in some others = I don't have it. I would like to know what the causes could be and how = to fix it.
>
> root@marietto:/usr/home/marietto # devctl detach pci0:1:0:0
>
> devctl: Failed to detach pci0:1:0:0: Device busy
>
> root@marietto:/usr/home/marietto # devctl detach pci0:2:0:0
>
> devctl: Failed to detach pci0:2:0:0: Device busy
>
> Not even it works if instead of detach them, I try to attach them dire= ctly to the ppt driver :
>
> root@marietto:/usr/home/marietto/bhyve/Files # devctl set driver pci0:= 2:0:0 ppt
> devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy
>=C2=A0
> root@marietto:/usr/home/marietto/bhyve/Files # devctl detach pci0:2:0:= 0
> devctl: Failed to detach pci0:2:0:0: Device busy
>
> Pci addresses 1:0:0 and 2:0:0 belong to the two GPUs that I have on my= PC :
>
> vgapci0@pci0:1:0:0:=C2=A0 =C2=A0 =C2=A0class=3D0x030000 rev=3D0xa1 hdr= =3D0x00 vendor=3D0x10de device=3D0x1c02 subvendor=3D0x19da subdevice=3D0x24= 38
> vgapci1@pci0:2:0:0:=C2=A0 =C2=A0 =C2=A0class=3D0x030000 rev=3D0xa1 hdr= =3D0x00 vendor=3D0x10de device=3D0x1e04 subvendor=3D0x19da subdevice=3D0x25= 03
>
> Actually I have commented this line on /boot/loader.conf,because it ma= kes no difference if I keep it uncommented or not. It is totally ignored. >
> #pptdevs=3D"1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3"

Do you have 'vmm_load=3D"YES"' in loader.conf?

If not I am not sure it will have an effect - vmm must be loaded early othe= rwise the card will be grabbed by the vgapci driver rather than be put unde= r ppt control.

> In every FreeBSD machine I load the nvidia kernel modules adding the l= ine below to /etc/rc.conf :
>
> kld_list=3D"nvidia nvidia-modeset"
>
> and I have installed the nvidia-driver package on every machine. But a= s I said,in some of them I see the error above,in some others I don't s= ee it. I'm not able to isolate the dysfunctional pattern. The xorg.conf= file is the same for each machine. Can someone help me to troubleshoot the= error ? thanks.

If you want to pass through the video card (as evidenced by devctl and pptd= evs) then why have the nvidia driver installed?

Perhaps explain what you are trying to achieve rather than asking for help = on how to sold what you think the problem is.

PS I have trimmed the CCs as that seems impolite.

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
-- Andrew Tanenbaum



--
Mario.
=
--0000000000001b03b505f3a184aa-- From nobody Wed Feb 1 12:48:58 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P6MDw3kGWz3c7vK for ; Wed, 1 Feb 2023 12:49:20 +0000 (UTC) (envelope-from darius@dons.net.au) Received: from midget.dons.net.au (2403-5800-5200-4700-225-90ff-fe47-39b4.ip6.aussiebb.net [IPv6:2403:5800:5200:4700:225:90ff:fe47:39b4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "dons.net.au", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P6MDv5fpXz46sm for ; Wed, 1 Feb 2023 12:49:19 +0000 (UTC) (envelope-from darius@dons.net.au) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (2403-5800-5200-4700-74ab-27d6-db94-a375.ip6.aussiebb.net [IPv6:2403:5800:5200:4700:74ab:27d6:db94:a375]) (authenticated bits=0) by midget.dons.net.au (8.17.1/8.15.2) with ESMTPSA id 311Cn9Wi051804 (version=TLSv1.2 cipher=ECDHE-ECDSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 1 Feb 2023 23:19:09 +1030 (ACDT) (envelope-from darius@dons.net.au) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=dons.net.au; s=default; t=1675255749; bh=rOE7948BtWytbyWOEfSYaXkoWD8lu3te/wgaifMsVPY=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=A8otQApg43oV0ObsIwRqH1PPngYBkFlExzFh/uwF4PcY8D0wij0qe2H5Ph420DFuG uK2c78KeAtNwBep3lPdle2PW9wMF6nAlm6q1nqSTB8D40krTjMSMPdV21FjnD3S3kh W5W70FpBiUDVGa9uU6iHFxSTilPlq1qpkEgz1NWw= X-Authentication-Warning: midget.dons.net.au: Host 2403-5800-5200-4700-74ab-27d6-db94-a375.ip6.aussiebb.net [IPv6:2403:5800:5200:4700:74ab:27d6:db94:a375] claimed to be smtpclient.apple Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy From: "Daniel O'Connor" In-Reply-To: Date: Wed, 1 Feb 2023 23:18:58 +1030 Cc: freebsd-hackers Content-Transfer-Encoding: quoted-printable Message-Id: <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au> References: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> To: Mario Marietto X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spam-Status: No, score=0.40 X-Rspamd-Server: midget.dons.net.au X-Rspamd-Queue-Id: 4P6MDv5fpXz46sm X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:4764, ipnet:2403:5800::/32, country:AU] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On 1 Feb 2023, at 21:41, Mario Marietto = wrote: > ---> Do you have 'vmm_load=3D"YES"' in loader.conf ? >=20 > yes. >=20 > ---> If you want to pass through the video card (as evidenced by = devctl and pptdevs) then why have the nvidia driver installed? >=20 > because I have 3 gpus on my PC. I use the intel gpu on the host ; I = would like to use the nvidia gtx 1060 within a bhyve Linux vm and the = 2080 ti with the Linuxulator to run stable diffusion,or vice versa. = Stable diffusion needs pytorch + cuda and they need the nVidia driver = installed on FreeBSD and the modules loaded. >=20 > My goal is to run stable diffusion and a bhyve / linux vm +my nvidia = gpus attached. I note that https://wiki.freebsd.org/bhyve/pci_passthru says "Note: VGA = / GPU pass-through devices are not currently supported" so perhaps that = is why it is ignoring your pptdevs line. Although that said some googling suggests that is not the case so I = don't know, unfortunately I haven't tried it myself. There are some threads about it on the forums, eg: https://forums.freebsd.org/threads/bhyve-gpu-pass-through.83152/ Those suggest that modifications to bhyve are required before it will = work properly but none of them are for vmm that I can see. -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum From nobody Thu Feb 2 14:28:48 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P71PS3xlPz3fHZH for ; Thu, 2 Feb 2023 14:29:00 +0000 (UTC) (envelope-from jbo@insane.engineer) Received: from mail-4317.proton.ch (mail-4317.proton.ch [185.70.43.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P71PQ2hFtz4Y2H for ; Thu, 2 Feb 2023 14:28:58 +0000 (UTC) (envelope-from jbo@insane.engineer) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=insane.engineer header.s=protonmail3 header.b=RJyPrKYC; spf=pass (mx1.freebsd.org: domain of jbo@insane.engineer designates 185.70.43.17 as permitted sender) smtp.mailfrom=jbo@insane.engineer; dmarc=pass (policy=none) header.from=insane.engineer Date: Thu, 02 Feb 2023 14:28:48 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=insane.engineer; s=protonmail3; t=1675348135; x=1675607335; bh=JP6l5QOg7bLtGnszhRpsEuoimHG4o3HbLhBHcroScjs=; h=Date:To:From:Subject:Message-ID:Feedback-ID:From:To:Cc:Date: Subject:Reply-To:Feedback-ID:Message-ID:BIMI-Selector; b=RJyPrKYCwRrzemRveKxuOM+OW+qL1D3ReoLWuw2p67MqXK+iWCl6JNZdE/fu1nDpT SrfNxQ/XxPsB0Jm0Kpb2089NOx5/e+ctuaKVeBJUdSaE2U9mNRP3y5YsbJj/zoGp2E nJhdCARotwckSlwBiFqi4X8XbyeqLmK/nq3+sFR6FLY2ja7jUi8GOmZRzzqNqsLp+z vn8hbwu0AYLznhfWn5wtiMEFJwQkCShcNs06zEGt+7QUxt9SVkwOYfaRysaWH6RX9f enkjWBBKwyHMWl73nEnsHOjfT+UM7LDDkuuaLXrneE4qCrA7/wMmNyk+eCm8VgzZjP 07dyIJwZymIeg== To: "hackers@freebsd.org" From: jbo@insane.engineer Subject: Swap, ZFS & ARC Message-ID: Feedback-ID: 40997969:user:proton List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="b1_hsD2aARXOhg4sFp8EdgWkAv1ymA04gVBAXGHHZAi8" X-Spamd-Result: default: False [-2.48 / 15.00]; MIME_BASE64_TEXT_BOGUS(1.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-0.98)[-0.980]; NEURAL_HAM_SHORT(-0.60)[-0.601]; DMARC_POLICY_ALLOW(-0.50)[insane.engineer,none]; R_DKIM_ALLOW(-0.20)[insane.engineer:s=protonmail3]; R_SPF_ALLOW(-0.20)[+ip4:185.70.43.0/24]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MIME_BASE64_TEXT(0.10)[]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[hackers@freebsd.org]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:62371, ipnet:185.70.43.0/24, country:CH]; FROM_EQ_ENVFROM(0.00)[]; TO_DN_EQ_ADDR_ALL(0.00)[]; DKIM_TRACE(0.00)[insane.engineer:+]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_NO_DN(0.00)[]; HAS_PHPMAILER_SIG(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; RWL_MAILSPIKE_POSSIBLE(0.00)[185.70.43.17:from] X-Rspamd-Queue-Id: 4P71PQ2hFtz4Y2H X-Spamd-Bar: -- X-ThisMailContainsUnwantedMimeParts: N This is a multi-part message in MIME format. --b1_hsD2aARXOhg4sFp8EdgWkAv1ymA04gVBAXGHHZAi8 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: base64 SGVsbG8gZm9sa3MsCgpCYXNlZCBvbiBhIGRpc2N1c3Npb24gb24gdGhlIGZvcnVtcyBub3Qgc28g bG9uZyBhZ28gSSB0cmllZCB0byBmaWd1cmUgb3V0IGhvdyBzd2FwIHVzYWdlIG9uIGEgWkZTIHN5 c3RlbSBwbGF5cyB0b2dldGhlciB3aXRoIEFSQy4gSG93ZXZlciwgSSBjb3VsZCBmaW5kIHZlcnkg bGl0dGxlIHRvIG5vIGluZm9ybWF0aW9uIG9uIHRoaXMgd2hpY2ggbGVhZHMgbWUgdG8gYmVsaWV2 ZSB0aGF0IHRoZXJlIGlzIHNvbWUgImNvcmUgY29uY2VwdCIgSSBtaWdodCBiZSBvYmxpdmlvdXMg dG8uCgpUaGUgbWFpbiBxdWVzdGlvbiBpcyBiYXNpY2FsbHkgdGhpczogWW91ciBzeXN0ZW0gc3Rh cnRzIHRvIHN3YXAgb3V0IGRhdGEgZnJvbSBSQU0gdG8geW91ciBzd2FwIHBhcnRpdGlvbi4gVGhp cyBzd2FwIGRhdGEgb24gZGlzayB1bHRpbWF0ZWx5IHJlc2lkZXMgc29tZXdoZXJlIGluIGEgWkZT IHBvb2wuIElmIHRoaXMgZGF0YSB0aGVuIGdldHMgYWNjZXNzZWQsIGl0IG1pZ2h0IGJlIGNhY2hl ZCBieSBBUkMgZXNzZW50aWFsbHkgZWF0aW5nIHVwIG1lbW9yeSBhZ2FpbiB3aGljaCBzZWVtcyBj b3VudGVyIHByb2R1Y3RpdmUuCklzIHRoZXJlIGFueSBtYWdpYyB3aGljaCBwcmV2ZW50cyBzd2Fw IHBhcnRpdGlvbnMgZnJvbSBiZWluZyBsb2FkZWQgaW50byBBUkM/IE9yIGlzIHRoaXMgYSBub24t aXNzdWUgZm9yIHNvbWUgb3RoZXIgcmVhc29uPwoKQmVzdCByZWdhcmRzLAp+IGpvZWw= --b1_hsD2aARXOhg4sFp8EdgWkAv1ymA04gVBAXGHHZAi8 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: base64 PGRpdiBzdHlsZT0iZm9udC1mYW1pbHk6IEFyaWFsOyBmb250LXNpemU6IDE0cHg7Ij5IZWxsbyBm b2xrcyw8L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTogQXJpYWw7IGZvbnQtc2l6ZTogMTRw eDsiPjxicj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LWZhbWlseTogQXJpYWw7IGZvbnQtc2l6ZTog MTRweDsiPkJhc2VkIG9uIGEgZGlzY3Vzc2lvbiBvbiB0aGUgZm9ydW1zIG5vdCBzbyBsb25nIGFn byBJIHRyaWVkIHRvIGZpZ3VyZSBvdXQgaG93IHN3YXAgdXNhZ2Ugb24gYSBaRlMgc3lzdGVtIHBs YXlzIHRvZ2V0aGVyIHdpdGggQVJDLiBIb3dldmVyLCBJIGNvdWxkIGZpbmQgdmVyeSBsaXR0bGUg dG8gbm8gaW5mb3JtYXRpb24gb24gdGhpcyB3aGljaCBsZWFkcyBtZSB0byBiZWxpZXZlIHRoYXQg dGhlcmUgaXMgc29tZSAiY29yZSBjb25jZXB0IiBJIG1pZ2h0IGJlIG9ibGl2aW91cyB0by48YnI+ PGJyPlRoZSBtYWluIHF1ZXN0aW9uIGlzIGJhc2ljYWxseSB0aGlzOiBZb3VyIHN5c3RlbSBzdGFy dHMgdG8gc3dhcCBvdXQgZGF0YSBmcm9tIFJBTSB0byB5b3VyIHN3YXAgcGFydGl0aW9uLiBUaGlz IHN3YXAgZGF0YSBvbiBkaXNrIHVsdGltYXRlbHkgcmVzaWRlcyBzb21ld2hlcmUgaW4gYSBaRlMg cG9vbC4gSWYgdGhpcyBkYXRhIHRoZW4gZ2V0cyBhY2Nlc3NlZCwgaXQgbWlnaHQgYmUgY2FjaGVk IGJ5IEFSQyBlc3NlbnRpYWxseSBlYXRpbmcgdXAgbWVtb3J5IGFnYWluIHdoaWNoIHNlZW1zIGNv dW50ZXIgcHJvZHVjdGl2ZS48YnI+SXMgdGhlcmUgYW55IG1hZ2ljIHdoaWNoIHByZXZlbnRzIHN3 YXAgcGFydGl0aW9ucyBmcm9tIGJlaW5nIGxvYWRlZCBpbnRvIEFSQz8gT3IgaXMgdGhpcyBhIG5v bi1pc3N1ZSBmb3Igc29tZSBvdGhlciByZWFzb24/PGJyPjwvZGl2PjxkaXYgc3R5bGU9ImZvbnQt ZmFtaWx5OiBBcmlhbDsgZm9udC1zaXplOiAxNHB4OyI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9ImZv bnQtZmFtaWx5OiBBcmlhbDsgZm9udC1zaXplOiAxNHB4OyI+PGJyPjwvZGl2PjxkaXYgc3R5bGU9 ImZvbnQtZmFtaWx5OiBBcmlhbDsgZm9udC1zaXplOiAxNHB4OyI+QmVzdCByZWdhcmRzLDwvZGl2 PjxkaXYgc3R5bGU9ImZvbnQtZmFtaWx5OiBBcmlhbDsgZm9udC1zaXplOiAxNHB4OyI+fiBqb2Vs PGJyPjwvZGl2Pg0KPGRpdiBjbGFzcz0icHJvdG9ubWFpbF9zaWduYXR1cmVfYmxvY2sgcHJvdG9u bWFpbF9zaWduYXR1cmVfYmxvY2stZW1wdHkiIHN0eWxlPSJmb250LWZhbWlseTogQXJpYWw7IGZv bnQtc2l6ZTogMTRweDsiPg0KICAgIDxkaXYgY2xhc3M9InByb3Rvbm1haWxfc2lnbmF0dXJlX2Js b2NrLXVzZXIgcHJvdG9ubWFpbF9zaWduYXR1cmVfYmxvY2stZW1wdHkiPg0KICAgICAgICANCiAg ICAgICAgICAgIDwvZGl2Pg0KICAgIA0KICAgICAgICAgICAgPGRpdiBjbGFzcz0icHJvdG9ubWFp bF9zaWduYXR1cmVfYmxvY2stcHJvdG9uIHByb3Rvbm1haWxfc2lnbmF0dXJlX2Jsb2NrLWVtcHR5 Ij4NCiAgICAgICAgDQogICAgICAgICAgICA8L2Rpdj4NCjwvZGl2Pg0K --b1_hsD2aARXOhg4sFp8EdgWkAv1ymA04gVBAXGHHZAi8-- From eugen@grosbein.net Thu Feb 2 14:41:44 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P71hn2MJKz3fK29 for ; Thu, 2 Feb 2023 14:42:17 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P71hm6cTMz4cDX for ; Thu, 2 Feb 2023 14:42:16 +0000 (UTC) (envelope-from eugen@grosbein.net) Authentication-Results: mx1.freebsd.org; none Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.16.1/8.16.1) with ESMTPS id 312Eg8om093229 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 2 Feb 2023 14:42:09 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: jbo@insane.engineer Received: from [10.58.0.11] (dadvw [10.58.0.11] (may be forged)) by eg.sd.rdtc.ru (8.16.1/8.16.1) with ESMTPS id 312Eg7xx099740 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Thu, 2 Feb 2023 21:42:07 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Swap, ZFS & ARC To: jbo@insane.engineer, "hackers@freebsd.org" References: From: Eugene Grosbein Message-ID: <1689ba8c-a317-ce1f-2854-99566468a9ed@grosbein.net> Date: Thu, 2 Feb 2023 21:41:44 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT autolearn=disabled version=3.4.6 X-Spam-Report: * -0.0 SHORTCIRCUIT No description available. * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on hz.grosbein.net X-Rspamd-Queue-Id: 4P71hm6cTMz4cDX X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N 02.02.2023 21:28, jbo@insane.engineer wrote: > Hello folks, > > Based on a discussion on the forums not so long ago I tried to figure out how swap usage on a ZFS system plays together with ARC. However, I could find very little to no information on this which leads me to believe that there is some "core concept" I might be oblivious to. > > The main question is basically this: Your system starts to swap out data from RAM to your swap partition. > This swap data on disk ultimately resides somewhere in a ZFS pool. I prefer not doing this. That is, all my systems have swap partition outside of ZFS pool. From nobody Thu Feb 2 15:45:38 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P73666sMQz3fmtm for ; Thu, 2 Feb 2023 15:45:50 +0000 (UTC) (envelope-from jbo@insane.engineer) Received: from mail-4323.proton.ch (mail-4323.proton.ch [185.70.43.23]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "protonmail.com", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P73646W8Nz3MCG for ; Thu, 2 Feb 2023 15:45:48 +0000 (UTC) (envelope-from jbo@insane.engineer) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=insane.engineer header.s=protonmail3 header.b=nv3G1LpN; spf=pass (mx1.freebsd.org: domain of jbo@insane.engineer designates 185.70.43.23 as permitted sender) smtp.mailfrom=jbo@insane.engineer; dmarc=pass (policy=none) header.from=insane.engineer Date: Thu, 02 Feb 2023 15:45:38 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=insane.engineer; s=protonmail3; t=1675352746; x=1675611946; bh=4bcXwElOg8iv3S5lRZPuWnMFqoUgZnHca23VMVc3ZWc=; h=Date:To:From:Subject:Message-ID:In-Reply-To:References: Feedback-ID:From:To:Cc:Date:Subject:Reply-To:Feedback-ID: Message-ID:BIMI-Selector; b=nv3G1LpNqfNM5TPwc5P/MbR6r3NZzQNkzkKFd45t0gATpftdw8LR5EUlUGCMq3sJ3 2KU6uaFKRl67Nk215mb9Vns/BbRUl0/maLXUCGvy84Gx97bCfoWnaUOguJDluXLSE4 NSpp3GSkr4kvWCE2AgLAWpkPZNS0eeQy0oD06L2iJqOhJkHt9xLpUF8r9+2IFvj1qA oBnABzjzEkY9zVXI3d8a1qkOIXEkNJPOkATTqnqbAw4ChpUdwGwl+syjQw7RPNYYKL BLsHbegqlf+vKqk6biGAMRmn/v4+Rd3yVsoGGmZha6gB9mYgp2qE6V61/pIoT4HBOT 2RzHc4ok/ooyg== To: "hackers@freebsd.org" From: jbo@insane.engineer Subject: Re: Swap, ZFS & ARC Message-ID: In-Reply-To: <1689ba8c-a317-ce1f-2854-99566468a9ed@grosbein.net> References: <1689ba8c-a317-ce1f-2854-99566468a9ed@grosbein.net> Feedback-ID: 40997969:user:proton List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[insane.engineer,none]; R_SPF_ALLOW(-0.20)[+ip4:185.70.43.0/24]; R_DKIM_ALLOW(-0.20)[insane.engineer:s=protonmail3]; MIME_GOOD(-0.10)[text/plain]; MLMMJ_DEST(0.00)[hackers@freebsd.org]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:62371, ipnet:185.70.43.0/24, country:CH]; TO_DN_EQ_ADDR_ALL(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; FROM_NO_DN(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DKIM_TRACE(0.00)[insane.engineer:+]; RWL_MAILSPIKE_POSSIBLE(0.00)[185.70.43.23:from] X-Rspamd-Queue-Id: 4P73646W8Nz3MCG X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N ------- Original Message ------- On Thursday, February 2nd, 2023 at 14:41, Eugene Grosbein wrote: >=20 >=20 > 02.02.2023 21:28, jbo@insane.engineer wrote: >=20 > > Hello folks, > >=20 > > Based on a discussion on the forums not so long ago I tried to figure o= ut how swap usage on a ZFS system plays together with ARC. However, I could= find very little to no information on this which leads me to believe that = there is some "core concept" I might be oblivious to. > >=20 > > The main question is basically this: Your system starts to swap out dat= a from RAM to your swap partition. > > This swap data on disk ultimately resides somewhere in a ZFS pool. >=20 >=20 > I prefer not doing this. That is, all my systems have swap partition outs= ide of ZFS pool. This isn't really achievable on your average "single disk" desktop systems,= right? Assuming I'm not missing something (which is why I came here to ask), a "re= gular" or "default" installation through the FreeBSD installer would setup = a swap partition within the zpool, right? From nobody Thu Feb 2 16:04:02 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P73WN5Bgvz3fp09 for ; Thu, 2 Feb 2023 16:04:16 +0000 (UTC) (envelope-from asomers@gmail.com) Received: from mail-vs1-f50.google.com (mail-vs1-f50.google.com [209.85.217.50]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P73WN30yVz3RLX for ; Thu, 2 Feb 2023 16:04:16 +0000 (UTC) (envelope-from asomers@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-vs1-f50.google.com with SMTP id s24so2299133vsi.12 for ; Thu, 02 Feb 2023 08:04:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=4feOC/QctWD9Y/YLctfByAuDLyTN7Sj/4YbXr0z5QGs=; b=Z6rpmg9AbAwoJHivHMwCrGnPLhxwHkE5pY5zoAhLWwVJQ9RAqjsTB0WxMM2jqIZxtr l7NeoZl1TFfsApgRzTwdIBLixusszzCNTKA93qEI9Uyk+n0WyIWZ5wFvuE6dP4wAoal+ 98kX7FrIZ7vZYb33ihppK3i5M+M0czPoAbJ/2KFRzlmr+2g5+iU8b9lzjQEQUm3VPYvh Yoka0hFRZ//cJnfaaju2vCJJEvHcz6fHPTMfxhvMz9j6JEe+PEJne0sxZ2cIVnvdPH/U M3NjhEcduoPeI7+jqpHDvGLT5aSCfzvWQUV+7t/iMZJptbaSejjHfG9YhafHCoTVGt0L nx/g== X-Gm-Message-State: AO0yUKVyW6Ed2yDnMvindRkpEazTP6xOeXm8AAcQKcMlOWvvDqhzh+Ff U6Gl2T0bGXd7QpamAcdjPi+5gGAMzYl2OODMHv0H6YuB0Ew= X-Google-Smtp-Source: AK7set8iFiutMn9S0ddGEi9+d39B9izwm6v75zaxO0sG5cZ4GvCJPXnfUKvLQVXLC8LlfV/55BZjYDpUzHDXILBkisQ= X-Received: by 2002:a67:f501:0:b0:3f6:c8ee:b841 with SMTP id u1-20020a67f501000000b003f6c8eeb841mr1035766vsn.76.1675353854841; Thu, 02 Feb 2023 08:04:14 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <1689ba8c-a317-ce1f-2854-99566468a9ed@grosbein.net> In-Reply-To: From: Alan Somers Date: Thu, 2 Feb 2023 09:04:02 -0700 Message-ID: Subject: Re: Swap, ZFS & ARC To: jbo@insane.engineer Cc: "hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4P73WN30yVz3RLX X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N On Thu, Feb 2, 2023 at 8:45 AM wrote: > > > > > > > ------- Original Message ------- > On Thursday, February 2nd, 2023 at 14:41, Eugene Grosbein wrote: > > > > > > > > 02.02.2023 21:28, jbo@insane.engineer wrote: > > > > > Hello folks, > > > > > > Based on a discussion on the forums not so long ago I tried to figure= out how swap usage on a ZFS system plays together with ARC. However, I cou= ld find very little to no information on this which leads me to believe tha= t there is some "core concept" I might be oblivious to. > > > > > > The main question is basically this: Your system starts to swap out d= ata from RAM to your swap partition. > > > This swap data on disk ultimately resides somewhere in a ZFS pool. > > > > > > I prefer not doing this. That is, all my systems have swap partition ou= tside of ZFS pool. > > This isn't really achievable on your average "single disk" desktop system= s, right? > Assuming I'm not missing something (which is why I came here to ask), a "= regular" or "default" installation through the FreeBSD installer would setu= p a swap partition within the zpool, right? Actually, the installer by default sets up a swap partition beside the zroot pool. It doesn't use swap-on-zvol. Here's what it looks like: > gpart show =3D> 40 1953525088 nvd0 GPT (932G) 40 532480 1 efi (260M) 532520 1024 2 freebsd-boot (512K) 533544 984 - free - (492K) 534528 16777216 3 freebsd-swap (8.0G) 17311744 1936211968 4 freebsd-zfs (923G) 1953523712 1416 - free - (708K) =3D> 40 1953525088 nvd1 GPT (932G) 40 532480 1 efi (260M) 532520 1024 2 freebsd-boot (512K) 533544 984 - free - (492K) 534528 16777216 3 freebsd-swap (8.0G) 17311744 1936211968 4 freebsd-zfs (923G) 1953523712 1416 - free - (708K) > zpool status zroot pool: zroot state: ONLINE scan: resilvered 314G in 00:10:19 with 0 errors on Sat Dec 24 22:31:04 20= 22 config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nvd0p4 ONLINE 0 0 0 nvd1p4 ONLINE 0 0 0 > swapinfo Device 1K-blocks Used Avail Capacity /dev/gpt/swap0 8388608 0 8388608 0% From nobody Thu Feb 2 16:17:09 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P73pV3SGXz3fpVx for ; Thu, 2 Feb 2023 16:17:22 +0000 (UTC) (envelope-from bakul@iitbombay.org) Received: from mail-io1-xd29.google.com (mail-io1-xd29.google.com [IPv6:2607:f8b0:4864:20::d29]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P73pV1MZhz3jMt for ; Thu, 2 Feb 2023 16:17:22 +0000 (UTC) (envelope-from bakul@iitbombay.org) Authentication-Results: mx1.freebsd.org; none Received: by mail-io1-xd29.google.com with SMTP id r6so928999ioj.5 for ; Thu, 02 Feb 2023 08:17:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=iitbombay-org.20210112.gappssmtp.com; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:from:to:cc:subject:date:message-id:reply-to; bh=/1P19ak9VIN28ud035JOAJZLnXcvlEYRFxrsJB1xNxc=; b=v4CgXKG5TaVg4WKWVEjWJlXeEP3eHNsEsQtw+d7TrY1ZzP5oGwsDuT+CbAzpG1JrRf Cpif/wLciuEywljvTGZjD10NTufrCGL5VqjMbx/6w6e61TzXhTPcVDfQpkdcD6o91jfI Q7fPg8uXt9a8zkAg04hB7eq+QNXNLYneOamDqoAcSn2wXq7Avcq601zwrnBPatYpD5Gq gv6Iefh3ELxSL4T5Vgzb4TtTZtBUzuNRqTYQyQ/xNXNeo3PhaKulRFLA+aACpedM6b9D V+LwoOXCfJVB0avBUalvRo5tAtLb+wXzZR++8ubsu5fQ3A15Gw8QcnlIPA+Qny8aY7ug ikdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=references:to:cc:in-reply-to:date:subject:mime-version:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/1P19ak9VIN28ud035JOAJZLnXcvlEYRFxrsJB1xNxc=; b=4cZNvlgM3XODSa/DfGZibGqoJzgIV/v/+gXinKE3WeE+Oa6MBWqWbH4DdRP/epWjFZ LsRQ43jjEqTcMa/0SdQN/Q2gGXEZLoxoL1w9SeDhL5zgwpVOdk2ACGjz5FRmGymN9ACG 9TaCkHdic+tPoZS2kGKyiN1HW6J/RsjkU0kTXUv1E1K6EfvqRXK1Q+nYn4uIxJCDZMDY ounltGqQiHXmdlTmd2k/0WkdY8AOKLhB22LsOn6+cdlEHtFw2fGhBPtdtnk/c2aKY9Ss whA/RHfTz17NSiS+jmxjaP5HTgG4TLB/JhQkTWn1tEZqZa2S49EWwW4yOuDaFhAKhKXJ vlzQ== X-Gm-Message-State: AO0yUKWBv7Gz/NFkMzh7KurCw3RHuHCWonLb3R4eDdnxmLSSf64Ci1Sm TexcplCnUqZaJ7SHu5Ji31yrAA5N2Vm+1PoE X-Google-Smtp-Source: AK7set/wyGkKON4BA8lOTqvCq0K8Wv8gXTiMb5DrEA15qoINP5JOHrTpAOLcVQObYkkJCnDTcfYeWQ== X-Received: by 2002:a05:6602:2116:b0:6de:3e2c:d791 with SMTP id x22-20020a056602211600b006de3e2cd791mr4748797iox.1.1675354641140; Thu, 02 Feb 2023 08:17:21 -0800 (PST) Received: from smtpclient.apple (107-215-223-229.lightspeed.sntcca.sbcglobal.net. [107.215.223.229]) by smtp.gmail.com with ESMTPSA id h26-20020a02c73a000000b003b778515852sm17416jao.168.2023.02.02.08.17.20 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 02 Feb 2023 08:17:20 -0800 (PST) From: Bakul Shah Message-Id: <3DEFA166-79E8-49AD-B0DD-7E524C8EAAE8@iitbombay.org> Content-Type: multipart/alternative; boundary="Apple-Mail=_3A2A755A-EAD4-4C04-ADE4-2E010592DBA9" List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: Swap, ZFS & ARC Date: Thu, 2 Feb 2023 08:17:09 -0800 In-Reply-To: Cc: "hackers@freebsd.org" To: jbo@insane.engineer References: X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Rspamd-Queue-Id: 4P73pV1MZhz3jMt X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --Apple-Mail=_3A2A755A-EAD4-4C04-ADE4-2E010592DBA9 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Feb 2, 2023, at 6:28 AM, jbo@insane.engineer wrote: >=20 > Hello folks, >=20 > Based on a discussion on the forums not so long ago I tried to figure = out how swap usage on a ZFS system plays together with ARC. However, I = could find very little to no information on this which leads me to = believe that there is some "core concept" I might be oblivious to. >=20 > The main question is basically this: Your system starts to swap out = data from RAM to your swap partition. This swap data on disk ultimately = resides somewhere in a ZFS pool. If this data then gets accessed, it = might be cached by ARC essentially eating up memory again which seems = counter productive. > Is there any magic which prevents swap partitions from being loaded = into ARC? Or is this a non-issue for some other reason? I suspect this bug affects FreeBSD as well: https://github.com/openzfs/zfs/issues/7734 =46rom https://github.com/openzfs/zfs/issues/7734#issuecomment-422082279 I'm not an expert in this area of the code, but I think that swap on = ZVOL is inherently unreliable due to writes to the swap ZVOL having to = go through the normal TXG sync and ZIO write paths, which can require = lots of memory allocations by design (and these memory allocations can = stall due to a low memory situation). I believe this to be true for swap = on ZVOL for illumos, as well as Linux, and presumably FreeBSD too = (although I have no experience using it on FreeBSD, so I could be = wrong). FYI. I don't know enough about zfs internals so can't say if this poster = is right or not but I too have just used a disk partition as opposed to = a zvolume for swap.= --Apple-Mail=_3A2A755A-EAD4-4C04-ADE4-2E010592DBA9 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

On Feb 2, 2023, at 6:28 AM, = jbo@insane.engineer wrote:

Hello folks,

Based on a = discussion on the forums not so long ago I tried to figure out how swap = usage on a ZFS system plays together with ARC. However, I could find = very little to no information on this which leads me to believe that = there is some "core concept" I might be oblivious to.

The main = question is basically this: Your system starts to swap out data from RAM = to your swap partition. This swap data on disk ultimately resides = somewhere in a ZFS pool. If this data then gets accessed, it might be = cached by ARC essentially eating up memory again which seems counter = productive.
Is there any magic which prevents swap partitions from = being loaded into ARC? Or is this a non-issue for some other = reason?

I suspect this bug affects FreeBSD as = well:

https://github.com/openzfs/zfs/issues/7734

From = https://github.com/openzfs/zfs/issues/7734#issuecomment-422082279

<= blockquote style=3D"margin: 0 0 0 40px; border: none; padding: = 0px;">
I'm not an expert in this area of the code, but I think that = swap on ZVOL is inherently unreliable due to writes to the swap ZVOL = having to go through the normal TXG sync and ZIO write paths, which can = require lots of memory allocations by design (and these memory = allocations can stall due to a low memory situation). I believe this to = be true for swap on ZVOL for illumos, as well as Linux, and presumably = FreeBSD too (although I have no experience using it on FreeBSD, so I = could be wrong).

FYI. I don't know enough = about zfs internals so can't say if this poster is right or not but I = too have just used a disk partition as opposed to a zvolume for = swap.
= --Apple-Mail=_3A2A755A-EAD4-4C04-ADE4-2E010592DBA9-- From nobody Thu Feb 2 16:21:06 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P73ts0Hk7z3fpfn for ; Thu, 2 Feb 2023 16:21:09 +0000 (UTC) (envelope-from bacon4000@gmail.com) Received: from mail-yw1-x1135.google.com (mail-yw1-x1135.google.com [IPv6:2607:f8b0:4864:20::1135]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P73tr2WFZz3kld for ; Thu, 2 Feb 2023 16:21:08 +0000 (UTC) (envelope-from bacon4000@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=hGQki1iX; spf=pass (mx1.freebsd.org: domain of bacon4000@gmail.com designates 2607:f8b0:4864:20::1135 as permitted sender) smtp.mailfrom=bacon4000@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-yw1-x1135.google.com with SMTP id 00721157ae682-50e7a0f0cc8so33353967b3.2 for ; Thu, 02 Feb 2023 08:21:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :from:to:cc:subject:date:message-id:reply-to; bh=qulT7supUEEWdHJmJDoY8oNENPcat2PnOsVEDcH/now=; b=hGQki1iXry/9bbUHEBnsyVqN3ojTVwWY43Vp4OA2LPqsNOPPBR7QNpo37inEKx+MYE gopuVO6PYIcObLnO7b2+jYXyNCqIbfSg5KIdzt5fJUW/Hj+dh1uExC1eqlRBruOruKGY HXl9gTCiUBYZdM0EQqYplb+AsAvA1NXc8Bpymcjmw1nA1J/JEiezwjzL/X9MRA5g+pWk xmsLFABKvGGKOk92JV5qVeafL8IIHHUNAXZ5QWWxgiBBRx+5xXgL0jcMgYvjzvBGIvmf 1MdNz88zB9j/AKggzQ/wA/ikqVxFsgaNi7tA8ydUFNR6bjnUt+Y3l/mAbihq76bCc0wH d9pA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:from:references:to :content-language:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=qulT7supUEEWdHJmJDoY8oNENPcat2PnOsVEDcH/now=; b=OHjJOlGW5Zc13UX5DdxRYRIA5P2yTlmVA4B8GRvB2ZWTXpIDxp9/QvyE+UBV3FcON+ 4cJPxqntCQqlauoYNqLpNMSU1EQ4uM1f0wfKuFaX6YE0VfQPCcLKkh00Xo2IpCzkEajU I+gWYwV51nC2ezd27aT8LJwl0qWdUrW107y8pg+AyBCrWdz3rPOPGAbeBwHnKwNUNV54 H8T1iDZNWRdjF8VE3sq0nnRTg0IyNRC1OfWstiZ4+dl4owPAziSbrbxuqlzsuN3rH9Vx 8c1W/M/WURcCEXrWB81+q6enToh/EgfxG15uSBNzbTWGcx/xN2S72kyV1ytAjLxWo5xV PnFg== X-Gm-Message-State: AO0yUKVF4QIwpYhSReP7PlZwg3vj3pUTiat6rhBrZ0zN+pXiBLPs/Tdw 4ugsH77FkwifdfplwwE417kXVKvIiEA= X-Google-Smtp-Source: AK7set/NhJce8lap4L5IWmqYActWGLz3Yb4wbJAsS/DaZ/yUPJMuD1mwMm6Mz1MR8c7Y+rcy6EXgFQ== X-Received: by 2002:a81:ef03:0:b0:4fe:e703:109e with SMTP id o3-20020a81ef03000000b004fee703109emr4149684ywm.39.1675354867296; Thu, 02 Feb 2023 08:21:07 -0800 (PST) Received: from [192.168.0.3] (cpe-184-58-230-200.wi.res.rr.com. [184.58.230.200]) by smtp.gmail.com with ESMTPSA id ea7-20020a05620a488700b007290be5557bsm2528qkb.38.2023.02.02.08.21.06 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 02 Feb 2023 08:21:06 -0800 (PST) Message-ID: <1484659b-216a-3491-e569-8e67fda3bfcc@gmail.com> Date: Thu, 2 Feb 2023 10:21:06 -0600 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: Swap, ZFS & ARC Content-Language: en-US To: freebsd-hackers@freebsd.org References: <1689ba8c-a317-ce1f-2854-99566468a9ed@grosbein.net> From: Jason Bacon In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; MIME_GOOD(-0.10)[text/plain]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; ARC_NA(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::1135:from]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_NONE(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_TRACE(0.00)[0:+]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org] X-Rspamd-Queue-Id: 4P73tr2WFZz3kld X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N On 2/2/23 09:45, jbo@insane.engineer wrote: >> I prefer not doing this. That is, all my systems have swap partition outside of ZFS pool. > This isn't really achievable on your average "single disk" desktop systems, right? > Assuming I'm not missing something (which is why I came here to ask), a "regular" or "default" installation through the FreeBSD installer would setup a swap partition within the zpool, right? > Yes, it is. You can install to a UFS partition + swap, leaving most of the disk unused. After install, create the ZFS pool on the free space. I use this strategy for mission-critical servers, since I've run into occasional boot problems using root on ZFS. -- Life is a game. Play hard. Play fair. Have fun. From nobody Thu Feb 2 16:49:39 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P74X244Kvz3kM5l for ; Thu, 2 Feb 2023 16:49:54 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x630.google.com (mail-ej1-x630.google.com [IPv6:2a00:1450:4864:20::630]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P74X215HWz3q2f for ; Thu, 2 Feb 2023 16:49:54 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x630.google.com with SMTP id mc11so7640987ejb.10 for ; Thu, 02 Feb 2023 08:49:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=gE7ThT1KWpqlw+gTTZIL7C9BVutbs9mjO1A65zlTe5Y=; b=iJ7FOoI3sVbtKG/66BA0DligyJ2x/wHcnkfR4uRm1/Su6aQLdFLoGfm8BvlH2CAcFB CAJuMCYXvmVOxJWlJtwWa/fjzAN69mOc/FXFrrqn9NHnLGq1MG18jEx05mgfMNDFV3fv mt1CfDhKExd5l4WZvI2QCH2lBb+JpLnme16XPXAPWjLdI5Rt+5qBb5dvCkWU8mTenjhk Ds3++F/jvd1LpdKox+IThAjZ+vic9BVqKcyOIPvQ2cZxHMTHQ0nh7myVj+yKWkW5Yoxs mYVLv/VKkGtd2qYCeVzFwqa3UIVzyL2y5fRK65tr5PHvW+hY/wM+zwUxNiFj0bCGKG+K fnGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=gE7ThT1KWpqlw+gTTZIL7C9BVutbs9mjO1A65zlTe5Y=; b=Ei7+ZnXpKrLkfxAeV9RU/PDJzBipxjObIIi08lt+4iJWg9i2LyTZFwBNjSVTANrZ26 v8XT10tBzlKrNGuyTjUHSppcOJQfP+F0ZgInHigLIWL0zI5qWlIedOzvWPE/lr+PfzU7 Zfd1DFflU5QYouBM5fBpygdD2cGQybxqXABvs4sy6mL+vjcPL1rOVwr2q7Z5yUG0oVjV 1lpZtCnX4K/85KxckGXXS7jUkTZLVDcUU4FU7TGIBjHbSOqsRwgUfVpATJ1fLP0AiKaE pYEbh2etetXpus/3OsFHZM+dXX/tWPeM7JNksbGJ/w7CVLgNoDBFjjWiu4T471J4aMRn HwGg== X-Gm-Message-State: AO0yUKWngpn4JSN1eZ2Eu8biZcTk8ENe9nVPPp7BOKvawgzYCtgq/aNy LC9fzv7hZlncge9ugNyDnkup2k3c4up3HsJKgc8v9Gb5mildwQ== X-Google-Smtp-Source: AK7set9cQ3vmcenQMs97xsZ6DlM/LAxDUW9bk1CR9icOxQUwIS9Hu+cn2znuht74zDMwxRlp/5z57YwsqjW8Gb5cffo= X-Received: by 2002:a17:906:e09a:b0:877:ec5e:87ce with SMTP id gh26-20020a170906e09a00b00877ec5e87cemr2080783ejb.262.1675356591114; Thu, 02 Feb 2023 08:49:51 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <1689ba8c-a317-ce1f-2854-99566468a9ed@grosbein.net> In-Reply-To: <1689ba8c-a317-ce1f-2854-99566468a9ed@grosbein.net> From: Warner Losh Date: Thu, 2 Feb 2023 09:49:39 -0700 Message-ID: Subject: Re: Swap, ZFS & ARC To: Eugene Grosbein Cc: jbo@insane.engineer, "hackers@freebsd.org" Content-Type: multipart/alternative; boundary="0000000000007c5f6d05f3ba5acf" X-Rspamd-Queue-Id: 4P74X215HWz3q2f X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000007c5f6d05f3ba5acf Content-Type: text/plain; charset="UTF-8" On Thu, Feb 2, 2023 at 7:42 AM Eugene Grosbein wrote: > 02.02.2023 21:28, jbo@insane.engineer wrote: > > Hello folks, > > > > Based on a discussion on the forums not so long ago I tried to figure > out how swap usage on a ZFS system plays together with ARC. However, I > could find very little to no information on this which leads me to believe > that there is some "core concept" I might be oblivious to. > > > > The main question is basically this: Your system starts to swap out data > from RAM to your swap partition. > > This swap data on disk ultimately resides somewhere in a ZFS pool. > > I prefer not doing this. That is, all my systems have swap partition > outside of ZFS pool. > I agree. Don't swap to anything but a raw partition if you have any way possible to avoid it. Swapping to anything ZFS provides is a bad idea because ZFS has to do memory allocations to do the I/O, which might not be possible when heavily swapping on a memory constrained system. If it can't allocate memory to do the swapping, it can't release the dirty memory that is being swapped out, possibly leading to deadlock. I actively avoid it unless there's no alternative, just like I actively avoid swapping to a file in a filesystem like UFS if I can avoid it too... Warner --0000000000007c5f6d05f3ba5acf Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Feb 2, 2023 at 7:42 AM Eugene= Grosbein <eugen@grosbein.net&= gt; wrote:
02.02= .2023 21:28, jbo@insane.engineer wrote:
> Hello folks,
>
> Based on a discussion on the forums not so long ago I tried to figure = out how swap usage on a ZFS system plays together with ARC. However, I coul= d find very little to no information on this which leads me to believe that= there is some "core concept" I might be oblivious to.
>
> The main question is basically this: Your system starts to swap out da= ta from RAM to your swap partition.
> This swap data on disk ultimately resides somewhere in a ZFS pool.

I prefer not doing this. That is, all my systems have swap partition outsid= e of ZFS pool.

I agree. Don't swap = to anything but a raw partition if you have any way possible to avoid it. S= wapping to anything ZFS provides is a bad idea because ZFS has to do memory= allocations to do the I/O, which might not be possible when heavily swappi= ng on a memory constrained system. If it can't allocate memory to do th= e swapping, it can't release the dirty memory that is being swapped out= , possibly leading to deadlock.

I actively avoid i= t unless there's no alternative, just like I actively avoid swapping to= a file in a filesystem like UFS if I can avoid it too...

Warner
--0000000000007c5f6d05f3ba5acf-- From nobody Thu Feb 2 16:52:34 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P74bM35N2z3kM63 for ; Thu, 2 Feb 2023 16:52:47 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-ej1-x632.google.com (mail-ej1-x632.google.com [IPv6:2a00:1450:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P74bM0cDYz3rR7 for ; Thu, 2 Feb 2023 16:52:47 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x632.google.com with SMTP id bk15so7695652ejb.9 for ; Thu, 02 Feb 2023 08:52:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20210112.gappssmtp.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=kMPXAsqukwGEKN2a2WZUNHRhYP3peWhXdJroy1XhKUE=; b=jBdHlYGc/Su6nOr98jhEnooR2ljpCGeHLveTn6bdsx/s7gwrRVXHaj2Imi2uL28hWv G3boplbhvRYiXrPNd6LCuuTPU7eZfQdFlhpG9YE+j1W1kiNZdWomVrVZE5YwGeNgTRG9 22NbV+UsOISnbYs6rdO8BYA6nDUkeFmRq5BzToUWMzquU41Ufb2cbZ8nhnqaoMwSq/li HXrqEPhjRTfOyeaI3o5fM4y+l7wGn9h7Tawk6rN6Ds7nVgm311pTVSxVZul4m+7MvkeQ 0NN+41GVOStljPQMhf2stdHftDLX5ah1VYkGMNsG1HyGZ0t/A9y2UavmS93hxZtrbvfI 68eA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=kMPXAsqukwGEKN2a2WZUNHRhYP3peWhXdJroy1XhKUE=; b=dTgBIsTKAddfTpF17aEMRph9x9HJPZsBI/8/eLksdjzMgLWV0M6O2a4Kk0pICxFrCw 1Im2adz4HkOzjtkftdGlRFxr6IK486N56jQK2UdT3cduvg8l5a0upHDQIHwAUhdf5sdt P6QB9uo8/nDFGnIlXnJIoomGI6DW119Nl3jO/4S8+O13vCWJNh9kOaRLDWFcasfFsbMN ecIM2GReHCXlenZCjDmMsnKiUqJ6iy3mMxQxuCNrXFPHOfd3DWQ63/kkMQ3d0isBnZ/3 BLo8ib9sue2+HBsIdAkqX2knunsYINpNGzJ17CwqaU+yy+W4kSM6/RqRfr/7dsbEaXhS ad1Q== X-Gm-Message-State: AO0yUKVcgDvqYA7L1L3c1J0Kn8Lp8hpWLRuKN5RB+xTX36R4T/dQIx1S p7MlQDvTiCxakAZVq12trmQNxK+Gj7jRcr3KXPzxiA== X-Google-Smtp-Source: AK7set85vxj5O4hUiYd2uWSZgz0YRo0YFftvRVaBT1zb4DJrGgLk4MSSWRmIKqfRw2m2a8wHcqKnGBLj8/vjc30d52c= X-Received: by 2002:a17:906:5390:b0:888:1f21:4424 with SMTP id g16-20020a170906539000b008881f214424mr2281506ejo.141.1675356765066; Thu, 02 Feb 2023 08:52:45 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <3DEFA166-79E8-49AD-B0DD-7E524C8EAAE8@iitbombay.org> In-Reply-To: <3DEFA166-79E8-49AD-B0DD-7E524C8EAAE8@iitbombay.org> From: Warner Losh Date: Thu, 2 Feb 2023 09:52:34 -0700 Message-ID: Subject: Re: Swap, ZFS & ARC To: Bakul Shah Cc: jbo@insane.engineer, "hackers@freebsd.org" Content-Type: multipart/alternative; boundary="000000000000dab04b05f3ba6460" X-Rspamd-Queue-Id: 4P74bM0cDYz3rR7 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --000000000000dab04b05f3ba6460 Content-Type: text/plain; charset="UTF-8" On Thu, Feb 2, 2023 at 9:17 AM Bakul Shah wrote: > > On Feb 2, 2023, at 6:28 AM, jbo@insane.engineer wrote: > > > Hello folks, > > Based on a discussion on the forums not so long ago I tried to figure out > how swap usage on a ZFS system plays together with ARC. However, I could > find very little to no information on this which leads me to believe that > there is some "core concept" I might be oblivious to. > > The main question is basically this: Your system starts to swap out data > from RAM to your swap partition. This swap data on disk ultimately resides > somewhere in a ZFS pool. If this data then gets accessed, it might be > cached by ARC essentially eating up memory again which seems counter > productive. > Is there any magic which prevents swap partitions from being loaded into > ARC? Or is this a non-issue for some other reason? > > > I suspect this bug affects FreeBSD as well: > > https://github.com/openzfs/zfs/issues/7734 > > From https://github.com/openzfs/zfs/issues/7734#issuecomment-422082279 > > I'm not an expert in this area of the code, but I think that swap on ZVOL > is inherently unreliable due to writes to the swap ZVOL having to go > through the normal TXG sync and ZIO write paths, which can require lots of > memory allocations by design (and these memory allocations can stall due to > a low memory situation). I believe this to be true for swap on ZVOL for > illumos, as well as Linux, and presumably FreeBSD too (although I have no > experience using it on FreeBSD, so I could be wrong). > > > FYI. I don't know enough about zfs internals so can't say if this poster > is right or not but I too have just used a disk partition as opposed to a > zvolume for swap. > They have it right. FreeBSD tries hard (though isn't perfect in this regard) to not allocate memory in the I/O path, especially if it knows the request is for swap. It's possible to swap to zvols, and it actually works when the memory pressures are light (since the stalls usually aren't too long and usually complete). When memory pressure is high, though, these can turn into deadlocks. Warner --000000000000dab04b05f3ba6460 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Thu, Feb 2, 2023 at 9:17 AM Bakul = Shah <bakul@iitbombay.org>= wrote:

On Feb 2, 2023, at 6:28 AM, jbo@insane.engineer wrote:

Hello folks,

Based on a discussion on the f= orums not so long ago I tried to figure out how swap usage on a ZFS system = plays together with ARC. However, I could find very little to no informatio= n on this which leads me to believe that there is some "core concept&q= uot; I might be oblivious to.

The main question is basically this: Y= our system starts to swap out data from RAM to your swap partition. This sw= ap data on disk ultimately resides somewhere in a ZFS pool. If this data th= en gets accessed, it might be cached by ARC essentially eating up memory ag= ain which seems counter productive.
Is there any magic which prevents sw= ap partitions from being loaded into ARC? Or is this a non-issue for some o= ther reason?

I suspect this bug affects FreeBSD as well= :

https://github.com/openzfs/zfs/issues/7734

From=C2=A0https://github.com/openzfs/zfs/issues/7734#issuecomment-4= 22082279

I'm not an expert in this area of the code, but I t= hink that swap on ZVOL is inherently unreliable due to writes to the swap Z= VOL having to go through the normal TXG sync and ZIO write paths, which can= require lots of memory allocations by design (and these memory allocations= can stall due to a low memory situation). I believe this to be true for sw= ap on ZVOL for illumos, as well as Linux, and presumably FreeBSD too (altho= ugh I have no experience using it on FreeBSD, so I could be wrong).

FYI. I don't know enough about zfs internals so can= 't say if this poster is right or not but I too have just used a disk p= artition as opposed to a zvolume for swap.

They have it right. FreeBSD tries hard (though isn't = perfect in this regard) to not allocate memory in the I/O path, especially = if it knows the request is for swap. It's possible to swap to zvols, an= d it actually works when the memory pressures are light (since the stalls u= sually aren't too long and usually complete). When memory pressure is h= igh, though, these can turn into deadlocks.

Warner=
--000000000000dab04b05f3ba6460-- From eugen@grosbein.net Thu Feb 2 23:40:07 2023 X-Original-To: hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P7Fdv272Bz3kjRB for ; Thu, 2 Feb 2023 23:40:35 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:c2c:26d8::2]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P7Fdt3rlhz3kQH for ; Thu, 2 Feb 2023 23:40:34 +0000 (UTC) (envelope-from eugen@grosbein.net) Authentication-Results: mx1.freebsd.org; none Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.16.1/8.16.1) with ESMTPS id 312NeVYU098639 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Thu, 2 Feb 2023 23:40:31 GMT (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: jbo@insane.engineer Received: from [10.58.0.11] (dadvw [10.58.0.11] (may be forged)) by eg.sd.rdtc.ru (8.16.1/8.16.1) with ESMTPS id 312NeUJd003594 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Fri, 3 Feb 2023 06:40:30 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Swap, ZFS & ARC To: jbo@insane.engineer, "hackers@freebsd.org" References: <1689ba8c-a317-ce1f-2854-99566468a9ed@grosbein.net> From: Eugene Grosbein Message-ID: <5149591a-1bd8-c3e8-eaea-fc08808f1b0f@grosbein.net> Date: Fri, 3 Feb 2023 06:40:07 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,SHORTCIRCUIT autolearn=disabled version=3.4.6 X-Spam-Report: * -0.0 SHORTCIRCUIT No description available. * -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP X-Spam-Checker-Version: SpamAssassin 3.4.6 (2021-04-09) on hz.grosbein.net X-Rspamd-Queue-Id: 4P7Fdt3rlhz3kQH X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/32, country:DE] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N 02.02.2023 22:45, jbo@insane.engineer wrote: >> 02.02.2023 21:28, jbo@insane.engineer wrote: >> >>> Hello folks, >>> >>> Based on a discussion on the forums not so long ago I tried to figure out how swap usage on a ZFS system plays together with ARC. However, I could find very little to no information on this which leads me to believe that there is some "core concept" I might be oblivious to. >>> >>> The main question is basically this: Your system starts to swap out data from RAM to your swap partition. >>> This swap data on disk ultimately resides somewhere in a ZFS pool. >> >> >> I prefer not doing this. That is, all my systems have swap partition outside of ZFS pool. > > This isn't really achievable on your average "single disk" desktop systems, right? For desktop systems you must use GPT/MBR to boot off single disk and with partitioning there is no problem in creating dedicated slice for swap. From nobody Fri Feb 3 13:07:42 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P7bYx3PFXz2p5Mp for ; Fri, 3 Feb 2023 13:08:21 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P7bYw3nBzz3xmQ for ; Fri, 3 Feb 2023 13:08:20 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=OnGZA8yM; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::636 as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ej1-x636.google.com with SMTP id me3so15162209ejb.7 for ; Fri, 03 Feb 2023 05:08:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=4+MWOLqU4uNgcthREDDvVeAbpux5YkqI0KKP+ZlL1ls=; b=OnGZA8yMVZgAGjJRiRGNtOspWhm4UL5lcfDmihtlLFikBP05/B2PAXJIfP7MwV5b6t +IDgdH/C+9FrXH/psQ/w61S9/MKDYoT7ko1fT8Gv+V5jU3xRVGm8Pr48ZoFE+LAgvdUh WfxQbz6LaFpXtGUE80vr7lMgOv+q9FwiaM4piRP2HkVcSAGIwZy6g597D4Qn+JNDCArY PuzNzw0Flt1RWMRoz2dePra4RrvMLrCVaCXEB78GWul8je8623JIH1dYcWPAGlKkkzH8 lnbV8MvjfvzgZ1Bvy+ThHC7JbQIsknTzWw/SQ9uiZfvY0oN9Bd4N9anhDBHSvEAOojJp EGow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=4+MWOLqU4uNgcthREDDvVeAbpux5YkqI0KKP+ZlL1ls=; b=MQ0Hgr8kB5Ytwv/O/tEjEzlPOpFncVZJZL4enPbdZDKigJsv8YdMpNH/lNHknMZ4+L Nh98Yz2N7hvWRoSV4aImMQHPqz6EkMcoeLE41R6jGyBW8qu+3TSJMapPvdsKuWXsUskE RT8S0oJahJB6ioOssqx3YxoBflARSZHc2bwVOxIhaIty5EsDpsD6cqr1DIl1rv/ZaXM2 W5MiISpUJYcPkze1kUny7RQ9GvmzMdtyaZvF13xgObdHIvlyJLrTqhdj6gJudaGjwemc y0Hd7Rh2DTsa6dw8xMO0Q9aSFyoegP7wM2mS3pb7RZm2l8ZmQ1MyfCAFCzZx72L5VNdE u83w== X-Gm-Message-State: AO0yUKXhgebDe+4M26X/pcga7nSN+NIRs8oNWZNba1qQpmZJtXNZ/nKE 2+i5/N1OkXZIJcdHg5z6XqQfVhYMq8f/fFLVFhoj15k9LdY51Q== X-Google-Smtp-Source: AK7set9rcdKHWqNcpxBZy6CU026pRZ/cgC+WVM6303XLp30HwGuDPN05pAr9EsLlPdoEe3T9MT+hDSPAHhM7CsZKuJI= X-Received: by 2002:a17:907:cb25:b0:880:b580:d93 with SMTP id um37-20020a170907cb2500b00880b5800d93mr3044801ejc.276.1675429699107; Fri, 03 Feb 2023 05:08:19 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au> In-Reply-To: <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au> From: Mario Marietto Date: Fri, 3 Feb 2023 14:07:42 +0100 Message-ID: Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy To: "Daniel O'Connor" Cc: freebsd-hackers Content-Type: multipart/alternative; boundary="0000000000000fc01705f3cb609f" X-Spamd-Result: default: False [-3.94 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.94)[-0.940]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MID_RHS_MATCH_FROMTLD(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::636:from]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4P7bYw3nBzz3xmQ X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --0000000000000fc01705f3cb609f Content-Type: text/plain; charset="UTF-8" Hello Daniel and everyone. ---> If you want to pass through the video card (as evidenced by devctl and pptdevs) then why have the nvidia driver installed? Why did you say this ? Are you aware of some bug that prevents a pci device from being passed if its driver is installed ? Maybe there is. Infact I made some progress. I have commented this line on /etc/rc.conf : #kld_list="nvidia nvidia-modeset" and I've been able to detach and reattach the 4 slots of the 2080 ti. I think that if I load the nvidia modules on the rc.conf,they interfere with a later attaching / detaching of 1 or 2 slots of the gpu (hdac and vgapci). Do you think that it could be caused by a bug on the nvidia driver or in the bhyve source code ? What do you suggest I do ? Il giorno mer 1 feb 2023 alle ore 13:49 Daniel O'Connor ha scritto: > > > > On 1 Feb 2023, at 21:41, Mario Marietto wrote: > > ---> Do you have 'vmm_load="YES"' in loader.conf ? > > > > yes. > > > > ---> If you want to pass through the video card (as evidenced by devctl > and pptdevs) then why have the nvidia driver installed? > > > > because I have 3 gpus on my PC. I use the intel gpu on the host ; I > would like to use the nvidia gtx 1060 within a bhyve Linux vm and the 2080 > ti with the Linuxulator to run stable diffusion,or vice versa. Stable > diffusion needs pytorch + cuda and they need the nVidia driver installed on > FreeBSD and the modules loaded. > > > > My goal is to run stable diffusion and a bhyve / linux vm +my nvidia > gpus attached. > > I note that https://wiki.freebsd.org/bhyve/pci_passthru says "Note: VGA / > GPU pass-through devices are not currently supported" so perhaps that is > why it is ignoring your pptdevs line. > > Although that said some googling suggests that is not the case so I don't > know, unfortunately I haven't tried it myself. > > There are some threads about it on the forums, eg: > https://forums.freebsd.org/threads/bhyve-gpu-pass-through.83152/ > > Those suggest that modifications to bhyve are required before it will work > properly but none of them are for vmm that I can see. > > -- > Daniel O'Connor > "The nice thing about standards is that there > are so many of them to choose from." > -- Andrew Tanenbaum > > -- Mario. --0000000000000fc01705f3cb609f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello Daniel and everyone.

 ---> If you want to pass through the video card (as evidenced by devctl = and pptdevs) then why have the nvidia driver installed?

Why did you say this ? Are you aware of some bug that prevents a pci = device from being passed if its driver is installed ?

Maybe there is. Infact I made some progress. I have commented this= line on /etc/rc.conf :

#kld_list=3D"nvidia nvidia-modeset"

and I've been able to detach and reattach the 4 slots of the 2080 ti. I=20 think that if I load the nvidia modules on the rc.conf,they interfere=20 with a later attaching / detaching of 1 or 2 slots of the gpu (hdac and=20 vgapci). Do you think that it could be caused by a bug on the nvidia driver= or in the bhyve source code ? What do you suggest I do ?
=

Il giorno mer 1 feb 2023 alle ore 13:49 Daniel O'Connor <darius@dons.net.au> ha scritto:
=


> On 1 Feb 2023, at 21:41, Mario Marietto <marietto2008@gmail.com> wrote:
> ---> Do you have 'vmm_load=3D"YES"' in loader.con= f ?
>
> yes.
>
> ---> If you want to pass through the video card (as evidenced by de= vctl and pptdevs) then why have the nvidia driver installed?
>
> because I have 3 gpus on my PC. I use the intel gpu on the host ; I wo= uld like to use the nvidia gtx 1060 within a bhyve Linux vm and the 2080 ti= with the Linuxulator to run stable diffusion,or vice versa. Stable diffusi= on needs pytorch + cuda and they need the nVidia driver installed on FreeBS= D and the modules loaded.
>
> My goal is to run stable diffusion and a bhyve / linux vm +my nvidia g= pus attached.

I note that https://wiki.freebsd.org/bhyve/pci_passthru says "Note: VGA / GPU pass-through devices are not currently suppor= ted" so perhaps that is why it is ignoring your pptdevs line.

Although that said some googling suggests that is not the case so I don'= ;t know, unfortunately I haven't tried it myself.

There are some threads about it on the forums, eg:
https://forums.freebsd.org/threads/b= hyve-gpu-pass-through.83152/

Those suggest that modifications to bhyve are required before it will work = properly but none of them are for vmm that I can see.

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
-- Andrew Tanenbaum



--
Mario.
--0000000000000fc01705f3cb609f-- From nobody Fri Feb 3 13:17:20 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P7bmr4FbBz2p6JF for ; Fri, 3 Feb 2023 13:17:48 +0000 (UTC) (envelope-from darius@dons.net.au) Received: from midget.dons.net.au (2403-5800-5200-4700-225-90ff-fe47-39b4.ip6.aussiebb.net [IPv6:2403:5800:5200:4700:225:90ff:fe47:39b4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "dons.net.au", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P7bmq5ybnz40wS for ; Fri, 3 Feb 2023 13:17:46 +0000 (UTC) (envelope-from darius@dons.net.au) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (2403-5800-5200-4700-d5f-4320-bb5b-a71.ip6.aussiebb.net [IPv6:2403:5800:5200:4700:d5f:4320:bb5b:a71]) (authenticated bits=0) by midget.dons.net.au (8.17.1/8.15.2) with ESMTPSA id 313DHVex074684 (version=TLSv1.2 cipher=ECDHE-ECDSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 3 Feb 2023 23:47:31 +1030 (ACDT) (envelope-from darius@dons.net.au) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=dons.net.au; s=default; t=1675430251; bh=FyLvgyXB446Tpu8fJ/SdQbUKYw8xhwZwrqD964ekPWE=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=YjKXQEmpNyAmMGNUWqnCNGvuluI4nHMbEx6DQ0sZJYHM3Z9f7WYINwPnqCBAuCXYn uoIvJ0AjeFovxmdZSvj7g99z8I2It6gOfbbr7FeTfYYIAlX/invj7XwzMzO86elGhf Exr7/m3HqTBJoeO4iDiLcwe9oXaho+jVwXJjF+7Q= X-Authentication-Warning: midget.dons.net.au: Host 2403-5800-5200-4700-d5f-4320-bb5b-a71.ip6.aussiebb.net [IPv6:2403:5800:5200:4700:d5f:4320:bb5b:a71] claimed to be smtpclient.apple Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy From: "Daniel O'Connor" In-Reply-To: Date: Fri, 3 Feb 2023 23:47:20 +1030 Cc: freebsd-hackers Content-Transfer-Encoding: quoted-printable Message-Id: <4E67D54E-CF0E-4A56-9B6B-51C359B7CEE2@dons.net.au> References: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au> To: Mario Marietto X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spam-Status: No, score=0.40 X-Rspamd-Server: midget.dons.net.au X-Rspamd-Queue-Id: 4P7bmq5ybnz40wS X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:4764, ipnet:2403:5800::/32, country:AU] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On 3 Feb 2023, at 23:37, Mario Marietto = wrote: > ---> If you want to pass through the video card (as evidenced by = devctl and pptdevs) then why have the nvidia driver installed? >=20 > Why did you say this ? Are you aware of some bug that prevents a pci = device from being passed if its driver is installed ?=20 No, it just seems pointless to have a driver for a card that it will = never attach to because it is 'stolen' by the passthrough driver. > Maybe there is. Infact I made some progress. I have commented this = line on /etc/rc.conf : >=20 > #kld_list=3D"nvidia nvidia-modeset" >=20 > and I've been able to detach and reattach the 4 slots of the 2080 ti. = I think that if I load the nvidia modules on the rc.conf,they interfere = with a later attaching / detaching of 1 or 2 slots of the gpu (hdac and = vgapci). Do you think that it could be caused by a bug on the nvidia = driver or in the bhyve source code ? What do you suggest I do ? It seems very odd that loading the nvidia driver via rc.conf would = 'beat' the passthrough driver loaded earlier in loader.conf. However I've never tried any pass through stuff so no idea sorry. -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum From nobody Fri Feb 3 13:22:39 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P7bv91HYXz2p76P for ; Fri, 3 Feb 2023 13:23:17 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x636.google.com (mail-ej1-x636.google.com [IPv6:2a00:1450:4864:20::636]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P7bv90jSHz425f for ; Fri, 3 Feb 2023 13:23:17 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x636.google.com with SMTP id mc11so15229027ejb.10 for ; Fri, 03 Feb 2023 05:23:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=0YfC6RwCZFg/jReBvibYHOmMdr2eSB3Tkh1XD9+4Qd4=; b=PuKF+lONek6dUyMqpxJ2gVwZQjA3G9K+qG/kJxvR7aVFSjNcXNoMl9b145ajNJB8bC 50Flq8U17HbIUJTCCuPJ88qhC4iKwsxwU/xvEZZM05CNFdkfdm5qZlHzbBlinqHxzaJM EMW5/rnTWOdNKfiRLC133X9eeYk5feKq/MWpd7EiuySb2NO3ZLPddxtDcpOR3VU7Hi6q 96tKvSLHK/Sc4vYZShtfz6Iu+5+8sP7H08s2A3TILn2OMdRfVLl3QkP8gzBD57rqwQXU PXf7Ea/whDG4s3EUhOqsR8g4m2AOxbEwmEUxIr6yAmCFir/pHNMB0QL78HPRsBEOKTqv 04Yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=0YfC6RwCZFg/jReBvibYHOmMdr2eSB3Tkh1XD9+4Qd4=; b=Psezh6bFN2O0GpatmW7+u2XrkI2Y5zOs0JSWIQtU/Ak05ZjQutTKpovRE/w+AAUtQ3 Bxfo6/1M6kW6aOOes2syQIphoTjauTUHSBaogavVDNwrpjUv/hvgCWmaK6T5GPX9IEKG J+Q6NJC/uwEnw7/QW7QcpeH5smMxaegYZfUFR+oKy1PKtmW2XRZXpXwuo0RajBwOURPC CRtfnL1cGZ/W6GF3Oqbq+683FVzodR9Am6wxEysJRN/XxMORuUP7KPhlsIKR8+BkIpx+ ItjFDtQ5ozMU33aEXzsUw4QNsqcmZOmPw+6d1aD3+A6THTAMPvo9HWsmACw9ej7utFzG 5n8Q== X-Gm-Message-State: AO0yUKXNJQ9gY1EJv0+kN9qB4njGFv3FooYRElApljhjh5rpTUTcpXBI ElFKYviCQLrDIUf9KymY9xLkcDVBzfh4ERNu968emXtRWhLF4Q== X-Google-Smtp-Source: AK7set+cXWO+S1LiUe6RUohIfQnKeSsRC9m+FoLTstoYLHxT6HVx/9VAIG+lBNGg4r8YsXhfOs4VW7hs/SvhI/4rPxU= X-Received: by 2002:a17:906:fc16:b0:84d:4d2d:1c5f with SMTP id ov22-20020a170906fc1600b0084d4d2d1c5fmr3009340ejb.114.1675430595830; Fri, 03 Feb 2023 05:23:15 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au> <4E67D54E-CF0E-4A56-9B6B-51C359B7CEE2@dons.net.au> In-Reply-To: <4E67D54E-CF0E-4A56-9B6B-51C359B7CEE2@dons.net.au> From: Mario Marietto Date: Fri, 3 Feb 2023 14:22:39 +0100 Message-ID: Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy To: "Daniel O'Connor" Cc: freebsd-hackers Content-Type: multipart/alternative; boundary="00000000000082a63305f3cb957a" X-Rspamd-Queue-Id: 4P7bv90jSHz425f X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --00000000000082a63305f3cb957a Content-Type: text/plain; charset="UTF-8" To put the pci addresses under ppt using the loader.conf always worked for me. I would like to understand why it does not work anymore for some months. Maybe some new feature and some obscure bug has been introduced inside the bhyve source code. This is important to understand why I'm not able to attach / detach the nVidia driver from the host to the guest os and it lets me think that your consideration may be relevant in some way. Il giorno ven 3 feb 2023 alle ore 14:17 Daniel O'Connor ha scritto: > > > > On 3 Feb 2023, at 23:37, Mario Marietto wrote: > > ---> If you want to pass through the video card (as evidenced by devctl > and pptdevs) then why have the nvidia driver installed? > > > > Why did you say this ? Are you aware of some bug that prevents a pci > device from being passed if its driver is installed ? > > No, it just seems pointless to have a driver for a card that it will never > attach to because it is 'stolen' by the passthrough driver. > > > Maybe there is. Infact I made some progress. I have commented this line > on /etc/rc.conf : > > > > #kld_list="nvidia nvidia-modeset" > > > > and I've been able to detach and reattach the 4 slots of the 2080 ti. I > think that if I load the nvidia modules on the rc.conf,they interfere with > a later attaching / detaching of 1 or 2 slots of the gpu (hdac and vgapci). > Do you think that it could be caused by a bug on the nvidia driver or in > the bhyve source code ? What do you suggest I do ? > > It seems very odd that loading the nvidia driver via rc.conf would 'beat' > the passthrough driver loaded earlier in loader.conf. > > However I've never tried any pass through stuff so no idea sorry. > > -- > Daniel O'Connor > "The nice thing about standards is that there > are so many of them to choose from." > -- Andrew Tanenbaum > > -- Mario. --00000000000082a63305f3cb957a Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
To put the pci addresses under ppt using the loader.= conf=20 always worked for me. I would like to understand why it does not work=20 anymore for some months. Maybe some new feature and some obscure bug has be= en introduced inside the bhyve source code. This is important to understand= why I'm not able to attach /=20 detach the nVidia driver from the host to the guest os and it lets me think= that your consideration may be relevant in some way.

Il giorno ven 3= feb 2023 alle ore 14:17 Daniel O'Connor <darius@dons.net.au> ha scritto:


> On 3 Feb 2023, at 23:37, Mario Marietto <marietto2008@gmail.com> wrote:
> ---> If you want to pass through the video card (as evidenced by de= vctl and pptdevs) then why have the nvidia driver installed?
>
> Why did you say this ? Are you aware of some bug that prevents a pci d= evice from being passed if its driver is installed ?

No, it just seems pointless to have a driver for a card that it will never = attach to because it is 'stolen' by the passthrough driver.

> Maybe there is. Infact I made some progress. I have commented this lin= e on /etc/rc.conf :
>
> #kld_list=3D"nvidia nvidia-modeset"
>
> and I've been able to detach and reattach the 4 slots of the 2080 = ti. I think that if I load the nvidia modules on the rc.conf,they interfere= with a later attaching / detaching of 1 or 2 slots of the gpu (hdac and vg= apci). Do you think that it could be caused by a bug on the nvidia driver o= r in the bhyve source code ? What do you suggest I do ?

It seems very odd that loading the nvidia driver via rc.conf would 'bea= t' the passthrough driver loaded earlier in loader.conf.

However I've never tried any pass through stuff so no idea sorry.

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
-- Andrew Tanenbaum



--
Mario.
--00000000000082a63305f3cb957a-- From nobody Fri Feb 3 13:27:15 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P7c0209Xpz2p6w2 for ; Fri, 3 Feb 2023 13:27:30 +0000 (UTC) (envelope-from darius@dons.net.au) Received: from midget.dons.net.au (2403-5800-5200-4700-225-90ff-fe47-39b4.ip6.aussiebb.net [IPv6:2403:5800:5200:4700:225:90ff:fe47:39b4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature ECDSA (P-384) client-digest SHA384) (Client CN "dons.net.au", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P7c013TDBz43gx for ; Fri, 3 Feb 2023 13:27:29 +0000 (UTC) (envelope-from darius@dons.net.au) Authentication-Results: mx1.freebsd.org; none Received: from smtpclient.apple (2403-5800-5200-4700-d5f-4320-bb5b-a71.ip6.aussiebb.net [IPv6:2403:5800:5200:4700:d5f:4320:bb5b:a71]) (authenticated bits=0) by midget.dons.net.au (8.17.1/8.15.2) with ESMTPSA id 313DRQIg075384 (version=TLSv1.2 cipher=ECDHE-ECDSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 3 Feb 2023 23:57:26 +1030 (ACDT) (envelope-from darius@dons.net.au) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=dons.net.au; s=default; t=1675430846; bh=zzdi1ilUdspVUTWVOJGzdU5ro0sHnhwuGKwW6K7IYhY=; h=Subject:From:In-Reply-To:Date:Cc:References:To; b=aUmCrdFBWo7um1r07vglSXKNSCRQqEwowdEjUZdcR/g3JVGzpNtrObOL6MS0h/E5F G8MFYrUaO5pZfjwzQJzQ93K4JmE2+ysPxMXd1KCt9j0PUPeHRwHBHR68K/+MT//FVE JSTaI67oj9CfWXIyLtLGtb7jmGvaXDDq/wimZ+Hg= X-Authentication-Warning: midget.dons.net.au: Host 2403-5800-5200-4700-d5f-4320-bb5b-a71.ip6.aussiebb.net [IPv6:2403:5800:5200:4700:d5f:4320:bb5b:a71] claimed to be smtpclient.apple Content-Type: text/plain; charset=us-ascii List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3731.400.51.1.1\)) Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy From: "Daniel O'Connor" In-Reply-To: Date: Fri, 3 Feb 2023 23:57:15 +1030 Cc: freebsd-hackers Content-Transfer-Encoding: quoted-printable Message-Id: References: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au> <4E67D54E-CF0E-4A56-9B6B-51C359B7CEE2@dons.net.au> To: Mario Marietto X-Mailer: Apple Mail (2.3731.400.51.1.1) X-Spam-Status: No, score=0.40 X-Rspamd-Server: midget.dons.net.au X-Rspamd-Queue-Id: 4P7c013TDBz43gx X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:4764, ipnet:2403:5800::/32, country:AU] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N > On 3 Feb 2023, at 23:52, Mario Marietto = wrote: > To put the pci addresses under ppt using the loader.conf always worked = for me. I would like to understand why it does not work anymore for some = months. Maybe some new feature and some obscure bug has been introduced = inside the bhyve source code. This is important to understand why I'm = not able to attach / detach the nVidia driver from the host to the guest = os and it lets me think that your consideration may be relevant in some = way. It would have been more helpful if you had said up front that it used to = work for you.. Do you mean it used to work for these nvidia devices or something else, = or..? -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum From nobody Fri Feb 3 14:17:19 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P7d6F3Xfdz2pB7k for ; Fri, 3 Feb 2023 14:17:57 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x633.google.com (mail-ej1-x633.google.com [IPv6:2a00:1450:4864:20::633]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P7d6F0Ny1z4GX4 for ; Fri, 3 Feb 2023 14:17:57 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x633.google.com with SMTP id hx15so15647105ejc.11 for ; Fri, 03 Feb 2023 06:17:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=MfGfhdavec9GewEU4JSBpHsAzmOA5ti4jwEdnVQAKqE=; b=pNdNvsAbCkgWjFTzB5d7ebFujC6p3YIV0ji0TZqp7CZwjgYhNGA8oyhG2STNMaHY6G pVyteBWiwV2y7buUSIxzQVWkkw0JDNztCXCmvRGO3ZIrdwA/qCQCJ6NJYrQcKAR/pAPZ ArPz+qtv8eH0c3yqzdr8thsiexoIfcsSdX/HtDZI1Noa7UtVp85LRyXMMisqj480Z1ax cmJv1gozL8NKYZy+W9DdKoVfQkHbn2FxlheyHNXXFqx9KtlWnWivcslJPZTa7kvMpa9Q Y0en5W6hjQCSVoF0iIvDfjU7MI/YkgAwc9yhQeMuUcRRbGEYrpBYBlP1QTfDYKpWP+2U NaOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=MfGfhdavec9GewEU4JSBpHsAzmOA5ti4jwEdnVQAKqE=; b=Eau9b3Dyljgy8Zg4vl/dh9QT3BpZ9Fl1cIlEIAPDkIXw/FJ04ERvM8dv+NqGQTVphg CKZqHdsRLWeiuoD3uo3XYOZJnZcOesaM+kLba/nuasDQsVQEH8BHJTCD+9/cQuTl5RcY yjmDGI1o8j4/Pbx7eZLxFVSR5RNNMoyTiuUfyXAQHq9hVLaPxqSNs4qd5OlTa5X3LY5A FvdoiDYAQkQHOiEEVzRhVvAb22XgogGOQD+89oOdSDT4ackFcu/zeFvVO8gFq2JLwbJG QMX6Fx7rt3g4unO8RpThABmxKl3ZKlwl6iyPSpxd86UTDOC3ZcSOvbO0bbgisBJ6zpw3 KylQ== X-Gm-Message-State: AO0yUKUsiTm5lJi12JUa9kUTplqhtq0/2Ai6Zn7sc+ExUdNJkto3bQsb 5tBMs3hwLYHG9g39LzVP9DJk6SN6i8klDbnY0YI6AV2CMmmXFg== X-Google-Smtp-Source: AK7set+NFIT5zZOL5hLRKsfxIiSS7bn+0mYCCjs5QkQIYBVAY9c/9eQ4NJoU6/ovanAvIW1AqS41/y3bjcDsqbh5Ln4= X-Received: by 2002:a17:907:cb25:b0:880:b580:d93 with SMTP id um37-20020a170907cb2500b00880b5800d93mr3111128ejc.276.1675433875645; Fri, 03 Feb 2023 06:17:55 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au> <4E67D54E-CF0E-4A56-9B6B-51C359B7CEE2@dons.net.au> In-Reply-To: From: Mario Marietto Date: Fri, 3 Feb 2023 15:17:19 +0100 Message-ID: Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy To: "Daniel O'Connor" Cc: freebsd-hackers Content-Type: multipart/alternative; boundary="00000000000000a96f05f3cc59f4" X-Rspamd-Queue-Id: 4P7d6F0Ny1z4GX4 X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --00000000000000a96f05f3cc59f4 Content-Type: text/plain; charset="UTF-8" Very interesting behavior. I've added this new line in /etc/rc.conf : (until some minutes ago I didn't use it because I was using the intel driver,installed by the package : xf86-video-intel-2.99.917.916_2,1) kld_list="i915kms acpi_video" and this line in /boot/loader.conf : pptdevs="1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3 5/0/0" and boom. pptdevs worked again: # pciconf -l ppt0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1c02 subvendor=0x19da subdevice=0x2438 ppt1@pci0:1:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10f1 subvendor=0x19da subdevice=0x2438 ppt2@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1e04 subvendor=0x19da subdevice=0x2503 ppt3@pci0:2:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de device=0x10f7 subvendor=0x19da subdevice=0x2503 ppt4@pci0:2:0:2: class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad6 subvendor=0x19da subdevice=0x2503 ppt5@pci0:2:0:3: class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de device=0x1ad7 subvendor=0x19da subdevice=0x2503 So,maybe there is some incompatibility between the intel and the pptdevs driver ? take also in consideration that on xorg.cong I'm still using the intel driver : Section "Device" Identifier "Card0" Driver "intel" BusID "PCI:0:2:0" EndSection. It's a confusing situation. Il giorno ven 3 feb 2023 alle ore 14:27 Daniel O'Connor ha scritto: > > > > On 3 Feb 2023, at 23:52, Mario Marietto wrote: > > To put the pci addresses under ppt using the loader.conf always worked > for me. I would like to understand why it does not work anymore for some > months. Maybe some new feature and some obscure bug has been introduced > inside the bhyve source code. This is important to understand why I'm not > able to attach / detach the nVidia driver from the host to the guest os and > it lets me think that your consideration may be relevant in some way. > > It would have been more helpful if you had said up front that it used to > work for you.. > > Do you mean it used to work for these nvidia devices or something else, > or..? > > -- > Daniel O'Connor > "The nice thing about standards is that there > are so many of them to choose from." > -- Andrew Tanenbaum > > -- Mario. --00000000000000a96f05f3cc59f4 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Very interesting= behavior. I've added this new line in /etc/rc.conf : (until some minut= es ago I didn't use it because I was using the intel driver,installed b= y the package : xf86-video-intel-2.99.917.916_2,1)

kld_list=3D"i915kms acpi_video"

and this line in /boot/loader.conf = :

<= /span>
pptdevs=3D"1/0= /0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3 5/0/0"

<= /div>
and boom. pptdevs worked aga= in:

# pciconf -l

ppt0@pci0:1:0:0: =C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x030000 rev=3D0xa1 hdr=3D0x00 = vendor=3D0x10de device=3D0x1c02 subvendor=3D0x19da subdevice=3D0x2438
ppt1@pci0:1:0:1: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0class=3D0x040300 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device= =3D0x10f1 subvendor=3D0x19da subdevice=3D0x2438
ppt2@pci0:2:0:0: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x03= 0000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e04 subvendor=3D0x19= da subdevice=3D0x2503
ppt3@pci0:2:0:1: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x04= 0300 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x10f7 subvendor=3D0x19= da subdevice=3D0x2503
ppt4@pci0:2:0:2: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x0c= 0330 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1ad6 subvendor=3D0x19= da subdevice=3D0x2503
ppt5@pci0:2:0:3: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x0c= 8000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1ad7 subvendor=3D0x19= da subdevice=3D0x2503

So,maybe there i= s some incompatibility between the intel and the pptdevs driver ? take also= in consideration that on xorg.cong I'm still using the intel driver :<= br>

Section "Device"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Identifier =C2=A0&quo= t;Card0"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Driver =C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0"intel"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0BusID =C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0"PCI:0:2:0"
EndSection.

It's a confusing situation.

Il g= iorno ven 3 feb 2023 alle ore 14:27 Daniel O'Connor <darius@dons.net.au> ha scri= tto:


> On 3 Feb 2023, at 23:52, Mario Marietto <marietto2008@gmail.com> wrote:
> To put the pci addresses under ppt using the loader.conf always worked= for me. I would like to understand why it does not work anymore for some m= onths. Maybe some new feature and some obscure bug has been introduced insi= de the bhyve source code. This is important to understand why I'm not a= ble to attach / detach the nVidia driver from the host to the guest os and = it lets me think that your consideration may be relevant in some way.

It would have been more helpful if you had said up front that it used to wo= rk for you..

Do you mean it used to work for these nvidia devices or something else, or.= .?

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
-- Andrew Tanenbaum



--
Mario.
=
--00000000000000a96f05f3cc59f4-- From nobody Fri Feb 3 14:20:43 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P7dBB0mmzz2pBD0 for ; Fri, 3 Feb 2023 14:21:22 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x52e.google.com (mail-ed1-x52e.google.com [IPv6:2a00:1450:4864:20::52e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P7dB932M8z4HKs for ; Fri, 3 Feb 2023 14:21:21 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b="bo3Gh/ZA"; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::52e as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x52e.google.com with SMTP id f7so5322645edw.5 for ; Fri, 03 Feb 2023 06:21:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=PE7pGeRJKfzVPr/tgJI7WtjLz2gvLr5LYWWdLP1Fbvs=; b=bo3Gh/ZANvpiq8SkGhaKjZs6VvrfUgWA4n+4xmLNJ6ix6s1Lk6ZRX7G87hTLKcsO+6 i84XG+RjpQUJCKjONX9AaQpCY/6F8C5GupOYuHFoMgodAB3Gw8SV/WdP1j/IkaS24brE 8BCwVTD+v2+KZ2aau3axq4qakJvdfu0wvKlDMIpuWaY8sYrtSXN2MMETQfZMQ00DRapo eKfxSi6aKXF4KtizAUl7udoM7Afrj9aMxFBezJolJu1/TJjSNUYFIvDGqsaVPQPaBVe3 1A1iOgVgHqV1md0hQeveBcidkjyPpWJOplvhIpob5DpFo9B0UVHI/RrnpR82vlq4t7lY nLdw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=PE7pGeRJKfzVPr/tgJI7WtjLz2gvLr5LYWWdLP1Fbvs=; b=0e/qxppHhB1rzQ3QIGmvqSpHgcb4JT5ZnygcIS0YR2/MUvC7Rai61C8m/W7lR2amXc 6f26QiFRoKTzY9KLd9YN1+p5tLdzzB2QMc36802afVVbwWRmZdoBe+Ep2tNk/aYhSEAJ lm8JULXqGT+JGiZOkGkG5urPJSKuPjCvryp2kKEFngmJh2w4muZcISdCqQZtPFGoTmsq FtC1Yvd7AFpmh4aqTrv8QEaZ639swUBqbyBkpaDCtaL9WVIwb2YIrAk/i2c06SlMMB4t PouFE+6DTNN83OCjQbyARoi8+t4d7ozgidWxh67wOP5+bHlw7RyRvpmMhktFONtDF5iL IoNw== X-Gm-Message-State: AO0yUKVy7dVljJMI3WmZbuBtI8sB3SIDr7qI5VYStbx73+Tw7C5lFMrU Lb8l0jvbscb9i0uhzR43EfHYkRdhacnze/dEktObkdAlo+s= X-Google-Smtp-Source: AK7set9YQbNNMRg0XRVt/PpPVhgLY4KucSADadkrqzoSHdSBsMnbfG7fMAyGuVIxzHguFpPkaOIUNiqz/4JJdAVBDl4= X-Received: by 2002:a05:6402:17cf:b0:4a1:bb88:94b7 with SMTP id s15-20020a05640217cf00b004a1bb8894b7mr3146822edy.65.1675434080012; Fri, 03 Feb 2023 06:21:20 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <09A3D05B-E269-437F-8ACB-A150EBCACF4E@dons.net.au> <128F6D81-D71B-4BC7-9143-8205E826088F@dons.net.au> <4E67D54E-CF0E-4A56-9B6B-51C359B7CEE2@dons.net.au> In-Reply-To: From: Mario Marietto Date: Fri, 3 Feb 2023 15:20:43 +0100 Message-ID: Subject: Re: devctl: Failed to detach pci0:1:0:0: Device busy / devctl: Failed to set pci0:1:0:0 driver to ppt: Device busy To: "Daniel O'Connor" Cc: freebsd-hackers Content-Type: multipart/alternative; boundary="0000000000002f0bf005f3cc6574" X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-0.999]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; MID_RHS_MATCH_FROMTLD(0.00)[]; MLMMJ_DEST(0.00)[freebsd-hackers@freebsd.org]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::52e:from]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_COUNT_TWO(0.00)[2]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; TO_DN_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4P7dB932M8z4HKs X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --0000000000002f0bf005f3cc6574 Content-Type: text/plain; charset="UTF-8" I forgot that the pptdevs driver covers also the slot of the USB controller : ppt6@pci0:5:0:0: class=0x0c0330 rev=0x10 hdr=0x00 vendor=0x1b73 device=0x1100 subvendor=0x1b73 subdevice=0x1100 Il giorno ven 3 feb 2023 alle ore 15:17 Mario Marietto < marietto2008@gmail.com> ha scritto: > Very interesting behavior. I've added this new line in /etc/rc.conf : > (until some minutes ago I didn't use it because I was using the intel > driver,installed by the package : xf86-video-intel-2.99.917.916_2,1) > > kld_list="i915kms acpi_video" > > and this line in /boot/loader.conf : > > pptdevs="1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3 5/0/0" > > and boom. pptdevs worked again: > > # pciconf -l > > ppt0@pci0:1:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1c02 subvendor=0x19da subdevice=0x2438 > ppt1@pci0:1:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x10f1 subvendor=0x19da subdevice=0x2438 > ppt2@pci0:2:0:0: class=0x030000 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1e04 subvendor=0x19da subdevice=0x2503 > ppt3@pci0:2:0:1: class=0x040300 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x10f7 subvendor=0x19da subdevice=0x2503 > ppt4@pci0:2:0:2: class=0x0c0330 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1ad6 subvendor=0x19da subdevice=0x2503 > ppt5@pci0:2:0:3: class=0x0c8000 rev=0xa1 hdr=0x00 vendor=0x10de > device=0x1ad7 subvendor=0x19da subdevice=0x2503 > > So,maybe there is some incompatibility between the intel and the pptdevs > driver ? take also in consideration that on xorg.cong I'm still using the > intel driver : > > Section "Device" > Identifier "Card0" > Driver "intel" > BusID "PCI:0:2:0" > EndSection. > > It's a confusing situation. > > Il giorno ven 3 feb 2023 alle ore 14:27 Daniel O'Connor < > darius@dons.net.au> ha scritto: > >> >> >> > On 3 Feb 2023, at 23:52, Mario Marietto wrote: >> > To put the pci addresses under ppt using the loader.conf always worked >> for me. I would like to understand why it does not work anymore for some >> months. Maybe some new feature and some obscure bug has been introduced >> inside the bhyve source code. This is important to understand why I'm not >> able to attach / detach the nVidia driver from the host to the guest os and >> it lets me think that your consideration may be relevant in some way. >> >> It would have been more helpful if you had said up front that it used to >> work for you.. >> >> Do you mean it used to work for these nvidia devices or something else, >> or..? >> >> -- >> Daniel O'Connor >> "The nice thing about standards is that there >> are so many of them to choose from." >> -- Andrew Tanenbaum >> >> > > -- > Mario. > -- Mario. --0000000000002f0bf005f3cc6574 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
I forgot that th= e pptdevs driver covers also the slot of the USB controller :

ppt6@pci0:5:0:0: =C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x0c0330 rev=3D0x10 hdr=3D0x00 vendor=3D= 0x1b73 device=3D0x1100 subvendor=3D0x1b73 subdevice=3D0x1100<= /div>

Il giorno ven = 3 feb 2023 alle ore 15:17 Mario Marietto <marietto2008@gmail.com> ha scritto:
<= div>Very interesting behavior. I'v= e added this new line in /etc/rc.conf : (until some minutes ago I didn'= t use it because I was using the intel driver,installed by the package : xf= 86-video-intel-2.99.917.916_2,1)

<= /span>
= kld_list= =3D"i915kms acpi_video"

and this line in /boot/loader.conf :

<= span style=3D"font-family:arial,sans-serif"><= span style=3D"font-family:arial,sans-serif">pptdevs=3D"1/0/0 1/0/1 2/0/0 2/0/1= 2/0/2 2/0/3 5/0/0"
<= span style=3D"font-family:arial,sans-serif">
and boom. pptdevs worked again:

<= span style=3D"font-family:arial,sans-serif"># pciconf -l

ppt0@pci0:1:0:0: =C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0class=3D0x030000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de devic= e=3D0x1c02 subvendor=3D0x19da subdevice=3D0x2438
ppt1@pci0:1:0:1: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0class=3D0x040300 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device= =3D0x10f1 subvendor=3D0x19da subdevice=3D0x2438
ppt2@pci0:2:0:0: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x03= 0000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e04 subvendor=3D0x19= da subdevice=3D0x2503
ppt3@pci0:2:0:1: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x04= 0300 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x10f7 subvendor=3D0x19= da subdevice=3D0x2503
ppt4@pci0:2:0:2: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x0c= 0330 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1ad6 subvendor=3D0x19= da subdevice=3D0x2503
ppt5@pci0:2:0:3: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x0c= 8000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1ad7 subvendor=3D0x19= da subdevice=3D0x2503

So,maybe there i= s some incompatibility between the intel and the pptdevs driver ? take also= in consideration that on xorg.cong I'm still using the intel driver :<= br>

Section "Device"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Identifier =C2=A0&quo= t;Card0"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Driver =C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0"intel"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0BusID =C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0"PCI:0:2:0"
EndSection.

It's a confusing situation.

Il g= iorno ven 3 feb 2023 alle ore 14:27 Daniel O'Connor <darius@dons.net.au> ha scri= tto:


> On 3 Feb 2023, at 23:52, Mario Marietto <marietto2008@gmail.com> wrote:
> To put the pci addresses under ppt using the loader.conf always worked= for me. I would like to understand why it does not work anymore for some m= onths. Maybe some new feature and some obscure bug has been introduced insi= de the bhyve source code. This is important to understand why I'm not a= ble to attach / detach the nVidia driver from the host to the guest os and = it lets me think that your consideration may be relevant in some way.

It would have been more helpful if you had said up front that it used to wo= rk for you..

Do you mean it used to work for these nvidia devices or something else, or.= .?

--
Daniel O'Connor
"The nice thing about standards is that there
are so many of them to choose from."
-- Andrew Tanenbaum



--
Mario.
=


--
Mario.
=
--0000000000002f0bf005f3cc6574-- From nobody Fri Feb 3 17:09:40 2023 X-Original-To: freebsd-hackers@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4P7hx74T8Hz3kX8B; Fri, 3 Feb 2023 17:10:19 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ed1-x529.google.com (mail-ed1-x529.google.com [IPv6:2a00:1450:4864:20::529]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4P7hx66RCrz41q8; Fri, 3 Feb 2023 17:10:18 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=SmIP3s8e; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2a00:1450:4864:20::529 as permitted sender) smtp.mailfrom=marietto2008@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-ed1-x529.google.com with SMTP id u21so5842486edv.3; Fri, 03 Feb 2023 09:10:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :from:to:cc:subject:date:message-id:reply-to; bh=m0VVQL1QfIEeYWx3l6GSicSwdKzAImU44xxtq2ED4ww=; b=SmIP3s8ehsiOj/ZIYJQ6g/eYFv9UmmJlaQvUQvBcrZN4kRxCHTptWj6eS/qq7JGF5/ BneI4IEHS+foEIuCfysphz2FFVOqFHyE5P6nwVOFFqysysNZSWn51bJcKf6uGSLUiNCM 6mMq/AdNYSrRB2enpblen86OjbGe35PLwUgKZdEw+9trCF5rDlzj3Q2T4uP1X0makQ7a 0a11XAcvqxKUMMgR89ytF82+CDhArWA+YSxExSrHykcAxin8wdyUpKVQ2lX50ElkhW/N uOu3fUiccXevXYHzM1f2PMMzGzPDLd6DK2jB5JU2pyQuJvM2l23ne9hZf/lACB4OMcd9 uRFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:from:in-reply-to:references:mime-version :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=m0VVQL1QfIEeYWx3l6GSicSwdKzAImU44xxtq2ED4ww=; b=xuyb9i4Ua/B823AY4hE/8I1cQIYkIU9XHADB7ba0jLeeAg/R0GvfUYvEH6PQSYLsX/ 5yARCfj+FHT+YAQFwZV7K3YjAMTudPAFPGy5YiJ8RYFk6yUISZDDH43zdXq6Eod1eu7u dmAwUzFfOlqK+QyEWOkh9ln4Zy90tmxC/bAdnjjoYPrtkG+H8nmlgfCWStkUPaIzmLvL akOpMWmWICUkbvGFuTQVN3kWoXYL+udDjSbNXbZGF1uUIlsnjvFON4FKAvhDQLzd6ZN+ CCzVVHV/qzxInYgRmduoycWqHoNdjaG1AlbLs457ag4fhb/zH6BVO6bAtZ7SwjbC6lUk /82w== X-Gm-Message-State: AO0yUKXem2V/ldv5+oZFKSMJk+qRKqroNWV2D6R1oH4+bWS3sDILaiKh Hs2ZmIIOX9n7v6C2fxlGvC6YmcFhxA5PO39oX1o= X-Google-Smtp-Source: AK7set/+nP2otbuRwcjHokJ+q4aJnbBrEPu2StIyX7N6Gbkf8fq9ldM/W+Q+HM8VcSVycMcRwPh306/D3HK9ln9xgqo= X-Received: by 2002:a50:cd56:0:b0:49c:a68c:8b6b with SMTP id d22-20020a50cd56000000b0049ca68c8b6bmr3406777edj.84.1675444217111; Fri, 03 Feb 2023 09:10:17 -0800 (PST) List-Id: Technical discussions relating to FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-hackers List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-hackers@freebsd.org MIME-Version: 1.0 References: <15af67efdab54e81bee59aedd2598839@beckhoff.com> <57a5ccfd7c3148f99da730b076a85e01@beckhoff.com> In-Reply-To: From: Mario Marietto Date: Fri, 3 Feb 2023 18:09:40 +0100 Message-ID: Subject: Re: devctl: Failed to detach pci0:2:0:0: Device busy To: =?UTF-8?Q?Corvin_K=C3=B6hne?= , FreeBSD virtualization , freebsd-hackers , Peter Grehan , - - Content-Type: multipart/alternative; boundary="00000000000066e96a05f3cec1de" X-Spamd-Result: default: False [-3.87 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.87)[-0.869]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[freebsd-virtualization@freebsd.org,freebsd-hackers@freebsd.org]; FREEMAIL_TO(0.00)[beckhoff.com,freebsd.org,yahoo.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::529:from]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_FROM(0.00)[gmail.com]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROMTLD(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_FIVE(0.00)[5]; TO_DN_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] X-Rspamd-Queue-Id: 4P7hx66RCrz41q8 X-Spamd-Bar: --- X-ThisMailContainsUnwantedMimeParts: N --00000000000066e96a05f3cec1de Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hello, bhyve and / or FreeBSD developers and lovers. I=E2=80=99m trying to pass my GTX 1060 from my host os (FreeBSD 13.1-RELEAS= E) to a Linux/Ubuntu VM virtualized with bhyve,but unsuccessfully. Last year I tried to do the same with another GPU that I have on my PC,the RTX 2080 ti and with the help of @generix ,after a tricky trouble shooting we have been able to do that. Point is that the same patches which worked (and that works right now) for the 2080 ti aren=E2=80=99t working fo= r the GTX 1060. The error that I see is the same that I saw when I have cooperated with Generix for passing through the 2080 ti. I really would like to know which differences there are between the two gpus because for the 1060 I'm using the same new driver that I=E2=80=99m using for the 2080 = ti and the latter one can be passed without problems. Anyway,I would know what I should do to fix that error. For this matter I've created a new post on the nvidia forum. If someone would like to give it a look,it is here : https://forums.developer.nvidia.com/t/rminitadapter-failed-error-produced-w= hen-i-try-to-passthru-my-gtx-1060-to-a-linux-ubuntu-bhyve-vm-instead-my-rtx= -2080ti-is-passed-correctly/241594 Il giorno ven 3 feb 2023 alle ore 15:21 Mario Marietto < marietto2008@gmail.com> ha scritto: > Very interesting behavior. I've added this new line in /etc/rc.conf : > (until some minutes ago I didn't use it because I was using the intel > driver,installed by the package : xf86-video-intel-2.99.917.916_2,1) > > kld_list=3D"i915kms acpi_video" > > and this line in /boot/loader.conf : > > pptdevs=3D"1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3 5/0/0" > > and boom. pptdevs worked again: > > # pciconf -l > > ppt0@pci0:1:0:0: class=3D0x030000 rev=3D0xa1 hdr=3D0x00 vendor=3D0= x10de > device=3D0x1c02 subvendor=3D0x19da subdevice=3D0x2438 > ppt1@pci0:1:0:1: class=3D0x040300 rev=3D0xa1 hdr=3D0x00 vendor=3D0= x10de > device=3D0x10f1 subvendor=3D0x19da subdevice=3D0x2438 > ppt2@pci0:2:0:0: class=3D0x030000 rev=3D0xa1 hdr=3D0x00 vendor=3D0= x10de > device=3D0x1e04 subvendor=3D0x19da subdevice=3D0x2503 > ppt3@pci0:2:0:1: class=3D0x040300 rev=3D0xa1 hdr=3D0x00 vendor=3D0= x10de > device=3D0x10f7 subvendor=3D0x19da subdevice=3D0x2503 > ppt4@pci0:2:0:2: class=3D0x0c0330 rev=3D0xa1 hdr=3D0x00 vendor=3D0= x10de > device=3D0x1ad6 subvendor=3D0x19da subdevice=3D0x2503 > ppt5@pci0:2:0:3: class=3D0x0c8000 rev=3D0xa1 hdr=3D0x00 vendor=3D0= x10de > device=3D0x1ad7 subvendor=3D0x19da subdevice=3D0x2503 > ppt6@pci0:5:0:0: class=3D0x0c0330 rev=3D0x10 hdr=3D0x00 vendor=3D0= x1b73 > device=3D0x1100 subvendor=3D0x1b73 subdevice=3D0x1100 > > So,maybe there is some incompatibility between the intel and the pptdevs > driver ? take also in consideration that on xorg.cong I'm still using the > intel driver : > > Section "Device" > Identifier "Card0" > Driver "intel" > BusID "PCI:0:2:0" > EndSection. > > It's a confusing situation. > > Il giorno ven 3 feb 2023 alle ore 14:13 Mario Marietto < > marietto2008@gmail.com> ha scritto: > >> I've sent an email to the hackers ML with this content : >> >> > On 30 Jan 2023, at 05:37, Mario Marietto >> wrote: >> > In some FreeBSD 13.1 machines I have the problem below,in some others = I >> don't have it. I would like to know what the causes could be and how to = fix >> it. >> > >> > root@marietto:/usr/home/marietto # devctl detach pci0:1:0:0 >> > >> > devctl: Failed to detach pci0:1:0:0: Device busy >> > >> > root@marietto:/usr/home/marietto # devctl detach pci0:2:0:0 >> > >> > devctl: Failed to detach pci0:2:0:0: Device busy >> > >> > Not even it works if instead of detach them, I try to attach them >> directly to the ppt driver : >> > >> > root@marietto:/usr/home/marietto/bhyve/Files # devctl set driver >> pci0:2:0:0 ppt >> > devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy >> > >> > root@marietto:/usr/home/marietto/bhyve/Files # devctl detach >> pci0:2:0:0 >> > devctl: Failed to detach pci0:2:0:0: Device busy >> > >> > Pci addresses 1:0:0 and 2:0:0 belong to the two GPUs that I have on my >> PC : >> > >> > vgapci0@pci0:1:0:0: class=3D0x030000 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de >> device=3D0x1c02 subvendor=3D0x19da subdevice=3D0x2438 >> > vgapci1@pci0:2:0:0: class=3D0x030000 rev=3D0xa1 hdr=3D0x00 vendor= =3D0x10de >> device=3D0x1e04 subvendor=3D0x19da subdevice=3D0x2503 >> > >> > Actually I have commented this line on /boot/loader.conf,because it >> makes no difference if I keep it uncommented or not. It is totally ignor= ed. >> > >> > #pptdevs=3D"1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3" >> >> and George replied to me like this : >> >> Do you have 'vmm_load=3D"YES"' in loader.conf? >> >> If not I am not sure it will have an effect - vmm must be loaded early >> otherwise the card will be grabbed by the vgapci driver rather than be p= ut >> under ppt control. >> >> I think this is an interesting answer. Yes,I keep vmm_load=3D"YES" in >> loader.conf,BUT putting the 4 slots under ppt control using the loader.c= onf >> did not work for me for months. We haven't taken this into consideration= . >> And you didn't reply when I've asked you if you have modified the bhyve >> source code,because putting the pci addresses under ppt using the >> loader.conf always worked for me. I would like to understand why it does >> not work anymore. This is important to understand why I'm not able to >> attach / detach the nVidia driver from the host to the guest os. And may= be >> George's consideration is relevant. >> >> Il giorno ven 3 feb 2023 alle ore 12:11 Corvin K=C3=B6hne < >> C.Koehne@beckhoff.com> ha scritto: >> >>> >>> - ---> Figure out if you can blacklist your gpu in the nvidia >>> driver. So, the nVidia driver never gets attached to your device. >>> >>> >>> - nope. I need to attach the nVidia driver to my gpu if I want to >>> run stable diffusion + pytorch + cuda on the Linuxulator on FreeBSD.= I'm >>> thinking of installing Stable Diffusion within the Linux VM,instead = of >>> doing it on FreeBSD. >>> >>> >>> - ---> Wait for drm drivers and use the nouveau driver. Nouveau >>> isn=E2=80=99t maintained by nVidia, so it=E2=80=99s more likely to g= et fixed. >>> >>> >>> - The nouveau driver is always bugged and it never worked well. I >>> wonder if it is still supported. It has even limited features. >>> >>> >>> >>> Afaik, you don=E2=80=99t have other options, sry. I=E2=80=99m no gpu dr= iver developer, >>> so I can=E2=80=99t help you. >>> >>> Btw: Do you refer to linux nouveau or freebsd nouveau? Freebsd made muc= h >>> progress in porting linux drm to freebsd with 14.0. So, if latest linux >>> nouveau fits your requirements, it=E2=80=99s likely that freebsd 14.0 m= ay fit them >>> too. >>> >>> >>> >>> - Are the following commands enough to update the bhyve source code >>> on my machine ? take in consideration that I need to grab only your = new >>> last patch. >>> >>> >>> - cd /usr/src >>> >>> >>> - git checkout -f origin/phab/corvink/13.1/nvidia-wip >>> >>> >>> - # cd /wherever/you/put/my/Makefile >>> >>> # make -j 8 >>> >>> >>> >>> My latest patch is a mouse driver patch for the host. So, no, you have >>> to use: >>> >>> cd /usr/src >>> >>> git fetch >>> >>> git checkout -f origin/phab/corvink/13.1/nvidia-wip >>> >>> make kernel >>> >>> reboot >>> >>> >>> >>> >>> >>> Best regards >>> >>> Corvin >>> >>> >>> >>> >>> >>> This email contains confidential information. If you have received it i= n >>> error, you must not read, use, copy or pass on this e-mail or its >>> attachments. >>> If you have received the e-mail in error, please inform me immediately >>> by reply e-mail and then delete this e-mail from your system. Thank you >>> >>> Diese E-Mail enthaelt vertrauliche Informationen. Sollten Sie sie >>> irrtuemlich erhalten haben, duerfen Sie diese E-Mail oder ihre Anhaenge >>> nicht lesen, verwenden, kopieren oder weitergeben. >>> Sollten Sie die Mail versehentlich erhalten haben, teilen Sie mir dies >>> bitte umgehend per Antwort-E-Mail mit und loeschen Sie diese E-Mail dan= n >>> aus Ihrem System. Vielen Dank >>> >>> Beckhoff Automation GmbH & Co. KG | Managing Director: Dipl. Phys. Hans >>> Beckhoff >>> Registered office: Verl, Germany | Register court: Guetersloh HRA 7075 >>> >>> >> >> -- >> Mario. >> > > > -- > Mario. > --=20 Mario. --00000000000066e96a05f3cec1de Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hello, bhyve and / or FreeBSD developer= s and lovers.

I=E2=80=99m trying to pass my GTX 1060 from my host= os (FreeBSD=20 13.1-RELEASE) to a Linux/Ubuntu VM virtualized with bhyve,but=20 unsuccessfully. Last year I tried to do the same with another GPU that I have on my PC,the RTX 2080 ti and with the help of @generix,aft= er a tricky trouble shooting we have been able to do that. Point is that=20 the same patches which worked (and that works right now) for the 2080 ti=20 aren=E2=80=99t working for the GTX 1060. The error that I see is the same t= hat I saw when I have cooperated with Generix for passing through the 2080 ti. I= really would like to know which differences there are between the two gpus= because for the 1060=20 I'm using the same new driver that I=E2=80=99m using for the 2080 ti an= d the=20 latter one can be passed without problems. Anyway,I would know what=20 I should do to fix that error. For this matter I've created a new post = on the nvidia forum. If someone would like to give it a look,it is here :


Il giorno ven 3 feb 2023= alle ore 15:21 Mario Marietto <marietto2008@gmail.com> ha scritto:
Very interesting behavior. I've added this new line in /etc/rc.conf : (unti= l some minutes ago I didn't use it because I was using the intel=20 driver,installed by the package : xf86-video-intel-2.99.917.916_2,1)=

<= span style=3D"font-family:arial,sans-serif">kld_list=3D"i915kms acpi_video"
=
and this line in = /boot/loader.conf :

ppt= devs=3D"1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3 5/0/0"<= br>
and boom. pp= tdevs worked again:

# p= ciconf -l
=
<= span style=3D"color:rgb(0,0,0);background-color:rgb(255,255,255)">ppt0@pci0= :1:0:0: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x030000 rev=3D0x= a1 hdr=3D0x00 vendor=3D0x10de device=3D0x1c02 subvendor=3D0x19da subdevice= =3D0x2438
ppt1@pci0:1:0:1: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0class=3D0x040300 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device= =3D0x10f1 subvendor=3D0x19da subdevice=3D0x2438
ppt2@pci0:2:0:0: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x03= 0000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1e04 subvendor=3D0x19= da subdevice=3D0x2503
ppt3@pci0:2:0:1: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x04= 0300 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x10f7 subvendor=3D0x19= da subdevice=3D0x2503
ppt4@pci0:2:0:2: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x0c= 0330 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1ad6 subvendor=3D0x19= da subdevice=3D0x2503
ppt5@pci0:2:0:3: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x0c= 8000 rev=3D0xa1 hdr=3D0x00 vendor=3D0x10de device=3D0x1ad7 subvendor=3D0x19= da subdevice=3D0x2503
ppt6@pci0:5:0:0: =C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0class=3D0x0c0330 rev=3D0x10 hdr=3D0x00 vendor= =3D0x1b73 device=3D0x1100 subvendor=3D0x1b73 subdevice=3D0x1100

So,maybe there is some incompatibility between the intel and the pptdevs driver ? take also in consideration that on xorg.cong I'm still using the intel= =20 driver :

Section "Device"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Identifier =C2=A0&quo= t;Card0"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Driver =C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0"intel"
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0BusID =C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0"PCI:0:2:0"
EndSection.

It's a confusing situation.
Il giorn= o ven 3 feb 2023 alle ore 14:13 Mario Marietto <marietto2008@gmail.com> ha scrit= to:
I've sent an email to the hackers ML with this content = :

> On 30 Jan 2023, a= t 05:37, Mario Marietto <marietto2008@gmail.com> wrote:
<= /span>
> In some FreeBSD 13.1 machines I have the problem below,in some=20 others I don't have it. I would like to know what the causes could be= =20 and how to fix it.
>
> root@marietto:/usr/home/marietto # devctl detach pci0:1:0:0
>
> devctl: Failed to detach pci0:1:0:0: Device busy
>
> root@marietto:/usr/home/marietto # devctl detach pci0:2:0:0
>
> devctl: Failed to detach pci0:2:0:0: Device busy
>
> Not even it works if instead of detach them, I try to attach them dire= ctly to the ppt driver :
>
> root@marietto:/usr/home/marietto/bhyve/Files # devctl set driver pci0:= 2:0:0 ppt
> devctl: Failed to set pci0:2:0:0 driver to ppt: Device busy
>=C2=A0
> root@marietto:/usr/home/marietto/bhyve/Files # devctl detach pci0:2:0:= 0
> devctl: Failed to detach pci0:2:0:0: Device busy
>
> Pci addresses 1:0:0 and 2:0:0 belong to the two GPUs that I have on my= PC :
>
> vgapci0@pci0:1:0:0:=C2=A0 =C2=A0 =C2=A0class=3D0x030000 rev=3D0xa1 hdr= =3D0x00 vendor=3D0x10de device=3D0x1c02 subvendor=3D0x19da subdevice=3D0x24= 38
> vgapci1@pci0:2:0:0:=C2=A0 =C2=A0 =C2=A0class=3D0x030000 rev=3D0xa1 hdr= =3D0x00 vendor=3D0x10de device=3D0x1e04 subvendor=3D0x19da subdevice=3D0x25= 03
>
> Actually I have commented this line on /boot/loader.conf,because it makes no difference if I keep it uncommented or not. It is totally=20 ignored.
>
> #pptdevs=3D"1/0/0 1/0/1 2/0/0 2/0/1 2/0/2 2/0/3"
=

and George replied to me like this :

Do you have 'vmm_load=3D"YES"' in loader.conf?

If not I am not sure it will have an effect - vmm must be loaded early=20 otherwise the card will be grabbed by the vgapci driver rather than be=20 put under ppt control.

I think this is an interesting answer. Yes,I keep vmm_load=3D"= ;YES" in loader.conf,BUT putting the 4 slots under ppt control using t= he loader.conf did not work for me for months. We haven't taken this in= to consideration. And you didn't reply when I've asked you if you h= ave modified the bhyve source code,because putting the pci addresses under = ppt using the loader.conf always worked for me. I would like to understand = why it does not work anymore. This is important to understand why I'm n= ot able to attach / detach the nVidia driver from the host to the guest os.= And maybe George's consideration is relevant.

Il giorno ven 3 feb 2023 alle ore 12:11 Corvin K=C3=B6hne = <C.Koehne@bec= khoff.com> ha scritto:
  • ---> Figure out if you can blacklist your = gpu in the nvidia driver. So, the nVidia driver never gets attached to your= device.
  • nope. I need to attach the nVidia driver to m= y gpu if I want to run stable diffusion + pytorch + cuda on the Linuxulator= on FreeBSD. I'm thinking of installing Stable Diffusion within the Linux VM,instead of doing it on FreeBSD.
  • ---> Wait for drm drivers and use the nouv= eau driver. Nouveau isn=E2=80=99t maintained by nVidia, so it=E2=80=99s mor= e likely to get fixed.
  • The nouveau driver is always bugged and it ne= ver worked well. I wonder if it is still supported. It has even limited fea= tures.

=C2=A0

Afaik, you don=E2=80=99t have other options, sry. I= =E2=80=99m no gpu driver developer, so I can=E2=80=99t help you.<= /u>

Btw: Do you refer to linux nouveau or freebsd nouvea= u? Freebsd made much progress in porting linux drm to freebsd with 14.0. So= , if latest linux nouveau fits your requirements, it=E2=80=99s likely that = freebsd 14.0 may fit them too.

=C2=A0

  • Are the following commands enough to update t= he bhyve source code on my machine ? take in consideration that I need to g= rab only your new last patch.
  • cd /usr/src
  • git checkout -f origin/phab/corvink/13.1/nvid= ia-wip
  • # cd /wherever/you/put/my/Makefile<= /u>

# make -j 8

=C2=A0

My latest patch is a mouse driver patch for the host= . So, no, you have to use:

cd /usr/src

git fetch

git checkout -f origin/phab/corvink/13.1/nvidia-wip<= u>

make kernel

reboot

=C2=A0

=C2=A0

Best regards

Corvin

=C2=A0

=C2=A0


This e= mail contains confidential information. If you have received it in error, y= ou must not read, use, copy or pass on this e-mail or its attachments.
If you have received the e-mail in error, please inform me immediately by r= eply e-mail and then delete this e-mail from your system. Thank you

Diese E-Mail enthaelt vertrauliche Informationen. Sollten Sie sie irrtuemli= ch erhalten haben, duerfen Sie diese E-Mail oder ihre Anhaenge nicht lesen,= verwenden, kopieren oder weitergeben.
Sollten Sie die Mail versehentlich erhalten haben, teilen Sie mir dies bitt= e umgehend per Antwort-E-Mail mit und loeschen Sie diese E-Mail dann aus Ih= rem System. Vielen Dank

Beckhoff Automation GmbH & Co. KG | Managing Director: Dipl. Phys. Hans= Beckhoff
Registered office: Verl, Germany | Register court: Guetersloh HRA 7075



--
Mari= o.


--
Mario.
=


--
Mario.
=
--00000000000066e96a05f3cec1de--