From nobody Sat Dec 3 18:14:29 2022 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4NPdJY3yprz4j2ZQ; Sat, 3 Dec 2022 18:15:09 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-ej1-x631.google.com (mail-ej1-x631.google.com [IPv6:2a00:1450:4864:20::631]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4NPdJY1RyBz49Bk; Sat, 3 Dec 2022 18:15:09 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; none Received: by mail-ej1-x631.google.com with SMTP id fy37so18512708ejc.11; Sat, 03 Dec 2022 10:15:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=iuGcZT2pzJVCu7zu6+KGlleD2uMnERat+5avNTDcCxY=; b=QMevgMxYNZlD0L0sLmHTsYynTgkBL69SOylFtPd6fr0o5WaV49wsCV3ZfXIdNxS7kH YgYxkupmNwe12AO79Dt57SBrH+K1l0hRHQKJ1zfaoMJrAWvJYxK7WWFgEgKbztvqYJDn dfsJ1IKYMqEt8b2rFcy49mtZ3zDg4PZr7mCJPpon/ZMutxeQMogGOMQxmaem5qxo4sMi FxUDxj0vpdYKEsptviXxZ6ofZ/jKTECbaN5ijop2j7h0b6FE6KXWfumESs4kT7kqpVHZ Pe1AKKGt79TGPjC4onaFcYQ0MxtGA7OyL0mdCN2NSJUXh5cZnDHmThskK0p1kg8PTUXW xlSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=iuGcZT2pzJVCu7zu6+KGlleD2uMnERat+5avNTDcCxY=; b=EbnqGDQdjUdljP1qqSrLOogrpL5u7gBdHbE0mHT/G4nAvh3FU3nOAgXp48TrLdQe90 aHC6iyq9UdmrGWeXnEgCiB6M/a4Z6rO0WDEo4+Kgr7XzHHaS1epHKS1lqL29nm25GWVK aoZY/2xgSdxtuIEPKtnGZrWYtzfKauU062jwQPcaRQ19Bf7Z6KdUtTSDXqZzO/VJQT4H 6ckOS7yRM3s0h33M4hKRCzfsEcKK38+pDbItvyZEtpBTrlumhC8zRTiztIVMBnDyZW1+ vJDdX4tPyHZ/ZFySlIq5Pz/EsQfaD+AbGBc2s+RIbHAISBoEV2owYrtCK6CdNY/lOa3P Olsg== X-Gm-Message-State: ANoB5plv6YDNClXTw4df73X6bqpxJbbDNkYCfquADZCgDXvUDy9iZ97K XuIDHe/yCHS9uoInbv0pD1T2AMbPYFQSb2oH+Gl0sg1fJ943IA== X-Google-Smtp-Source: AA0mqf5aMhV5NjqxoygsJ5D+r8HzKDEqXE+Fm12KayIjlxsmyi44mvng5O6qi1PPX6HHfcs968J9UtQ8ADXLAReQ7OY= X-Received: by 2002:a17:906:7848:b0:7ad:b286:72da with SMTP id p8-20020a170906784800b007adb28672damr65392471ejm.152.1670091306232; Sat, 03 Dec 2022 10:15:06 -0800 (PST) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-virtualization@freebsd.org X-BeenThere: freebsd-virtualization@freebsd.org MIME-Version: 1.0 References: <2E8B4BF7-A39E-4B93-820D-FC66F8C2F94B@FreeBSD.org> In-Reply-To: <2E8B4BF7-A39E-4B93-820D-FC66F8C2F94B@FreeBSD.org> From: Mario Marietto Date: Sat, 3 Dec 2022 19:14:29 +0100 Message-ID: Subject: Re: How to use the framebuffer as primary video device instead of the nvidia passed-through graphic card in a bhyve/linux vm To: =?UTF-8?Q?Corvin_K=C3=B6hne?= Cc: virtualization@freebsd.org, freebsd-x11+help@freebsd.org, FreeBSD virtualization , hackers@freebsd.org Content-Type: multipart/alternative; boundary="0000000000000cf23f05eef06ff9" X-Rspamd-Queue-Id: 4NPdJY1RyBz49Bk X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; TAGGED_RCPT(0.00)[help]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US] X-Rspamd-Pre-Result: action=no action; module=replies; Message is reply to one we originated X-ThisMailContainsUnwantedMimeParts: N --0000000000000cf23f05eef06ff9 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable ok. I tried like this,but it didn't work : bhyve -S -c sockets=3D1,cores=3D2,threads=3D2 -m 4G -w -H -A \ -s 0,hostbridge \ -s 2,virtio-blk,/mnt/$vmdisk1'p2'/bhyve/img/Linux/ubuntu2210.img,bootindex= =3D1 \ -s 3,virtio-blk,/dev/$vmdisk4 \ -s 4,virtio-blk,/dev/$vmdisk2 \-s 7,fbuf,tcp=3D0.0.0.0:5919,w=3D1600,h=3D95= 0,wait \ -s 8:0,passthru,2/0/0 \ -s 8:1,passthru,2/0/1 \ -s 8:2,passthru,2/0/2 \ -s 8:3,passthru,2/0/3 \ -s 10,virtio-net,tap19 \ -s 11,virtio-9p,sharename=3D/ \ -s 30,xhci,tablet \ -s 31,lpc \ -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ vm0:19 < /dev/null & sleep 2 && vncviewer 0:19 I tried specifying the bus ID of the framebuffer and I have used this xorg.conf file : Section "Files" ModulePath "/usr/lib/xorg/modules" FontPath "/usr/share/fonts/X11/misc" FontPath "/usr/share/fonts/X11/cyrillic" FontPath "/usr/share/fonts/X11/100dpi/:unscaled" FontPath "/usr/share/fonts/X11/75dpi/:unscaled" FontPath "/usr/share/fonts/X11/Type1" FontPath "/usr/share/fonts/X11/100dpi" FontPath "/usr/share/fonts/X11/75dpi" FontPath "built-ins" EndSection Section "Module" Load "vnc" Load "glx" EndSection Section "InputDevice" Identifier "Keyboard0" Driver "kbd" EndSection Section "InputDevice" Identifier "Mouse0" Driver "mouse" Option "Protocol" "auto" Option "Device" "/dev/input/mice" Option "ZAxisMapping" "4 5 6 7" EndSection Section "Monitor" Identifier "Monitor0" VendorName "Monitor Vendor" ModelName "Monitor Model" EndSection Section "Device" Identifier "Card0" Driver "modesetting" BusID "PCI:0:7:0" EndSection Section "Screen" Identifier "Screen0" Device "Card0" Monitor "Monitor0" SubSection "Display" Viewport 0 0 Depth 1 EndSubSection SubSection "Display" Viewport 0 0 Depth 4 EndSubSection SubSection "Display" Viewport 0 0 Depth 8 EndSubSection SubSection "Display" Viewport 0 0 Depth 15 EndSubSection SubSection "Display" Viewport 0 0 Depth 16 EndSubSection SubSection "Display" Viewport 0 0 Depth 24 EndSubSection EndSection The error reported has been : https://ibb.co/1KX2h26 https://ibb.co/Cv5FffB thanks. Il giorno sab 3 dic 2022 alle ore 17:34 Corvin K=C3=B6hne ha scritto: > On December 3, 2022 4:49:46 PM GMT+01:00, Mario Marietto < > marietto2008@gmail.com> wrote: >> >> Hello to everyone. >> >> what Im trying to do is to set the framebuffer video adapter as primary >> graphic card on my bhyve-ubuntu vm instead of the nvidia RTX 2080 ti car= d >> that I have passed through. What I want to do really is to use both the >> graphic adapters,but the primary should be the framebuffer and the >> secondary the nvidia. I tried different Xorg configurations,but what I'v= e >> got is that Xorg failed to display some errors. So,the controller that y= ou >> see below should be used as primary inside the ubuntu vm : >> >> 00:1d.0 VGA compatible controller: Device fb5d:40fb >> >> while the ones you see below as secondary : >> >> 08:00.0 VGA compatible controller: NVIDIA Corporation TU102 [GeForce RTX= 2080 Ti] (rev a1) >> 08:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Con= troller (rev a1) >> 08:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller= (rev a1) >> 08:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI = Controller (rev a1) >> >> >> The script that I use to launch the vm is the following : >> >> #!/bin/sh >> setxkbmap it >> vms=3D"$(ls /dev/vmm/*)" >> vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')" >> >> for vm in $vms; do >> session=3D"${vm##*/}" >> echo "bhyve session =3D $session" >> echo "vnc session =3D $vncs" >> if ! printf '%s\n' "${vncs}" | grep "${session}"; then >> printf 'VNC session not found,destroying= ghost vms\n' >> bhyvectl --vm=3D$session --destroy >> else >> printf 'Found VNC session %s\n' "${sessi= on},no ghost vms found,not destroying them" >> fi >> done >> >> vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (201= 5020204055E)/ && d{print d}'` >> echo "TOSHIBA External USB 3.0 1.8 TB ; $vmdisk1" >> >> mount -t ufs /dev/$vmdisk1'p2' /mnt/$vmdisk1'p2' >> >> bhyve -S -c sockets=3D1,cores=3D2,threads=3D2 -m 4G -w -H -A \ >> -s 0,hostbridge \ >> -s 2,virtio-blk,/mnt/$vmdisk1'p2'/bhyve/img/Linux/ubuntu2210.img,bootind= ex=3D1 \ >> -s 3,virtio-blk,/dev/$vmdisk4 \ >> -s 4,virtio-blk,/dev/$vmdisk2 \ >> -s 8:0,passthru,2/0/0 \ >> -s 8:1,passthru,2/0/1 \ >> -s 8:2,passthru,2/0/2 \ >> -s 8:3,passthru,2/0/3 \ >> -s 10,virtio-net,tap19 \ >> -s 11,virtio-9p,sharename=3D/ \ >> -s 29,fbuf,tcp=3D0.0.0.0:5919,w=3D1600,h=3D950,wait \ >> -s 30,xhci,tablet \ >> -s 31,lpc \ >> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \ >> vm0:19 < /dev/null & sleep 2 && vncviewer 0:19 >> >> For sure ,on /boot/loader.conf I've added : >> >> /boot/loader.conf >> >> pptdevs=3D"2/0/0 2/0/1 2/0/2 2/0/3" >> >> As I said before,I tried two xorg conf files to achieve the goal. On the >> first one I tried to add only the framebuffer,like this : >> >> >> Section "Files" >> ModulePath "/usr/lib/xorg/modules" >> FontPath "/usr/share/fonts/X11/misc" >> FontPath "/usr/share/fonts/X11/cyrillic" >> FontPath "/usr/share/fonts/X11/100dpi/:unscaled" >> FontPath "/usr/share/fonts/X11/75dpi/:unscaled" >> FontPath "/usr/share/fonts/X11/Type1" >> FontPath "/usr/share/fonts/X11/100dpi" >> FontPath "/usr/share/fonts/X11/75dpi" >> FontPath "built-ins" >> EndSection >> >> Section "Module" >> Load "vnc" >> Load "glx" >> EndSection >> >> >> Section "InputDevice" >> Identifier "Keyboard0" >> Driver "kbd" >> EndSection >> >> Section "InputDevice" >> Identifier "Mouse0" >> Driver "mouse" >> Option "Protocol" "auto" >> Option "Device" "/dev/input/mice" >> Option "ZAxisMapping" "4 5 6 7" >> >> EndSection >> >> Section "Monitor" >> Identifier "Monitor0" >> VendorName "Monitor Vendor" >> ModelName "Monitor Model" >> EndSection >> >> Section "Device" >> Identifier "Card0" >> Driver "modesetting" >> BusID "PCI:0:29:0" >> >> EndSection >> >> Section "Screen" >> Identifier "Screen0" >> Device "Card0" >> Monitor "Monitor0" >> SubSection "Display" >> Viewport 0 0 >> Depth 1 >> EndSubSection >> SubSection "Display" >> Viewport 0 0 >> Depth 4 >> EndSubSection >> SubSection "Display" >> Viewport 0 0 >> Depth 8 >> EndSubSection >> SubSection "Display" >> Viewport 0 0 >> Depth 15 >> EndSubSection >> SubSection "Display" >> Viewport 0 0 >> Depth 16 >> EndSubSection >> SubSection "Display" >> Viewport 0 0 >> Depth 24 >> EndSubSection >> EndSection >> >> >> but it didn't work. This is the log file that shows the errors reported = : https://pastebin.ubuntu.com/p/Gv7wgsDR4K/ >> I have also removed the xorg.conf file,but it didn't work either. This i= s the log file that shows the errors reported : >> >> >> https://pastebin.ubuntu.com/p/wNcfQTByQm/ >> >> Can someone give me some suggestions that can help me to understand wher= e the mistake is,please,thanks. >> >> -- >> Mario. >> > > Try to assign a lower pci slot number to the framebuffer device than to > the nvidia gpu in your bhyve command. > -- > Best regards, > Corvin > --=20 Mario. --0000000000000cf23f05eef06ff9 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
ok. I tried like this,but it didn't work :
<= div>
bhyve -S -c sockets=3D1,cores=3D2,threads=3D2 -m=
 4G -w -H -A \
-s 0,hostbridge \
-s 2,virtio-blk,/mnt/$vmdisk1'p2'/bhyve/img/Linux/ubuntu2210.img,bo=
otindex=3D1 \
-s 3,virtio-blk,/dev/$vmdisk4 \
-s 4,virtio-blk,/dev/$vmdisk2 \
-s 7,fbuf,tcp=3D0.0=
.0.0:5919,w=3D1600,h=3D950,wait \
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 10,virtio-net,tap19 \
-s 11,virtio-9p,sharename=3D/ \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:19 < /dev/null & sleep 2 && vncviewer 0:19

I tried specifying the bus ID of the framebuffer and I ha= ve used this xorg.conf file :

Section "Files"
    ModulePath   "/usr/lib/xorg/modules"
    FontPath     "/usr/share/fonts/X11/misc"
    FontPath     "/usr/share/fonts/X11/cyrillic"
    FontPath     "/usr/share/fonts/X11/100dpi/:unscaled"
    FontPath     "/usr/share/fonts/X11/75dpi/:unscaled"
    FontPath     "/usr/share/fonts/X11/Type1"
    FontPath     "/usr/share/fonts/X11/100dpi"
    FontPath     "/usr/share/fonts/X11/75dpi"
    FontPath     "built-ins"
EndSection

Section "Module"
    Load  "vnc"
    Load  "glx"
EndSection


Section "InputDevice"
    Identifier  "Keyboard0"
    Driver      "kbd"
EndSection

Section "InputDevice"
    Identifier  "Mouse0"
    Driver      "mouse"
    Option        "Protocol" "auto"
    Option        "Device" "/dev/input/mice"
    Option        "ZAxisMapping" "4 5 6 7"

EndSection

Section "Monitor"
    Identifier   "Monitor0"
    VendorName   "Monitor Vendor"
    ModelName    "Monitor Model"
EndSection

Section "Device"
   Identifier  "Card0"
   Driver      "modesetting"
   BusID       "PCI:0:7:0"

EndSection

Section "Screen"
    Identifier "Screen0"
    Device     "Card0"
    Monitor    "Monitor0"
    SubSection "Display"
        Viewport   0 0
        Depth     1
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     4
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     8
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     15
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     16
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     24
    EndSubSection
EndSection
The error reported has been :


thanks.

Il giorno sab 3 dic 2022 alle ore 17:34 Corvin K= =C3=B6hne <corvink@freebsd.org> ha scritto:
On December 3, 2022 4:49:46 PM GMT+01:00,= Mario Marietto <marietto2008@gmail.com> wrote:
Hello to everyone.

what Im trying to do is to set the framebuffer video adapter as primary=20 graphic card on my bhyve-ubuntu vm instead of the nvidia RTX 2080 ti=20 card that I have passed through. What I want to do really is to use both the graphic adapters,but the primary should be the framebuffer and the=20 secondary the nvidia. I tried different Xorg configurations,but what I'= ve got is that Xorg failed to display some errors. So,the controller that you see below should be used as primary inside the ubuntu vm :

00:1d.0 VGA compatible controller: Device fb5d:40f=
b

while the ones you see below as secondary :

08:00.0 VGA compatible controller: NVIDI=
A Corporation TU102 [GeForce RTX 2080 Ti] (rev a1)
08:00.1 Audio device: NVIDIA Corporation TU102 High Definition Audio Contro=
ller (rev a1)
08:00.2 USB controller: NVIDIA Corporation TU102 USB 3.1 Host Controller (r=
ev a1)
08:00.3 Serial bus controller: NVIDIA Corporation TU102 USB Type-C UCSI Con=
troller (rev a1)

The script that I use to launch the vm is the following :

#!/bin/sh
setxkbmap it
vms=3D"$(ls /dev/vmm/*)"
vncs=3D"$(ps ax | awk '/vncviewer [0]/{print $6}')"

for vm in $vms; do
                session=3D"${vm##*/}"
                echo "bhyve session =3D $session"
                echo "vnc session =3D $vncs"
                if ! printf '%s\n' "${vncs}" | grep "=
;${session}"; then
                                printf 'VNC session not found,destroyin=
g ghost vms\n'
                                bhyvectl --vm=3D$session --destroy         =
                 =20
                else
                                printf 'Found VNC session %s\n' &qu=
ot;${session},no ghost vms found,not destroying them"
                fi
done

vmdisk1=3D`geom disk list | awk '/^Geom name: /{d=3D$NF} /^ *ident: (20=
15020204055E)/ && d{print d}'`
echo "TOSHIBA External USB 3.0 1.8 TB ; $vmdisk1"

mount -t ufs /dev/$vmdisk1'p2' /mnt/$vmdisk1'p2'

bhyve -S -c sockets=3D1,cores=3D2,threads=3D2 -m 4G -w -H -A \
-s 0,hostbridge \
-s 2,virtio-blk,/mnt/$vmdisk1'p2'/bhyve/img/Linux/ubuntu2210.img,bo=
otindex=3D1 \
-s 3,virtio-blk,/dev/$vmdisk4 \
-s 4,virtio-blk,/dev/$vmdisk2 \
-s 8:0,passthru,2/0/0 \
-s 8:1,passthru,2/0/1 \
-s 8:2,passthru,2/0/2 \
-s 8:3,passthru,2/0/3 \
-s 10,virtio-net,tap19 \
-s 11,virtio-9p,sharename=3D/ \
-s 29,fbuf,tcp=3D0.0.0.0:=
5919,w=3D1600,h=3D950,wait \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI_CODE.fd \
vm0:19 < /dev/null & sleep 2 && vncviewer 0:19


For sure ,on /boot/loade=
r.conf I've added :

/boot/loader.conf

pptdevs=3D"2/0/0 2/0/1 2/0/2 2/0/3"


As I said before,I tried=
 two xorg conf files to achieve the goal. On the
first one I tried to add only the framebuffer,like this :

Section "Files"
    ModulePath   "/usr/lib/xorg/modules"
    FontPath     "/usr/share/fonts/X11/misc"
    FontPath     "/usr/share/fonts/X11/cyrillic"
    FontPath     "/usr/share/fonts/X11/100dpi/:unscaled"
    FontPath     "/usr/share/fonts/X11/75dpi/:unscaled"
    FontPath     "/usr/share/fonts/X11/Type1"
    FontPath     "/usr/share/fonts/X11/100dpi"
    FontPath     "/usr/share/fonts/X11/75dpi"
    FontPath     "built-ins"
EndSection

Section "Module"
    Load  "vnc"
    Load  "glx"
EndSection


Section "InputDevice"
    Identifier  "Keyboard0"
    Driver      "kbd"
EndSection

Section "InputDevice"
    Identifier  "Mouse0"
    Driver      "mouse"
    Option        "Protocol" "auto"
    Option        "Device" "/dev/input/mice"
    Option        "ZAxisMapping" "4 5 6 7"

EndSection

Section "Monitor"
    Identifier   "Monitor0"
    VendorName   "Monitor Vendor"
    ModelName    "Monitor Model"
EndSection

Section "Device"
   Identifier  "Card0"
   Driver      "modesetting"
   BusID       "PCI:0:29:0"

EndSection

Section "Screen"
    Identifier "Screen0"
    Device     "Card0"
    Monitor    "Monitor0"
    SubSection "Display"
        Viewport   0 0
        Depth     1
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     4
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     8
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     15
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     16
    EndSubSection
    SubSection "Display"
        Viewport   0 0
        Depth     24
    EndSubSection
EndSection


bu=
t it didn't work. This is the log file that shows the errors reported :=
 https://pastebin.ubuntu.com/p/Gv7wgsDR4K/
I have also rem=
oved the xorg.conf file,but it didn't work either. This is the log file=
 that shows the errors reported :


https://pastebin.ubuntu.co=
m/p/wNcfQTByQm/


Can someone give me some=
 suggestions that can help me to understand where the mistake is,please,tha=
nks.
--
Mario.

Try to assign a lower pci slot number = to the framebuffer device than to the nvidia gpu in your bhyve command.
=
--
Best regards,
Corvin


--
Mario.
--0000000000000cf23f05eef06ff9--