From nobody Thu May 2 08:13:56 2024 X-Original-To: freebsd-virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VVRYR3fZ7z5JY7y; Thu, 2 May 2024 08:14:35 +0000 (UTC) (envelope-from marietto2008@gmail.com) Received: from mail-pj1-x102c.google.com (mail-pj1-x102c.google.com [IPv6:2607:f8b0:4864:20::102c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VVRYQ53LBz4YK0; Thu, 2 May 2024 08:14:34 +0000 (UTC) (envelope-from marietto2008@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20230601 header.b=HrZwOD9D; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of marietto2008@gmail.com designates 2607:f8b0:4864:20::102c as permitted sender) smtp.mailfrom=marietto2008@gmail.com Received: by mail-pj1-x102c.google.com with SMTP id 98e67ed59e1d1-2b3374e08cfso870631a91.2; Thu, 02 May 2024 01:14:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1714637673; x=1715242473; darn=freebsd.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=ZD8A6fWL1kBPGPCaQrxJsjZxlU+dEyIeItOEFk2lHBw=; b=HrZwOD9D5XWhqd79WvuSeF6AuOsV+iys4BqlNzPX5vKBYZDTdmEKDZimi4GbNqcv8p FvfR4Qcb1guZm4hfSyFaRzkAXUv/4oq5x/MjYt3IRliTC0FoG/2ioZgJaBgqN33Yow4f 0JVhsAYvcYlDnL3hRvjVBjIgBBOeG+yVs2tlNjcW+xL1TxfNRmVncnAYDezf5rl9Ad8X 64pmh9JaGKBmm5CgyCMoqaJ1Rpr+Q3DoqKXHYg6P2udivasWMq8wIaYvRRI8TboBCb0s qymSEI8tWvMSpMVv0a3u0jd1LCsQuyuP57HywaOqQMpQO563nvh8BFJy8K36UlEgEH5p zrTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714637673; x=1715242473; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=ZD8A6fWL1kBPGPCaQrxJsjZxlU+dEyIeItOEFk2lHBw=; b=bb3FsuA5KxqZcKTG4a9wZ6QRa+AuP4221hKcPU/E+PlrBgM0OWxPCLZ239dHHoQmI3 WImHfa5PcjiJckfJnnTvgCzLrGBjs1Ew+ZloRAy+9zEYrL6f4HXiJEPHu7B34Le0KAPL etBJZAyKNW93nANZSwBm5DQ6Jts8BKCycy0CfLnb/NQSEhaTSfGgcEYasukYfUHAFruq E2+EBPtz2blsiWDskTh2x2QPrIfr14wC7TAYD7FDghmosvQ+ls+1wPcDcTeN6hU/Et+u Z5LYkt4cxiXaZpQyD5I9G+6ER0qMVod/jBCHVyhIpDJQLKQ4e4gbErxhBN093Jwu5Tbh JVHg== X-Forwarded-Encrypted: i=1; AJvYcCXKAeEoHhNJZnxNAu8UynutXbBHVjRxY0LM0L51bEvvb8oY8jilBZaPKekR/4cyCTQBvjepiFaAFSsBGR+Q/Y9m6fPqzZtXRjzhAZgFSD2Xxd3Z5YzJuLA+IIGBsccxqKHTx7vDLGIXMX6k+pHX1gBtHJ0g7HY5VfBe7fWtdl3TAGqXynTP5EFvYOtK8U34u0mBUmkqaPvECRetqGnQYuoVLTPqV+IZYQ== X-Gm-Message-State: AOJu0YzTYPoC1vG+5Q83vPCtTPTkKZxrcWgXh5Be+xAimWN72y9sNFhQ c7FHBLei3msIT24u9i2QHhMTA24nhgr2jeFQf/1ruzyVgqlU110D7vqrTA+TAHi6u2wql+dL+hD Oe4uuwsKYZVyD1smxvSEJL5EcHCLSVeMFv/Alog== X-Google-Smtp-Source: AGHT+IF71bsubqvkmbYg2dhtTKk6iYUomrtDuoKJIHwNZS9oKLenUnMVZ7aI57BJih4puCPXqQCaPsBcuuCzzXv+sZE= X-Received: by 2002:a17:90b:4c4c:b0:2ac:f010:b1c0 with SMTP id np12-20020a17090b4c4c00b002acf010b1c0mr5189757pjb.22.1714637673241; Thu, 02 May 2024 01:14:33 -0700 (PDT) List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 References: In-Reply-To: From: Mario Marietto Date: Thu, 2 May 2024 10:13:56 +0200 Message-ID: Subject: Re: How to use Virtio GPU on FreeBSD as guest OS. To: Yusuf Khan Cc: freebsd-drivers@freebsd.org, freebsd-x11@freebsd.org, freebsd-hackers , FreeBSD Mailing List , FreeBSD virtualization Content-Type: multipart/alternative; boundary="0000000000006ea444061774319f" X-Spamd-Bar: --- X-Spamd-Result: default: False [-4.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20230601]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; FREEMAIL_TO(0.00)[gmail.com]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_HAS_DN(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::102c:from]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; RCVD_TLS_LAST(0.00)[]; MISSING_XM_UA(0.00)[]; RCPT_COUNT_FIVE(0.00)[6]; TO_MATCH_ENVRCPT_SOME(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MID_RHS_MATCH_FROMTLD(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; MLMMJ_DEST(0.00)[freebsd-drivers@freebsd.org,freebsd-x11@freebsd.org,freebsd-hackers@freebsd.org,freebsd-questions@freebsd.org,freebsd-virtualization@freebsd.org]; RCVD_COUNT_ONE(0.00)[1]; FREEMAIL_ENVFROM(0.00)[gmail.com] X-Rspamd-Queue-Id: 4VVRYQ53LBz4YK0 --0000000000006ea444061774319f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable What I find strange is that this configuration works on Windows 11 to virtualize FreeBSD using -device vmware-svga : I:\OS\vms\qemu\qemu-system-x86_64w.exe -accel whpx -machine q35 -cpu kvm64,hv_relaxed,hv_time,hv_synic -m 8G -device vmware-svga,id=3Dvideo0,vgamem_mb=3D16,bus=3Dpcie.0,addr=3D0x1 -audiodev dsound,id=3Dsnd0 -device ich9-intel-hda -device hda-duplex,audiodev=3Dsnd0 = -hda "I:\Backup\FreeBSD\FreeBSD-140-zfs.img" -device hda-duplex,audiodev=3Dsnd0 -drive file=3D\\.\PhysicalDrive4 -drive file=3D\\.\PhysicalDrive5 -drive file=3D\\.\PhysicalDrive6 -drive file=3D\\.\PhysicalDrive8 -drive file=3D\\.\PhysicalDrive12 -rtc base=3Dlocaltime -device usb-ehci,id=3Dusb,bus=3Dpcie.0,addr=3D0x3 -device usb-tablet -device usb-kb= d -smbios type=3D2 -nodefaults -netdev tap,id=3Dmynet0,ifname=3D"OpenVPN-TAP-Windows",script=3Dno,downscript=3Dno = -device e1000,netdev=3Dmynet0,mac=3D52:55:00:d1:55:01 -device ich9-ahci,id=3Dsata -= bios "I:\OS\vms\qemu\OVMF_combined.fd" One can think that it will work even on Linux for the same setup (qemu + hyper-V on Windows 11),but it does not. I'm experiencing a lot of problems when I add -accel whpx on a Debian VM. These problems go away if I remove it. Adding " -device vmware-svga,id=3Dvideo0,vgamem_mb=3D16,bus=3Dpcie.0,addr= =3D0x1" to the qemu / Debian parameters will cause it won't boot. So,this : I:\OS\vms>I:\OS\vms\qemu\qemu-system-x86_64.exe -machine q35 -cpu kvm64,hv_relaxed,hv_time,hv_synic -m 8G -device vmware-svga,id=3Dvideo0,vgamem_mb=3D16,bus=3Dpcie.0,addr=3D0x1 -audiodev dsound,id=3Dsnd0 -device ich9-intel-hda -device hda-duplex,audiodev=3Dsnd0 = -hda "I:\Backup\Linux\Debian.img" -drive file=3D\\.\PhysicalDrive5 -drive file=3D\\.\PhysicalDrive6 -drive file=3D\\.\PhysicalDrive8 -rtc base=3Dloca= ltime -device usb-ehci,id=3Dusb,bus=3Dpcie.0,addr=3D0x3 -device usb-tablet -devic= e usb-kbd -smbios type=3D2 -nodefaults -netdev user,id=3Dnet0 -device e1000,netdev=3Dnet0,id=3Dnet0,mac=3D52:54:00:11:22:33 -device ich9-ahci,id= =3Dsata -bios "I:\OS\vms\qemu\OVMF_combined.fd" will cause the Debian VM to freeze before reaching the login prompt. I don't know if Linux has the proper vmware-svga driver,I didn't find it. Regarding virtualizing Windows 7 on Windows 11 using the same setup,I can't add -accel whpx and most importantly,what I care more, the -device vmware-svga does not work. So,in this kind of setup, *FreeBSD is the winner because it supports everything (hyper-V and the vmware-svga 3D accelerated device.* On Sat, Apr 27, 2024 at 9:41=E2=80=AFPM Yusuf Khan wrote: > So your issue is that the performance is worse than Linux? > > There is no 3d graphics driver for virtio in mainline drm-kmod but > https://github.com/freebsd/drm-kmod/pull/119 has > experimental support, although that PR exists it seems to be > non-functional with a crash. > --=20 Mario. --0000000000006ea444061774319f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
What I find strange is that this configuration works on Windo= ws 11 to virtualize FreeBSD using=20 -device vmware-svga :

I:\OS\vms\qemu\qemu-syst= em-x86_64w.exe -accel whpx -machine q35 -cpu kvm64,hv_relaxed,hv_time,hv_sy= nic -m 8G -device vmware-svga,id=3Dvideo0,vgamem_mb=3D16,bus=3Dpcie.0,addr= =3D0x1 -audiodev dsound,id=3Dsnd0 -device ich9-intel-hda -device=20 hda-duplex,audiodev=3Dsnd0 -hda "I:\Backup\FreeBSD\FreeBSD-140-zfs.img= " -device hda-duplex,audiodev=3Dsnd0 -drive file=3D\\.\PhysicalDrive4 -drive= =20 file=3D\\.\PhysicalDrive5 -drive file=3D\\.\PhysicalDrive6 -drive=20 file=3D\\.\PhysicalDrive8 -drive file=3D\\.\PhysicalDrive12 -rtc=20 base=3Dlocaltime -device usb-ehci,id=3Dusb,bus=3Dpcie.0,addr=3D0x3 -device = usb-tablet -device usb-kbd -smbios type=3D2 -nodefaults -netdev tap,id=3Dmy= net0,ifname=3D"OpenVPN-TAP-Windows",script=3Dno,downscript=3Dno -= device e1000,netdev=3Dmynet0,mac=3D52:55:00:d1:55:01 -device ich9-ahci,id= =3Dsata -bios "I:\OS\vms\qemu\OVMF_combined.fd"

One can think that it will work even on Linux for the same setup (qemu +=20 hyper-V on Windows 11),but it does not. I'm experiencing a lot of=20 problems when I add -accel whpx on a Debian VM. These problems go away=20 if I remove it.

Adding " -device vmware-svga,id=3Dvideo0,vgamem_mb=3D16,bus=3Dpcie.0,addr=3D0x1&quo= t; to the qemu / Debian parameters will cause it won't boot. So,this :<= /div>


I:\OS\vms>I:\OS\vms\qemu\qemu-sy= stem-x86_64.exe -machine q35 -cpu kvm64,hv_relaxed,hv_time,hv_synic -m 8G -= device vmware-svga,id=3Dvideo0,vgamem_mb=3D16,bus=3Dpcie.0,addr=3D0x1 -audiodev dsound,id=3Dsnd0 -device ich9-intel-hda -device=20 hda-duplex,audiodev=3Dsnd0 -hda "I:\Backup\Linux\Debian.img" -dri= ve=20 file=3D\\.\PhysicalDrive5 -drive file=3D\\.\PhysicalDrive6 -drive=20 file=3D\\.\PhysicalDrive8 -rtc base=3Dlocaltime -device=20 usb-ehci,id=3Dusb,bus=3Dpcie.0,addr=3D0x3 -device usb-tablet -device=20 usb-kbd -smbios type=3D2 -nodefaults -netdev user,id=3Dnet0 -device=20 e1000,netdev=3Dnet0,id=3Dnet0,mac=3D52:54:00:11:22:33 -device ich9-ahci,id= =3Dsata -bios "I:\OS\vms\qemu\OVMF_combined.fd"


will cause the Debian VM to freeze before reaching the login prompt. I=C2=A0=20 don't know if Linux has the proper vmware-svga driver,I didn't find= it.

Regarding virtualizing Windows 7 on Windows 11 using the same setup,I can't add -accel whpx and most= =20 importantly,what I care more, the=20 -device vmware-svga does not work. So,in this kind of setup,FreeBSD is= =20 the winner because it supports everything (hyper-V and the vmware-svga 3D a= ccelerated device.
<= /div>


On Sat, Apr 27, 2024 at 9:41=E2=80=AFPM Yusuf= Khan <yusisamerican@gmail.co= m> wrote:
So your issue is that the performance is worse than Linux?

There is no 3d graphics driver for virtio in mainline drm-kmod but
https://github.com/freebsd/drm-kmod/pull/119 has
experimental support, although that PR exists it seems to be
non-functional with a crash.


--
Mario.
--0000000000006ea444061774319f-- From nobody Sat May 4 18:06:59 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VWwc40mDSz5JJmF for ; Sat, 4 May 2024 18:07:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VWwc34k1Pz4vFB for ; Sat, 4 May 2024 18:06:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1714846019; a=rsa-sha256; cv=none; b=JCog1giuZ8iq7is5+aRIVg3YvW36XUbFWDVNw9SQqmNs6UIlQzuM+UCiRzpFCdT59F6S9Q cbjVajs7ww3TWsWq5k4SK6KWnWmgZm9nmcHmIFpHphEdyCeiLXOkTIO5q16n+JUYa9aQzo BvBbEIqbiclNlO2PgJmG4dsHzTzIKcsFNRyYLUb6BKIUgR2YR8tQXCBleKmUw6qW/OpvWt mpejBJtXYic4BPFZzYi6f52VZzpXJoiKwhmuEgdfzu1RkxHe+khorup6pS/zmM/2KMrJfG 2B1KRQ8ZqFr9tFiVn3aLypptB+UFpDAbVr1bieC1D0DPUTR3gnRi0n56dVJ3yg== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1714846019; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=FcF9DDeGYfx3B9AcEe6dju75AAkPXTZrk5d2Ckqlpp0=; b=X3kEzK721pCuLe9OCkMJwZJoctaFj8RwfEDrcNLerROCSJmyDQmfOTaaHayUnAF8ESgXIE d2EmDz7bbpU+6j/MLb8vqlQVk6uZDmDSBh2nNjfZ/IxOfR4sGSMeC9ROzpAZG9z+8cjOGq VMGTyJFOdBSQv8ZIxkZWqNr7yiGBiOST8ZzAqS2Mq2b0zcRz5KGbvSjTvF2J60v8Cr9Xxp A3Ga6kZIcBWQ25iEMxfG605C8+cJhxzYSmtxao/d6+OrU7w0HvLYGxb7S6mZD+lT1jdMIi aQFyWVPzuJ91D+veYzlhvZqSNcIUUP8Noue+KnQnmr34127zKCI1t/sWZKg+Kg== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VWwc34Dw3zg7x for ; Sat, 4 May 2024 18:06:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 444I6x1t080392 for ; Sat, 4 May 2024 18:06:59 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 444I6xiH080387 for virtualization@FreeBSD.org; Sat, 4 May 2024 18:06:59 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 278058] Simultaneous use of Bhyve AND vnet on network PCI devices causes network failure Date: Sat, 04 May 2024 18:06:59 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bhyve X-Bugzilla-Version: 14.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: mark@markmcb.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D278058 --- Comment #1 from Mark McBride --- I listened the the bhyve production user call [1] and noticed people still speculating that bhyve+vnet SR-IOV issues might be PCI card specific. To clarify, I can reproduce this issue 100% of the time on Chelsio, Intel, and Mellanox NICs. I have reproduced it on two Supermicro motherboards (X12STH-= F, X11SSM-F) with 3 different Intel CPUs (Xeon E-2388G, Xeon E-2324G, Xeon E3-= 1275 v6).=20 So if hardware is a consideration, the only common thread for me is Supermi= cro motherboards and Intel Xeon CPUs. It's worth noting again, if bhyve alone is used for passthrough, I have no issues at all. It's only when the on-metal FreeBSD host passes VFs to bhyve= AND vnet jails that I observe issues. I currently have 8 network VFs and 1 iGPU passed to 4 bhyve instances with = no problems. My workaround is to use vnet+epair for any jail networking in the host (instead of vnet+VF). 1. Bhyve Production User Call 2024-04-25 https://www.youtube.com/watch?v=3DgGzpLLJTHS0 --=20 You are receiving this mail because: You are the assignee for the bug.= From nobody Sun May 5 01:56:20 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VX71m2xLjz53Xxb for ; Sun, 05 May 2024 01:56:28 +0000 (UTC) (envelope-from robn@despairlabs.com) Received: from wfhigh1-smtp.messagingengine.com (wfhigh1-smtp.messagingengine.com [64.147.123.152]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4VX71l2DvFz4s1V for ; Sun, 5 May 2024 01:56:27 +0000 (UTC) (envelope-from robn@despairlabs.com) Authentication-Results: mx1.freebsd.org; dkim=pass header.d=despairlabs.com header.s=fm3 header.b=wmewUUvO; dkim=pass header.d=messagingengine.com header.s=fm3 header.b=TeNxBFIY; dmarc=none; spf=pass (mx1.freebsd.org: domain of robn@despairlabs.com designates 64.147.123.152 as permitted sender) smtp.mailfrom=robn@despairlabs.com Received: from compute2.internal (compute2.nyi.internal [10.202.2.46]) by mailfhigh.west.internal (Postfix) with ESMTP id 7E176180008C for ; Sat, 4 May 2024 21:56:25 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Sat, 04 May 2024 21:56:25 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=despairlabs.com; h=cc:content-transfer-encoding:content-type:content-type:date :date:from:from:in-reply-to:message-id:mime-version:reply-to :subject:subject:to:to; s=fm3; t=1714874185; x=1714960585; bh=eE 9m78M0UqWh/j0EAmmlp2oJTo/8v/ffwdkLYJlKBvk=; b=wmewUUvOme3OkUXSGf zZrCLDCtIc4+J4MxKF1sOBDNtAc3OFt1j8Kc19ta/DHRJIlnW3ZFgKsxWYaPGIev aB8OafVvKcQsr+Wqmg0II79S0aDojbht0M9YRqH4cyNgHHR7CaChWvG7EMsxTQQM TtnnTrOUnaUPwsc2OUPa4dpr5Ahh9B0F6aiuIgOWxmHrKG1rxjPDfCRS0TP8spne G5ea6bS9C77RUi4SrLeWPfgumA6x369E8FqLhTGg/fZx30iJVmRCdz2+LajBsSGd FsBlRgvR1avTw6X82nVfms3DtUeqs7YciwPD7GCU+gsJ0gUgMC0jBdoyTSaSezMY ukpg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:message-id:mime-version:reply-to:subject:subject:to :to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s= fm3; t=1714874185; x=1714960585; bh=eE9m78M0UqWh/j0EAmmlp2oJTo/8 v/ffwdkLYJlKBvk=; b=TeNxBFIYytHj01wqn9q0bYaZDwprQ4S9JO8ybt8ZsOcv 6uQzr3XOD2Q62LkYJxllVOMF88vYrRBc/NOEQaV7+bFkQvv1ZdkJ9dO/jUiIqCoq ATQR7xonvQXPPnkTGwJhlApHABTsNnp36wR0m9tqgEMr9+NAn7an+6k47WFIOwVN rH43heWdLpEGc6JWE4LGQTfMuN/MV2HF2KE/m2PuO7PTu6hsy/eun/4QJdF9/LCF freigA8HZX5rhjAGDz84U3B2AYEVWWncchlWogAl4DPShzW7MTObAamYW23b+jal qfKBYxLAOh55/iLPTaNGWXcVPXSV+QxGuAFgQkdWrg== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvledrvddvfedgheefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpefkffggfgfvhffutgfgsehtkeertd dtvdejnecuhfhrohhmpeftohgsucfpohhrrhhishcuoehrohgsnhesuggvshhprghirhhl rggsshdrtghomheqnecuggftrfgrthhtvghrnhepffdufedvteevkeffgfffteeiiefhff efffetheegteetffduuddtgedtveetheelnecuffhomhgrihhnpehgihhthhhusgdrtgho mhdprghstghiihhnvghmrgdrohhrghdpuggvshhprghirhhlrggsshdrtghomhenucevlh hushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehrohgsnhesuggv shhprghirhhlrggsshdrtghomh X-ME-Proxy: Feedback-ID: ia7b9477a:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA for ; Sat, 4 May 2024 21:56:23 -0400 (EDT) Message-ID: Date: Sun, 5 May 2024 11:56:20 +1000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: virtualization@freebsd.org From: Rob Norris Subject: Direct Linux loading for bhyve Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spamd-Bar: --- X-Spamd-Result: default: False [-3.57 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.98)[-0.979]; R_DKIM_ALLOW(-0.20)[despairlabs.com:s=fm3,messagingengine.com:s=fm3]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.128/27]; MIME_GOOD(-0.10)[text/plain]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.152:from]; XM_UA_NO_VERSION(0.01)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MIME_TRACE(0.00)[0:+]; RCPT_COUNT_ONE(0.00)[1]; ASN(0.00)[asn:29838, ipnet:64.147.123.0/24, country:US]; FROM_HAS_DN(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[messagingengine.com:dkim]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; DMARC_NA(0.00)[despairlabs.com]; PREVIOUSLY_DELIVERED(0.00)[virtualization@freebsd.org]; MLMMJ_DEST(0.00)[virtualization@freebsd.org]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; DKIM_TRACE(0.00)[despairlabs.com:+,messagingengine.com:+] X-Rspamd-Queue-Id: 4VX71l2DvFz4s1V Hi all, Last year I did some work on adding support to bhyve to load a Linux kernel directly, without needing to create a disk image or configure a bootloader. I showed a few people at the Dev Summit in Taipei in March, and the concept was generally well received, so I'm writing this email to describe where I'm at, where I want to take it and seek comments, ideas and guidance on how to proceed. The initial motivation was to be able to do the equivalent of QEMU's -kernel, -append and -initrd options with bhyve, to boot a Linux kernel directly. (For me, it's to use to port my kernel dev tool "quiz"[1] to FreeBSD, though that is only tangentially related). To do this I added a "loader" class to bhyve, and then wrote a loader that implements the Linux x86 boot protocol. Some links: * Prototype: https://github.com/robn/freebsd-src/tree/bhyve-loader-linux/usr.sbin/bhyve * Demo run using the kernel and initrd from a Debian installer iso:     https://asciinema.org/a/FuXehcd5MkWb7LE15s1VT2ugK I'll describe how it's put together here. loader.h and loader.c define a trivial struct loader, which each loader module defines and adds to loader_set.     struct loader {         const char *l_name;         int (*l_setup_memory)(struct vmctx *ctx);         int (*l_setup_boot_cpu)(struct vmctx *ctx, struct vcpu *vcpu);     };     static const struct loader loader_linux = {         .l_name = "linux",         .l_setup_memory = loader_linux_setup_memory,         .l_setup_boot_cpu = loader_linux_setup_boot_cpu,     };     LOADER_SET(loader_linux); It's pretty straightforward: after memory is created, l_setup_memory() is called to load whatever is wanted into it. Then, once the boot CPU is created, l_setup_boot_cpu() is called to set initial registers and insert anything needed to hook up the final memory map, device state or whatever else. It's not so different to the existing bootrom support (indeed, an early version was just setting it up as an alternate bootrom). The details are in amd64/loader_linux.c. For a second opinion, I wrote a loader_multiboot2.c[2], though it's not finished and not working properly. I suspect it's not very far away but in any case, it does show the shape. Apart from the normal matters of style, documentation, testing, and other "productionising" tasks, there's at least two things to address: * I need a way to create a VM that will be destroyed when bhyve exits,   including if it is killed. KVM does it by binding the VM to a file   descriptor; when the descriptor is closed, the VM is destroyed. That's   a fairly common pattern in Linux for managing lifetimes of kernel   resources from userspace. I'm new enough to FreeBSD to not know what   the common pattern for that kind of thing is (pointers appreciated!).   Regardless, this feels like it's mostly a case of plumbing. * I don't have dedicated command line options yet. '-o loader.name=foo'   is used to select the loader, and any other options the loader has to   sort out by itself (eg the Linux loader knows 'loader.kernel',   'loader.initrd' and 'loader.cmdline'). Maybe we don't _need_ dedicated   command line options; I don't know how to decide that. And then there's a bunch of observations around possible future areas for improvement in both bhyve and libvmm. These aren't showstoppers for some form of loader support landing, but they're definitely places things can be better: * The memory layout story doesn't seem very flexible. The Linux loader   needs to stake out multiple different regions, but there's not really   any help to know what's already claimed, or claim it in turn. I just   have to pick some spots that probably aren't going to be used by any   device mapping or similar. It's not that hard, but it seems like some   kind of allocator concept might be useful. Not unlike the e820   allocator perhaps, but they're not "physical" regions as such (that   is, shouldn't be exposed to the OS in the e820 table). * Semi-related, something that would help a lot is some ability to map a   region of host memory directly into the guest address space. Then   instead of copying things in, we can just mmap() and give the   pointer to the hypervisor directly. This isn't so important for a   loader, but as far as I can tell it's a requirement if QEMU itself   were ever to use bhyve/libvmm for acceleration, as it wants to set up   its memory layout directly and then just hand it to the hypervisor   (getting QEMU running is another side project I have on the go, so   I've thought about this a little bit). * It does seem like this loader concept has some overlap with the   bootrom support, and maybe bootrom should be just another kind of   loader. But, maybe not, since a bootrom is a real device, not just   stuff in memory. * It's really really hard to set up the register state properly. This   might just be a reflection of the complexity of the problem,   especially since I'm trying to set things up so the CPU starts in   64-bit long mode, and there's very few examples of that out there   (even QEMU installs a tiny bootrom to have the guest do the   transitions before bouncing into Linux). Regardless, it seems like   helpers to assist with building the GDT, or setting segment shadow   registers, or control registers, etc, would make this sort of thing a   lot easier (incidentally; there is some help for this within libvmm   just for setting up for a FreeBSD guest; at least, that seems out of   place). I think that's everything for now. I'm very interested in any thoughts, opinions, guidance or complaints people have. I'm also hang around in #bhyve on IRC, on the FreeBSD Discord, and I'll be at BSDCan later this month if you want to chat to me about it. Cheers, Rob. 1. https://despairlabs.com/blog/posts/2024-03-04-quiz-rapid-openzfs-development/ 2. https://github.com/robn/freebsd-src/blob/bhyve-loader-multiboot2/usr.sbin/bhyve/amd64/loader_multiboot2.c From nobody Sun May 5 21:00:25 2024 X-Original-To: virtualization@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4VXcPk41F8z5Kjcw for ; Sun, 05 May 2024 21:00:26 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4VXcPk0Dqxz4bgC for ; Sun, 5 May 2024 21:00:26 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1714942826; a=rsa-sha256; cv=none; b=Bd2FGr8ixSpouFBM5Yp0RkSFW4mC9HRI/WcIKdWs+d3CH+HGEpk8nELixSETBC9JAwm1H9 Dvra8pSpwL/hTS/fODqgf9NC33d9eI4vKjQX3QiHaZftIWGPeQQqWvsltQTB4BvXg45wYV SQcLA8fwbOL5527LbOSYl/YN4HI2NQqxN/k1j5DKbYjGit9uDSj35WfSdGC7AxbEikrn7B bLy4IXMliDTQt0v7JDoPbDt5sZlYhL+Amq5hqe2xyUQjo62AG8wZ8fyQV6WvyzWH7luh0C I5H2nI7sFeSTBaOYt4M7K9h2phrRWsQl29Dt1GVvdKZRr0kvY/ziRqo7mQh93A== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1714942826; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=Xe+hYZjnzVGyjNJ8f4NVWVO8w2U+2h0tqH25yh8XOxA=; b=dSKDuciIW2S/RrkD2ofc/qm/zUKTamkJPfmDcEg03/75x6XJuBHe/gf4Sd2eUJ5s2maZZJ w3GDlxCfLQ8D5l8by1xOLzgvlcoQpzBBlogaBhH5+veYNyDTbp/pUtKsUo0o+0INIFDFxw UaGBtinwaNztPptViU8PotN+l9b92cZPaikShahb65U/usStMC02jOiXIC/qBfGo+TfHyY gJurUO4KrkP0iu6TwkopoBqecd9y/PFj6YqZLgF9ANp3rDpS94UCRjf6O7S5yDVoLrSLNJ EsFeAy0isqx91CD7GqMQ1on5FYDSB804H8DP3lRG/3d+/AB5avsLvrQdiT62mQ== Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4VXcPj6xRhzTFC for ; Sun, 5 May 2024 21:00:25 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 445L0PnV092013 for ; Sun, 5 May 2024 21:00:25 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 445L0PK7092010 for virtualization@FreeBSD.org; Sun, 5 May 2024 21:00:25 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202405052100.445L0PK7092010@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 5 May 2024 21:00:25 +0000 List-Id: Discussion List-Archive: https://lists.freebsd.org/archives/freebsd-virtualization List-Help: List-Post: List-Subscribe: List-Unsubscribe: X-BeenThere: freebsd-virtualization@freebsd.org Sender: owner-freebsd-virtualization@FreeBSD.org MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="17149428252.EBBe8FB.87189" Content-Transfer-Encoding: 7bit --17149428252.EBBe8FB.87189 Date: Sun, 5 May 2024 21:00:25 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- Open | 264267 | UEFI Booting on Azure Generation 2 VMs crashes 1 problems total for which you should take action. --17149428252.EBBe8FB.87189 Date: Sun, 5 May 2024 21:00:25 +0000 MIME-Version: 1.0 Content-Type: text/html; charset="UTF-8" 
The following is a listing of current problems submitted by FreeBSD users,
which need special attention. These represent problem reports covering
all versions including experimental development code and obsolete releases.

Status      |    Bug Id | Description
------------+-----------+---------------------------------------------------
Open        |    264267 | UEFI Booting on Azure Generation 2 VMs crashes

1 problems total for which you should take action.
--17149428252.EBBe8FB.87189--