Date: Sun, 20 Mar 2022 11:43:58 +0100 From: Mario Marietto <marietto2008@gmail.com> To: jason@tubnor.net Cc: Chuck Tuffli <chuck@tuffli.net>, FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: bhyve NVMe 1.4 support Message-ID: <CA%2B1FSiijy0YjX0Nju9kRAY8hsYc42Y70V3tu-RfqCnaRhzLa8A@mail.gmail.com> In-Reply-To: <082b01d80697$64e95030$2ebbf090$@tubnor.net> References: <CAM0tzX1W1Do=uqA3PONyksY4dmob%2BZMi-ib7aECVx6AH3XW6Pw@mail.gmail.com> <00bf01d80104$e6ba5de0$b42f19a0$@tubnor.net> <CAM0tzX1EdQfTDUMU1dNtQHxG9SB3VzNP5UGmHuiHCY5HsxL2QA@mail.gmail.com> <CAM0tzX1qJOuqJWv_04oMvTqQrmLNQf8O%2B8PJ6cjLyh9bLqRmNQ@mail.gmail.com> <082b01d80697$64e95030$2ebbf090$@tubnor.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--000000000000cb9fe205daa41016
Content-Type: text/plain; charset="UTF-8"
Please check this situation :
(I'm on :
FreeBSD marietto 13.0-RELEASE-p8 FreeBSD 13.0-RELEASE-p8 #6
n244863-45b90a014c11: Thu Mar 17 18:42:4
0 CET 2022 marietto@marietto:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
amd64)
ubuntu21-10-nvidia495-vm0.sh :
bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/$vmdisk'p2'/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img
\
-s 2,nvme,/dev/nvd0 \
-s 3,passthru,5/0/0 \
-s 4,passthru,1/0/0 \
-s 8,virtio-net,tap0 \
-s 9,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0 < /dev/null & sleep 2 && vncviewer 0:0
on the Ubuntu 21.10 guest os :
mario@marietto-BHYVE:/home/marietto# fdisk -l
Disk /dev/nvme0n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: bhyve-NVMe
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
The nvme disk is not recognized. This is how it looks on FreeBSD :
1. Name: nvd0
Mediasize: 1000204886016 (932G)
Sectorsize: 512
Mode: r0w0e0
descr: CT1000P1SSD8
lunid: 000000000000000100a07520285f1175
ident: 2022285F1175
rotationrate: 0
fwsectors: 0
fwheads: 0
=> 34 1953525101 nvd0 GPT (932G)
34 2014 - free - (1.0M)
2048 1748992 1 efi (854M)
1751040 921985024 2 ms-basic-data (440G)
923736064 191522816 - free - (91G)
1115258880 833185547 7 ms-basic-data (397G)
1948444427 245 - free - (123K)
1948444672 1318912 3 ms-recovery (644M)
1949763584 2048 - free - (1.0M)
1949765632 1310720 4 ms-recovery (640M)
1951076352 2048 - free - (1.0M)
1951078400 1265657 5 ms-basic-data (618M)
1952344057 7 - free - (3.5K)
1952344064 1179641 6 ms-basic-data (576M)
1953523705 1430 - free - (715K)
Il giorno mar 11 gen 2022 alle ore 04:01 <jason@tubnor.net> ha scritto:
> > -----Original Message-----
> > From: Chuck Tuffli <chuck@tuffli.net>
> > Sent: Tuesday, 11 January 2022 10:58 AM
> > To: jason@tubnor.net
> > Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org>
> > Subject: Re: bhyve NVMe 1.4 support
> >
> > On Wed, Jan 5, 2022 at 4:33 PM Chuck Tuffli <chuck@tuffli.net> wrote:
> > >
> > > On Mon, Jan 3, 2022 at 4:49 PM <jason@tubnor.net> wrote:
> > > >
> > > > Hi Chuck,
> > > >
> > > > Testing on 14.0-CURRENT shows the following:
> > > >
> > > > OpenBSD 6.9 - NVMe read/write OK
> > > > Windows 10 - NVMe read OK, write FAIL Windows Server 2022 - NVMe
> > > > read OK, write FAIL Alma Linux 8.5 - NVMe read OK, write FAIL.
> > >
> > > Thanks for the report, Jason. I have Alma 8.5 installed and am
> investigating.
> >
> > OK, I found and fixed the regression. New version of the file in the same
> > place is up for folks who are interested.
> >
> > --chuck
>
> This version looks good. No regression on the following guest platforms
> under 14.0-CURRENT:
>
> OpenBSD 6.9 - NVMe read/write OK
> Windows 10 - NVMe read/write OK
> Windows Server 2022 - NVMe read/write OK
> Alma Linux 8.5 - NVMe read/write OK
> FreeBSD 13.0 - NVMe read/write OK
>
> From a storage presentation layer in a production perspective, I can't
> find any issues with the proposed update to pci_nvme.c
>
> Some of our tests included:
>
> The removal and addition of partitions on the NVMe presentation layer
> Installation of all the operating system listed above
> Where supported by the guest operating system, the TRIM command executed
> on the presentation layer and guest rebooted to ensure no data corruption
> was caused by TRIM
>
> Cheers,
>
> Jason.
>
>
>
--
Mario.
--000000000000cb9fe205daa41016
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>Please check this situation : <br></div><div><br></di=
v><div>(I'm on :</div><div><br></div><div><span style=3D"font-family:mo=
nospace"><span style=3D"color:rgb(0,0,0);background-color:rgb(255,255,255)"=
>FreeBSD marietto 13.0-RELEASE-p8 FreeBSD 13.0-RELEASE-p8 #6 n244863-45b90a=
014c11: Thu Mar 17 18:42:4</span><br>0 CET 2022 =C2=A0=C2=A0=C2=A0=C2=A0mar=
ietto@marietto:/usr/obj/usr/src/amd64.amd64/sys/GENERIC =C2=A0amd64)<br></s=
pan></div><div><br></div><div><br></div><div>ubuntu21-10-nvidia495-vm0.sh :=
<br>
=09
=09
<div class=3D"gmail-bbCodeBlock gmail-bbCodeBlock--screenLimited gmail-bbCo=
deBlock--code">
=09
<div class=3D"gmail-bbCodeBlock-content" dir=3D"ltr">
<pre class=3D"gmail-bbCodeCode" dir=3D"ltr"><code><br>bhyve -S -c sockets=
=3D2,cores=3D2,threads=3D2 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/$vmdisk'p2'/bhyve/os/Linux/impish-cuda-11-5-nv=
idia-495.img \
-s 2,nvme,/dev/nvd0 \
-s 3,passthru,5/0/0 \
-s 4,passthru,1/0/0 \
-s 8,virtio-net,tap0 \
-s 9,virtio-9p,sharename=3D/ \
-s 29,fbuf,tcp=3D<a href=3D"http://0.0.0.0:5900">0.0.0.0:5900</a>,w=3D1440,=
h=3D900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0 < /dev/null & sleep 2 && vncviewer 0:0</code></pre>
</div>
</div><br>
on the Ubuntu 21.10 guest os :<br>
<br>
=09
=09
<div class=3D"gmail-bbCodeBlock gmail-bbCodeBlock--screenLimited gmail-bbCo=
deBlock--code">
=09
<div class=3D"gmail-bbCodeBlock-content" dir=3D"ltr">
<pre class=3D"gmail-bbCodeCode" dir=3D"ltr"><code>mario@marietto-BHYVE:/h=
ome/marietto# fdisk -l
Disk /dev/nvme0n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: bhyve-NVMe =20
Units: sectors of 1 * 512 =3D 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes</code></pre>
</div>
</div><br>
The nvme disk is not recognized. This is how it looks on FreeBSD :<br></div=
><div><br></div><div><span style=3D"font-family:monospace"><span style=3D"c=
olor:rgb(0,0,0);background-color:rgb(255,255,255)">1. Name: nvd0
</span><br> =C2=A0=C2=A0Mediasize: 1000204886016 (932G)
<br> =C2=A0=C2=A0Sectorsize: 512
<br> =C2=A0=C2=A0Mode: r0w0e0
<br> =C2=A0=C2=A0descr: CT1000P1SSD8
<br> =C2=A0=C2=A0lunid: 000000000000000100a07520285f1175
<br> =C2=A0=C2=A0ident: 2022285F1175
<br> =C2=A0=C2=A0rotationrate: 0
<br> =C2=A0=C2=A0fwsectors: 0
<br> =C2=A0=C2=A0fwheads: 0</span></div><div><span style=3D"font-family:mon=
ospace"><br></span></div><div><span style=3D"font-family:monospace">
<span style=3D"font-family:monospace"><span style=3D"color:rgb(0,0,0);backg=
round-color:rgb(255,255,255)">=3D> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A034 =C2=A01953525101 =C2=A0nvd0 =C2=A0GPT =C2=A0(932G)=C2=A0</span></s=
pan></span></div><div><span style=3D"font-family:monospace"><span style=3D"=
font-family:monospace"><br></span></span></div><div><span style=3D"font-fam=
ily:monospace"><span style=3D"font-family:monospace"> =C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 34 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A020=
14 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(1.0M)
<br> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02048 =C2=A0=C2=A0=C2=A0=C2=
=A01748992 =C2=A0=C2=A0=C2=A0=C2=A01 =C2=A0efi =C2=A0(854M)
<br> =C2=A0=C2=A0=C2=A0=C2=A01751040 =C2=A0=C2=A0921985024 =C2=A0=C2=A0=C2=
=A0=C2=A02 =C2=A0ms-basic-data =C2=A0(440G)
<br> =C2=A0=C2=A0923736064 =C2=A0=C2=A0191522816 =C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0=C2=A0=C2=A0- free - =C2=A0(91G)
<br> =C2=A01115258880 =C2=A0=C2=A0833185547 =C2=A0=C2=A0=C2=A0=C2=A07 =C2=
=A0ms-basic-data =C2=A0(397G)
<br> =C2=A01948444427 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0245 =
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(123K)
<br> =C2=A01948444672 =C2=A0=C2=A0=C2=A0=C2=A01318912 =C2=A0=C2=A0=C2=A0=C2=
=A03 =C2=A0ms-recovery =C2=A0(644M)
<br> =C2=A01949763584 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02048 =C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(1.0M)
<br> =C2=A01949765632 =C2=A0=C2=A0=C2=A0=C2=A01310720 =C2=A0=C2=A0=C2=A0=C2=
=A04 =C2=A0ms-recovery =C2=A0(640M)
<br> =C2=A01951076352 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02048 =C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(1.0M)
<br> =C2=A01951078400 =C2=A0=C2=A0=C2=A0=C2=A01265657 =C2=A0=C2=A0=C2=A0=C2=
=A05 =C2=A0ms-basic-data =C2=A0(618M)
<br> =C2=A01952344057 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A07 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(3.5K)
<br> =C2=A01952344064 =C2=A0=C2=A0=C2=A0=C2=A01179641 =C2=A0=C2=A0=C2=A0=C2=
=A06 =C2=A0ms-basic-data =C2=A0(576M)
<br> =C2=A01953523705 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A01430 =C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0- free - =C2=A0(715K)<br>
<br></span><br><br></span></div></div><br><div class=3D"gmail_quote"><div d=
ir=3D"ltr" class=3D"gmail_attr">Il giorno mar 11 gen 2022 alle ore 04:01 &l=
t;<a href=3D"mailto:jason@tubnor.net">jason@tubnor.net</a>> ha scritto:<=
br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8e=
x;border-left:1px solid rgb(204,204,204);padding-left:1ex">> -----Origin=
al Message-----<br>
> From: Chuck Tuffli <<a href=3D"mailto:chuck@tuffli.net" target=3D"_=
blank">chuck@tuffli.net</a>><br>
> Sent: Tuesday, 11 January 2022 10:58 AM<br>
> To: <a href=3D"mailto:jason@tubnor.net" target=3D"_blank">jason@tubnor=
.net</a><br>
> Cc: FreeBSD virtualization <<a href=3D"mailto:freebsd-virtualizatio=
n@freebsd.org" target=3D"_blank">freebsd-virtualization@freebsd.org</a>>=
<br>
> Subject: Re: bhyve NVMe 1.4 support<br>
> <br>
> On Wed, Jan 5, 2022 at 4:33 PM Chuck Tuffli <<a href=3D"mailto:chuc=
k@tuffli.net" target=3D"_blank">chuck@tuffli.net</a>> wrote:<br>
> ><br>
> > On Mon, Jan 3, 2022 at 4:49 PM <<a href=3D"mailto:jason@tubnor=
.net" target=3D"_blank">jason@tubnor.net</a>> wrote:<br>
> > ><br>
> > > Hi Chuck,<br>
> > ><br>
> > > Testing on 14.0-CURRENT shows the following:<br>
> > ><br>
> > > OpenBSD 6.9 - NVMe read/write OK<br>
> > > Windows 10 - NVMe read OK, write FAIL Windows Server 2022 - =
NVMe<br>
> > > read OK, write FAIL Alma Linux 8.5 - NVMe read OK, write FAI=
L.<br>
> ><br>
> > Thanks for the report, Jason. I have Alma 8.5 installed and am in=
vestigating.<br>
> <br>
> OK, I found and fixed the regression. New version of the file in the s=
ame<br>
> place is up for folks who are interested.<br>
> <br>
> --chuck<br>
<br>
This version looks good. No regression on the following guest platforms und=
er 14.0-CURRENT:<br>
<br>
OpenBSD 6.9 - NVMe read/write OK<br>
Windows 10 - NVMe read/write OK<br>
Windows Server 2022 - NVMe read/write OK<br>
Alma Linux 8.5 - NVMe read/write OK<br>
FreeBSD 13.0 - NVMe read/write OK<br>
<br>
>From a storage presentation layer in a production perspective, I can't =
find any issues with the proposed update to pci_nvme.c<br>
<br>
Some of our tests included:<br>
<br>
The removal and addition of partitions on the NVMe presentation layer<br>
Installation of all the operating system listed above<br>
Where supported by the guest operating system, the TRIM command executed on=
the presentation layer and guest rebooted to ensure no data corruption was=
caused by TRIM<br>
<br>
Cheers,<br>
<br>
Jason.<br>
<br>
<br>
</blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"g=
mail_signature">Mario.<br></div>
--000000000000cb9fe205daa41016--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSiijy0YjX0Nju9kRAY8hsYc42Y70V3tu-RfqCnaRhzLa8A>