Date: Sun, 20 Mar 2022 11:43:58 +0100 From: Mario Marietto <marietto2008@gmail.com> To: jason@tubnor.net Cc: Chuck Tuffli <chuck@tuffli.net>, FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: bhyve NVMe 1.4 support Message-ID: <CA%2B1FSiijy0YjX0Nju9kRAY8hsYc42Y70V3tu-RfqCnaRhzLa8A@mail.gmail.com> In-Reply-To: <082b01d80697$64e95030$2ebbf090$@tubnor.net> References: <CAM0tzX1W1Do=uqA3PONyksY4dmob%2BZMi-ib7aECVx6AH3XW6Pw@mail.gmail.com> <00bf01d80104$e6ba5de0$b42f19a0$@tubnor.net> <CAM0tzX1EdQfTDUMU1dNtQHxG9SB3VzNP5UGmHuiHCY5HsxL2QA@mail.gmail.com> <CAM0tzX1qJOuqJWv_04oMvTqQrmLNQf8O%2B8PJ6cjLyh9bLqRmNQ@mail.gmail.com> <082b01d80697$64e95030$2ebbf090$@tubnor.net>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --]
Please check this situation :
(I'm on :
FreeBSD marietto 13.0-RELEASE-p8 FreeBSD 13.0-RELEASE-p8 #6
n244863-45b90a014c11: Thu Mar 17 18:42:4
0 CET 2022 marietto@marietto:/usr/obj/usr/src/amd64.amd64/sys/GENERIC
amd64)
ubuntu21-10-nvidia495-vm0.sh :
bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/$vmdisk'p2'/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img
\
-s 2,nvme,/dev/nvd0 \
-s 3,passthru,5/0/0 \
-s 4,passthru,1/0/0 \
-s 8,virtio-net,tap0 \
-s 9,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=0.0.0.0:5900,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0 < /dev/null & sleep 2 && vncviewer 0:0
on the Ubuntu 21.10 guest os :
mario@marietto-BHYVE:/home/marietto# fdisk -l
Disk /dev/nvme0n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: bhyve-NVMe
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
The nvme disk is not recognized. This is how it looks on FreeBSD :
1. Name: nvd0
Mediasize: 1000204886016 (932G)
Sectorsize: 512
Mode: r0w0e0
descr: CT1000P1SSD8
lunid: 000000000000000100a07520285f1175
ident: 2022285F1175
rotationrate: 0
fwsectors: 0
fwheads: 0
=> 34 1953525101 nvd0 GPT (932G)
34 2014 - free - (1.0M)
2048 1748992 1 efi (854M)
1751040 921985024 2 ms-basic-data (440G)
923736064 191522816 - free - (91G)
1115258880 833185547 7 ms-basic-data (397G)
1948444427 245 - free - (123K)
1948444672 1318912 3 ms-recovery (644M)
1949763584 2048 - free - (1.0M)
1949765632 1310720 4 ms-recovery (640M)
1951076352 2048 - free - (1.0M)
1951078400 1265657 5 ms-basic-data (618M)
1952344057 7 - free - (3.5K)
1952344064 1179641 6 ms-basic-data (576M)
1953523705 1430 - free - (715K)
Il giorno mar 11 gen 2022 alle ore 04:01 <jason@tubnor.net> ha scritto:
> > -----Original Message-----
> > From: Chuck Tuffli <chuck@tuffli.net>
> > Sent: Tuesday, 11 January 2022 10:58 AM
> > To: jason@tubnor.net
> > Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org>
> > Subject: Re: bhyve NVMe 1.4 support
> >
> > On Wed, Jan 5, 2022 at 4:33 PM Chuck Tuffli <chuck@tuffli.net> wrote:
> > >
> > > On Mon, Jan 3, 2022 at 4:49 PM <jason@tubnor.net> wrote:
> > > >
> > > > Hi Chuck,
> > > >
> > > > Testing on 14.0-CURRENT shows the following:
> > > >
> > > > OpenBSD 6.9 - NVMe read/write OK
> > > > Windows 10 - NVMe read OK, write FAIL Windows Server 2022 - NVMe
> > > > read OK, write FAIL Alma Linux 8.5 - NVMe read OK, write FAIL.
> > >
> > > Thanks for the report, Jason. I have Alma 8.5 installed and am
> investigating.
> >
> > OK, I found and fixed the regression. New version of the file in the same
> > place is up for folks who are interested.
> >
> > --chuck
>
> This version looks good. No regression on the following guest platforms
> under 14.0-CURRENT:
>
> OpenBSD 6.9 - NVMe read/write OK
> Windows 10 - NVMe read/write OK
> Windows Server 2022 - NVMe read/write OK
> Alma Linux 8.5 - NVMe read/write OK
> FreeBSD 13.0 - NVMe read/write OK
>
> From a storage presentation layer in a production perspective, I can't
> find any issues with the proposed update to pci_nvme.c
>
> Some of our tests included:
>
> The removal and addition of partitions on the NVMe presentation layer
> Installation of all the operating system listed above
> Where supported by the guest operating system, the TRIM command executed
> on the presentation layer and guest rebooted to ensure no data corruption
> was caused by TRIM
>
> Cheers,
>
> Jason.
>
>
>
--
Mario.
[-- Attachment #2 --]
<div dir="ltr"><div>Please check this situation : <br></div><div><br></div><div>(I'm on :</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">FreeBSD marietto 13.0-RELEASE-p8 FreeBSD 13.0-RELEASE-p8 #6 n244863-45b90a014c11: Thu Mar 17 18:42:4</span><br>0 CET 2022 marietto@marietto:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64)<br></span></div><div><br></div><div><br></div><div>ubuntu21-10-nvidia495-vm0.sh :<br>
<div class="gmail-bbCodeBlock gmail-bbCodeBlock--screenLimited gmail-bbCodeBlock--code">
<div class="gmail-bbCodeBlock-content" dir="ltr">
<pre class="gmail-bbCodeCode" dir="ltr"><code><br>bhyve -S -c sockets=2,cores=2,threads=2 -m 8G -w -H -A \
-s 0,hostbridge \
-s 1,virtio-blk,/mnt/$vmdisk'p2'/bhyve/os/Linux/impish-cuda-11-5-nvidia-495.img \
-s 2,nvme,/dev/nvd0 \
-s 3,passthru,5/0/0 \
-s 4,passthru,1/0/0 \
-s 8,virtio-net,tap0 \
-s 9,virtio-9p,sharename=/ \
-s 29,fbuf,tcp=<a href="http://0.0.0.0:5900">0.0.0.0:5900</a>,w=1440,h=900 \
-s 30,xhci,tablet \
-s 31,lpc \
-l bootrom,/usr/local/share/uefi-firmware/BHYVE_BHF_CODE.fd \
vm0 < /dev/null & sleep 2 && vncviewer 0:0</code></pre>
</div>
</div><br>
on the Ubuntu 21.10 guest os :<br>
<br>
<div class="gmail-bbCodeBlock gmail-bbCodeBlock--screenLimited gmail-bbCodeBlock--code">
<div class="gmail-bbCodeBlock-content" dir="ltr">
<pre class="gmail-bbCodeCode" dir="ltr"><code>mario@marietto-BHYVE:/home/marietto# fdisk -l
Disk /dev/nvme0n1: 931,51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: bhyve-NVMe
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes</code></pre>
</div>
</div><br>
The nvme disk is not recognized. This is how it looks on FreeBSD :<br></div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">1. Name: nvd0
</span><br> Mediasize: 1000204886016 (932G)
<br> Sectorsize: 512
<br> Mode: r0w0e0
<br> descr: CT1000P1SSD8
<br> lunid: 000000000000000100a07520285f1175
<br> ident: 2022285F1175
<br> rotationrate: 0
<br> fwsectors: 0
<br> fwheads: 0</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">
<span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">=> 34 1953525101 nvd0 GPT (932G) </span></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"><br></span></span></div><div><span style="font-family:monospace"><span style="font-family:monospace"> 34 2014 - free - (1.0M)
<br> 2048 1748992 1 efi (854M)
<br> 1751040 921985024 2 ms-basic-data (440G)
<br> 923736064 191522816 - free - (91G)
<br> 1115258880 833185547 7 ms-basic-data (397G)
<br> 1948444427 245 - free - (123K)
<br> 1948444672 1318912 3 ms-recovery (644M)
<br> 1949763584 2048 - free - (1.0M)
<br> 1949765632 1310720 4 ms-recovery (640M)
<br> 1951076352 2048 - free - (1.0M)
<br> 1951078400 1265657 5 ms-basic-data (618M)
<br> 1952344057 7 - free - (3.5K)
<br> 1952344064 1179641 6 ms-basic-data (576M)
<br> 1953523705 1430 - free - (715K)<br>
<br></span><br><br></span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mar 11 gen 2022 alle ore 04:01 <<a href="mailto:jason@tubnor.net">jason@tubnor.net</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> -----Original Message-----<br>
> From: Chuck Tuffli <<a href="mailto:chuck@tuffli.net" target="_blank">chuck@tuffli.net</a>><br>
> Sent: Tuesday, 11 January 2022 10:58 AM<br>
> To: <a href="mailto:jason@tubnor.net" target="_blank">jason@tubnor.net</a><br>
> Cc: FreeBSD virtualization <<a href="mailto:freebsd-virtualization@freebsd.org" target="_blank">freebsd-virtualization@freebsd.org</a>><br>
> Subject: Re: bhyve NVMe 1.4 support<br>
> <br>
> On Wed, Jan 5, 2022 at 4:33 PM Chuck Tuffli <<a href="mailto:chuck@tuffli.net" target="_blank">chuck@tuffli.net</a>> wrote:<br>
> ><br>
> > On Mon, Jan 3, 2022 at 4:49 PM <<a href="mailto:jason@tubnor.net" target="_blank">jason@tubnor.net</a>> wrote:<br>
> > ><br>
> > > Hi Chuck,<br>
> > ><br>
> > > Testing on 14.0-CURRENT shows the following:<br>
> > ><br>
> > > OpenBSD 6.9 - NVMe read/write OK<br>
> > > Windows 10 - NVMe read OK, write FAIL Windows Server 2022 - NVMe<br>
> > > read OK, write FAIL Alma Linux 8.5 - NVMe read OK, write FAIL.<br>
> ><br>
> > Thanks for the report, Jason. I have Alma 8.5 installed and am investigating.<br>
> <br>
> OK, I found and fixed the regression. New version of the file in the same<br>
> place is up for folks who are interested.<br>
> <br>
> --chuck<br>
<br>
This version looks good. No regression on the following guest platforms under 14.0-CURRENT:<br>
<br>
OpenBSD 6.9 - NVMe read/write OK<br>
Windows 10 - NVMe read/write OK<br>
Windows Server 2022 - NVMe read/write OK<br>
Alma Linux 8.5 - NVMe read/write OK<br>
FreeBSD 13.0 - NVMe read/write OK<br>
<br>
>From a storage presentation layer in a production perspective, I can't find any issues with the proposed update to pci_nvme.c<br>
<br>
Some of our tests included:<br>
<br>
The removal and addition of partitions on the NVMe presentation layer<br>
Installation of all the operating system listed above<br>
Where supported by the guest operating system, the TRIM command executed on the presentation layer and guest rebooted to ensure no data corruption was caused by TRIM<br>
<br>
Cheers,<br>
<br>
Jason.<br>
<br>
<br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Mario.<br></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSiijy0YjX0Nju9kRAY8hsYc42Y70V3tu-RfqCnaRhzLa8A>