Date: Sun, 20 Mar 2022 20:02:41 +0100 From: Mario Marietto <marietto2008@gmail.com> To: Chuck Tuffli <chuck@tuffli.net> Cc: jason@tubnor.net, FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: bhyve NVMe 1.4 support Message-ID: <CA%2B1FSijNKxNsHqHCTPE3swh_5ePc-_jtNHFtmBTbUbauEDPung@mail.gmail.com> In-Reply-To: <CAM0tzX03Sw23acPW8ZRcwXO2Rze12OuvBoO6QSUfePpfBdrbWA@mail.gmail.com> References: <CAM0tzX1W1Do=uqA3PONyksY4dmob%2BZMi-ib7aECVx6AH3XW6Pw@mail.gmail.com> <00bf01d80104$e6ba5de0$b42f19a0$@tubnor.net> <CAM0tzX1EdQfTDUMU1dNtQHxG9SB3VzNP5UGmHuiHCY5HsxL2QA@mail.gmail.com> <CAM0tzX1qJOuqJWv_04oMvTqQrmLNQf8O%2B8PJ6cjLyh9bLqRmNQ@mail.gmail.com> <082b01d80697$64e95030$2ebbf090$@tubnor.net> <CA%2B1FSiijy0YjX0Nju9kRAY8hsYc42Y70V3tu-RfqCnaRhzLa8A@mail.gmail.com> <CAM0tzX3SCOS2nOKaODVF2TCTOY_5F5sdqELA666uFRZ=ZreXpg@mail.gmail.com> <CA%2B1FSijNU7Nn9UZujU%2BCLgJkaYFrbfdu37RT7s_r%2Be08AMG0Pw@mail.gmail.com> <CAM0tzX03Sw23acPW8ZRcwXO2Rze12OuvBoO6QSUfePpfBdrbWA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] on FreeBSD : root@marietto:/usr/home/marietto/bhyve # hd -n 256 /dev/nvd0 00000000 33 c0 8e d0 bc 00 7c 8e c0 8e d8 be 00 7c bf 00 |3.....|......|..| 00000010 06 b9 00 02 fc f3 a4 50 68 1c 06 cb fb b9 04 00 |.......Ph.......| 00000020 bd be 07 80 7e 00 00 7c 0b 0f 85 0e 01 83 c5 10 |....~..|........| 00000030 e2 f1 cd 18 88 56 00 55 c6 46 11 05 c6 46 10 00 |.....V.U.F...F..| 00000040 b4 41 bb aa 55 cd 13 5d 72 0f 81 fb 55 aa 75 09 |.A..U..]r...U.u.| 00000050 f7 c1 01 00 74 03 fe 46 10 66 60 80 7e 10 00 74 |....t..F.f`.~..t| 00000060 26 66 68 00 00 00 00 66 ff 76 08 68 00 00 68 00 |&fh....f.v.h..h.| 00000070 7c 68 01 00 68 10 00 b4 42 8a 56 00 8b f4 cd 13 ||h..h...B.V.....| 00000080 9f 83 c4 10 9e eb 14 b8 01 02 bb 00 7c 8a 56 00 |............|.V.| 00000090 8a 76 01 8a 4e 02 8a 6e 03 cd 13 66 61 73 1c fe |.v..N..n...fas..| 000000a0 4e 11 75 0c 80 7e 00 80 0f 84 8a 00 b2 80 eb 84 |N.u..~..........| 000000b0 55 32 e4 8a 56 00 cd 13 5d eb 9e 81 3e fe 7d 55 |U2..V...]...>.}U| 000000c0 aa 75 6e ff 76 00 e8 8d 00 75 17 fa b0 d1 e6 64 |.un.v....u.....d| 000000d0 e8 83 00 b0 df e6 60 e8 7c 00 b0 ff e6 64 e8 75 |......`.|....d.u| 000000e0 00 fb b8 00 bb cd 1a 66 23 c0 75 3b 66 81 fb 54 |.......f#.u;f..T| 000000f0 43 50 41 75 32 81 f9 02 01 72 2c 66 68 07 bb 00 |CPAu2....r,fh...| 00000100 On Ubuntu 21.10 : root@marietto-BHYVE:/home/marietto# hd -n 256 /dev/nvme0n1 00000000 33 c0 8e d0 bc 00 7c 8e c0 8e d8 be 00 7c bf 00 |3.....|......|..| 00000010 06 b9 00 02 fc f3 a4 50 68 1c 06 cb fb b9 04 00 |.......Ph.......| 00000020 bd be 07 80 7e 00 00 7c 0b 0f 85 0e 01 83 c5 10 |....~..|........| 00000030 e2 f1 cd 18 88 56 00 55 c6 46 11 05 c6 46 10 00 |.....V.U.F...F..| 00000040 b4 41 bb aa 55 cd 13 5d 72 0f 81 fb 55 aa 75 09 |.A..U..]r...U.u.| 00000050 f7 c1 01 00 74 03 fe 46 10 66 60 80 7e 10 00 74 |....t..F.f`.~..t| 00000060 26 66 68 00 00 00 00 66 ff 76 08 68 00 00 68 00 |&fh....f.v.h..h.| 00000070 7c 68 01 00 68 10 00 b4 42 8a 56 00 8b f4 cd 13 ||h..h...B.V.....| 00000080 9f 83 c4 10 9e eb 14 b8 01 02 bb 00 7c 8a 56 00 |............|.V.| 00000090 8a 76 01 8a 4e 02 8a 6e 03 cd 13 66 61 73 1c fe |.v..N..n...fas..| 000000a0 4e 11 75 0c 80 7e 00 80 0f 84 8a 00 b2 80 eb 84 |N.u..~..........| 000000b0 55 32 e4 8a 56 00 cd 13 5d eb 9e 81 3e fe 7d 55 |U2..V...]...>.}U| 000000c0 aa 75 6e ff 76 00 e8 8d 00 75 17 fa b0 d1 e6 64 |.un.v....u.....d| 000000d0 e8 83 00 b0 df e6 60 e8 7c 00 b0 ff e6 64 e8 75 |......`.|....d.u| 000000e0 00 fb b8 00 bb cd 1a 66 23 c0 75 3b 66 81 fb 54 |.......f#.u;f..T| 000000f0 43 50 41 75 32 81 f9 02 01 72 2c 66 68 07 bb 00 |CPAu2....r,fh...| 00000100 Il giorno dom 20 mar 2022 alle ore 19:35 Chuck Tuffli <chuck@tuffli.net> ha scritto: > On Sun, Mar 20, 2022 at 8:13 AM Mario Marietto <marietto2008@gmail.com> > wrote: > > > > ----> Help me understand what "not recognized" means. The device nvme0n1 > : > > > > I don't see the partitions that are stored inside the disk nvme. And I'm > not able to mount the NTFS partition that's mapped as nvd0p2 under FreeBSD. > > Thank you, I understand the question now. As an experiment, I created > a zvol, copied a FreeBSD disk image to it, and verified that fdisk > showed what I expected on the guest. I.e. : > > # zfs create -V 20G zroot/vmvol/gptdisk > # dd if=/vms/.img/FreeBSD-14.0-CURRENT-amd64.raw > of=/dev/zvol/zroot/vmvol/gptdisk bs=1m > # gpart recover zvol/zroot/vmvol/gptdisk > < add /dev/zvol/zroot/vmvol/gptdisk to test-vm configuration > > # vm start test-vm > # ssh root@test-vm lsb_release -a > Distributor ID: Ubuntu > Description: Ubuntu 21.04 > Release: 21.04 > Codename: hirsute > No LSB modules are available. > # ssh root@test-vm fdisk -l /dev/nvme1n1 > Disk /dev/nvme1n1: 20 GiB, 21474836480 bytes, 41943040 sectors > Disk model: bhyve-NVMe > Units: sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disklabel type: gpt > Disk identifier: 1B73327C-EAE2-11EB-90A0-002590EC5BF2 > > Device Start End Sectors Size Type > /dev/nvme1n1p1 3 129 127 63.5K FreeBSD boot > /dev/nvme1n1p2 130 66713 66584 32.5M EFI System > /dev/nvme1n1p3 66714 2163865 2097152 1G FreeBSD swap > /dev/nvme1n1p4 2163866 10552473 8388608 4G FreeBSD UFS > > The bhyve invocation is: > bhyve -c 2 -m 2G -Hw \ > -s 0,hostbridge \ > -s 4:0,virtio-blk,/dev/zvol/zroot/vms/test-vm/disk0 \ > -s 5:0,nvme,/dev/zvol/zroot/vmvol/disk0 \ > -s 6:0,nvme,/dev/zvol/zroot/vmvol/gptdisk \ > -s 7:0,virtio-net,tap0,mac=58:9c:fc:0b:ed:d6 > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ > -l com1,/dev/nmdm-test-vm.1A \ > test-vm > > Note that my experiment is different from what you are doing. But I > would expect a Zvol and raw block device to behave the same as a > backing-store for an emulated NVMe drive in bhyve. My experiment ran > on -current, but the behavior in this area should be identical to the > 13.0-p8 version you are using. > > Does the output on the host of > # hd -n 256 /dev/nvd0 > match the output on the guest of > # hd -n 256 /dev/nvme0n1 > ? > -- Mario. [-- Attachment #2 --] <div dir="ltr"><div>on FreeBSD :</div><div><br></div><div><span style="font-family:monospace"><span style="color:rgb(0,0,0);background-color:rgb(255,255,255)">root@marietto:/usr/home/marietto/bhyve # hd -n 256 /dev/nvd0 </span><br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">00000000 33 c0 8e d0 bc 00 7c 8e c0 8e d8 be 00 7c bf 00 |3.....|......|..| <br>00000010 06 b9 00 02 fc f3 a4 50 68 1c 06 cb fb b9 04 00 |.......Ph.......| <br>00000020 bd be 07 80 7e 00 00 7c 0b 0f 85 0e 01 83 c5 10 |....~..|........| <br>00000030 e2 f1 cd 18 88 56 00 55 c6 46 11 05 c6 46 10 00 |.....V.U.F...F..| <br>00000040 b4 41 bb aa 55 cd 13 5d 72 0f 81 fb 55 aa 75 09 |.A..U..]r...U.u.| <br>00000050 f7 c1 01 00 74 03 fe 46 10 66 60 80 7e 10 00 74 |....t..F.f`.~..t| <br>00000060 26 66 68 00 00 00 00 66 ff 76 08 68 00 00 68 00 |&fh....f.v.h..h.| <br>00000070 7c 68 01 00 68 10 00 b4 42 8a 56 00 8b f4 cd 13 ||h..h...B.V.....| <br>00000080 9f 83 c4 10 9e eb 14 b8 01 02 bb 00 7c 8a 56 00 |............|.V.| <br>00000090 8a 76 01 8a 4e 02 8a 6e 03 cd 13 66 61 73 1c fe |.v..N..n...fas..| <br>000000a0 4e 11 75 0c 80 7e 00 80 0f 84 8a 00 b2 80 eb 84 |N.u..~..........| <br>000000b0 55 32 e4 8a 56 00 cd 13 5d eb 9e 81 3e fe 7d 55 |U2..V...]...>.}U| <br>000000c0 aa 75 6e ff 76 00 e8 8d 00 75 17 fa b0 d1 e6 64 |.un.v....u.....d| <br>000000d0 e8 83 00 b0 df e6 60 e8 7c 00 b0 ff e6 64 e8 75 |......`.|....d.u| <br>000000e0 00 fb b8 00 bb cd 1a 66 23 c0 75 3b 66 81 fb 54 |.......f#.u;f..T| <br>000000f0 43 50 41 75 32 81 f9 02 01 72 2c 66 68 07 bb 00 |CPAu2....r,fh...| <br>00000100<br> </span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">On Ubuntu 21.10 :</span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">root@marietto-BHYVE:/home/marietto# hd -n 256 /dev/nvme0n1<br></span></div><div><span style="font-family:monospace"><br></span></div><div><span style="font-family:monospace">00000000 33 c0 8e d0 bc 00 7c 8e c0 8e d8 be 00 7c bf 00 |3.....|......|..|<br>00000010 06 b9 00 02 fc f3 a4 50 68 1c 06 cb fb b9 04 00 |.......Ph.......|<br>00000020 bd be 07 80 7e 00 00 7c 0b 0f 85 0e 01 83 c5 10 |....~..|........|<br>00000030 e2 f1 cd 18 88 56 00 55 c6 46 11 05 c6 46 10 00 |.....V.U.F...F..|<br>00000040 b4 41 bb aa 55 cd 13 5d 72 0f 81 fb 55 aa 75 09 |.A..U..]r...U.u.|<br>00000050 f7 c1 01 00 74 03 fe 46 10 66 60 80 7e 10 00 74 |....t..F.f`.~..t|<br>00000060 26 66 68 00 00 00 00 66 ff 76 08 68 00 00 68 00 |&fh....f.v.h..h.|<br>00000070 7c 68 01 00 68 10 00 b4 42 8a 56 00 8b f4 cd 13 ||h..h...B.V.....|<br>00000080 9f 83 c4 10 9e eb 14 b8 01 02 bb 00 7c 8a 56 00 |............|.V.|<br>00000090 8a 76 01 8a 4e 02 8a 6e 03 cd 13 66 61 73 1c fe |.v..N..n...fas..|<br>000000a0 4e 11 75 0c 80 7e 00 80 0f 84 8a 00 b2 80 eb 84 |N.u..~..........|<br>000000b0 55 32 e4 8a 56 00 cd 13 5d eb 9e 81 3e fe 7d 55 |U2..V...]...>.}U|<br>000000c0 aa 75 6e ff 76 00 e8 8d 00 75 17 fa b0 d1 e6 64 |.un.v....u.....d|<br>000000d0 e8 83 00 b0 df e6 60 e8 7c 00 b0 ff e6 64 e8 75 |......`.|....d.u|<br>000000e0 00 fb b8 00 bb cd 1a 66 23 c0 75 3b 66 81 fb 54 |.......f#.u;f..T|<br>000000f0 43 50 41 75 32 81 f9 02 01 72 2c 66 68 07 bb 00 |CPAu2....r,fh...|<br>00000100</span></div><div><span style="font-family:monospace"><br></span></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno dom 20 mar 2022 alle ore 19:35 Chuck Tuffli <<a href="mailto:chuck@tuffli.net">chuck@tuffli.net</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Sun, Mar 20, 2022 at 8:13 AM Mario Marietto <<a href="mailto:marietto2008@gmail.com" target="_blank">marietto2008@gmail.com</a>> wrote:<br> ><br> > ----> Help me understand what "not recognized" means. The device nvme0n1 :<br> ><br> > I don't see the partitions that are stored inside the disk nvme. And I'm not able to mount the NTFS partition that's mapped as nvd0p2 under FreeBSD.<br> <br> Thank you, I understand the question now. As an experiment, I created<br> a zvol, copied a FreeBSD disk image to it, and verified that fdisk<br> showed what I expected on the guest. I.e. :<br> <br> # zfs create -V 20G zroot/vmvol/gptdisk<br> # dd if=/vms/.img/FreeBSD-14.0-CURRENT-amd64.raw<br> of=/dev/zvol/zroot/vmvol/gptdisk bs=1m<br> # gpart recover zvol/zroot/vmvol/gptdisk<br> < add /dev/zvol/zroot/vmvol/gptdisk to test-vm configuration ><br> # vm start test-vm<br> # ssh root@test-vm lsb_release -a<br> Distributor ID: Ubuntu<br> Description: Ubuntu 21.04<br> Release: 21.04<br> Codename: hirsute<br> No LSB modules are available.<br> # ssh root@test-vm fdisk -l /dev/nvme1n1<br> Disk /dev/nvme1n1: 20 GiB, 21474836480 bytes, 41943040 sectors<br> Disk model: bhyve-NVMe<br> Units: sectors of 1 * 512 = 512 bytes<br> Sector size (logical/physical): 512 bytes / 512 bytes<br> I/O size (minimum/optimal): 512 bytes / 512 bytes<br> Disklabel type: gpt<br> Disk identifier: 1B73327C-EAE2-11EB-90A0-002590EC5BF2<br> <br> Device Start End Sectors Size Type<br> /dev/nvme1n1p1 3 129 127 63.5K FreeBSD boot<br> /dev/nvme1n1p2 130 66713 66584 32.5M EFI System<br> /dev/nvme1n1p3 66714 2163865 2097152 1G FreeBSD swap<br> /dev/nvme1n1p4 2163866 10552473 8388608 4G FreeBSD UFS<br> <br> The bhyve invocation is:<br> bhyve -c 2 -m 2G -Hw \<br> -s 0,hostbridge \<br> -s 4:0,virtio-blk,/dev/zvol/zroot/vms/test-vm/disk0 \<br> -s 5:0,nvme,/dev/zvol/zroot/vmvol/disk0 \<br> -s 6:0,nvme,/dev/zvol/zroot/vmvol/gptdisk \<br> -s 7:0,virtio-net,tap0,mac=58:9c:fc:0b:ed:d6<br> -s 31,lpc \<br> -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \<br> -l com1,/dev/nmdm-test-vm.1A \<br> test-vm<br> <br> Note that my experiment is different from what you are doing. But I<br> would expect a Zvol and raw block device to behave the same as a<br> backing-store for an emulated NVMe drive in bhyve. My experiment ran<br> on -current, but the behavior in this area should be identical to the<br> 13.0-p8 version you are using.<br> <br> Does the output on the host of<br> # hd -n 256 /dev/nvd0<br> match the output on the guest of<br> # hd -n 256 /dev/nvme0n1<br> ?<br> </blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Mario.<br></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSijNKxNsHqHCTPE3swh_5ePc-_jtNHFtmBTbUbauEDPung>