Date: Sun, 20 Mar 2022 20:02:41 +0100 From: Mario Marietto <marietto2008@gmail.com> To: Chuck Tuffli <chuck@tuffli.net> Cc: jason@tubnor.net, FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: bhyve NVMe 1.4 support Message-ID: <CA%2B1FSijNKxNsHqHCTPE3swh_5ePc-_jtNHFtmBTbUbauEDPung@mail.gmail.com> In-Reply-To: <CAM0tzX03Sw23acPW8ZRcwXO2Rze12OuvBoO6QSUfePpfBdrbWA@mail.gmail.com> References: <CAM0tzX1W1Do=uqA3PONyksY4dmob%2BZMi-ib7aECVx6AH3XW6Pw@mail.gmail.com> <00bf01d80104$e6ba5de0$b42f19a0$@tubnor.net> <CAM0tzX1EdQfTDUMU1dNtQHxG9SB3VzNP5UGmHuiHCY5HsxL2QA@mail.gmail.com> <CAM0tzX1qJOuqJWv_04oMvTqQrmLNQf8O%2B8PJ6cjLyh9bLqRmNQ@mail.gmail.com> <082b01d80697$64e95030$2ebbf090$@tubnor.net> <CA%2B1FSiijy0YjX0Nju9kRAY8hsYc42Y70V3tu-RfqCnaRhzLa8A@mail.gmail.com> <CAM0tzX3SCOS2nOKaODVF2TCTOY_5F5sdqELA666uFRZ=ZreXpg@mail.gmail.com> <CA%2B1FSijNU7Nn9UZujU%2BCLgJkaYFrbfdu37RT7s_r%2Be08AMG0Pw@mail.gmail.com> <CAM0tzX03Sw23acPW8ZRcwXO2Rze12OuvBoO6QSUfePpfBdrbWA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000052d23605daab0835 Content-Type: text/plain; charset="UTF-8" on FreeBSD : root@marietto:/usr/home/marietto/bhyve # hd -n 256 /dev/nvd0 00000000 33 c0 8e d0 bc 00 7c 8e c0 8e d8 be 00 7c bf 00 |3.....|......|..| 00000010 06 b9 00 02 fc f3 a4 50 68 1c 06 cb fb b9 04 00 |.......Ph.......| 00000020 bd be 07 80 7e 00 00 7c 0b 0f 85 0e 01 83 c5 10 |....~..|........| 00000030 e2 f1 cd 18 88 56 00 55 c6 46 11 05 c6 46 10 00 |.....V.U.F...F..| 00000040 b4 41 bb aa 55 cd 13 5d 72 0f 81 fb 55 aa 75 09 |.A..U..]r...U.u.| 00000050 f7 c1 01 00 74 03 fe 46 10 66 60 80 7e 10 00 74 |....t..F.f`.~..t| 00000060 26 66 68 00 00 00 00 66 ff 76 08 68 00 00 68 00 |&fh....f.v.h..h.| 00000070 7c 68 01 00 68 10 00 b4 42 8a 56 00 8b f4 cd 13 ||h..h...B.V.....| 00000080 9f 83 c4 10 9e eb 14 b8 01 02 bb 00 7c 8a 56 00 |............|.V.| 00000090 8a 76 01 8a 4e 02 8a 6e 03 cd 13 66 61 73 1c fe |.v..N..n...fas..| 000000a0 4e 11 75 0c 80 7e 00 80 0f 84 8a 00 b2 80 eb 84 |N.u..~..........| 000000b0 55 32 e4 8a 56 00 cd 13 5d eb 9e 81 3e fe 7d 55 |U2..V...]...>.}U| 000000c0 aa 75 6e ff 76 00 e8 8d 00 75 17 fa b0 d1 e6 64 |.un.v....u.....d| 000000d0 e8 83 00 b0 df e6 60 e8 7c 00 b0 ff e6 64 e8 75 |......`.|....d.u| 000000e0 00 fb b8 00 bb cd 1a 66 23 c0 75 3b 66 81 fb 54 |.......f#.u;f..T| 000000f0 43 50 41 75 32 81 f9 02 01 72 2c 66 68 07 bb 00 |CPAu2....r,fh...| 00000100 On Ubuntu 21.10 : root@marietto-BHYVE:/home/marietto# hd -n 256 /dev/nvme0n1 00000000 33 c0 8e d0 bc 00 7c 8e c0 8e d8 be 00 7c bf 00 |3.....|......|..| 00000010 06 b9 00 02 fc f3 a4 50 68 1c 06 cb fb b9 04 00 |.......Ph.......| 00000020 bd be 07 80 7e 00 00 7c 0b 0f 85 0e 01 83 c5 10 |....~..|........| 00000030 e2 f1 cd 18 88 56 00 55 c6 46 11 05 c6 46 10 00 |.....V.U.F...F..| 00000040 b4 41 bb aa 55 cd 13 5d 72 0f 81 fb 55 aa 75 09 |.A..U..]r...U.u.| 00000050 f7 c1 01 00 74 03 fe 46 10 66 60 80 7e 10 00 74 |....t..F.f`.~..t| 00000060 26 66 68 00 00 00 00 66 ff 76 08 68 00 00 68 00 |&fh....f.v.h..h.| 00000070 7c 68 01 00 68 10 00 b4 42 8a 56 00 8b f4 cd 13 ||h..h...B.V.....| 00000080 9f 83 c4 10 9e eb 14 b8 01 02 bb 00 7c 8a 56 00 |............|.V.| 00000090 8a 76 01 8a 4e 02 8a 6e 03 cd 13 66 61 73 1c fe |.v..N..n...fas..| 000000a0 4e 11 75 0c 80 7e 00 80 0f 84 8a 00 b2 80 eb 84 |N.u..~..........| 000000b0 55 32 e4 8a 56 00 cd 13 5d eb 9e 81 3e fe 7d 55 |U2..V...]...>.}U| 000000c0 aa 75 6e ff 76 00 e8 8d 00 75 17 fa b0 d1 e6 64 |.un.v....u.....d| 000000d0 e8 83 00 b0 df e6 60 e8 7c 00 b0 ff e6 64 e8 75 |......`.|....d.u| 000000e0 00 fb b8 00 bb cd 1a 66 23 c0 75 3b 66 81 fb 54 |.......f#.u;f..T| 000000f0 43 50 41 75 32 81 f9 02 01 72 2c 66 68 07 bb 00 |CPAu2....r,fh...| 00000100 Il giorno dom 20 mar 2022 alle ore 19:35 Chuck Tuffli <chuck@tuffli.net> ha scritto: > On Sun, Mar 20, 2022 at 8:13 AM Mario Marietto <marietto2008@gmail.com> > wrote: > > > > ----> Help me understand what "not recognized" means. The device nvme0n1 > : > > > > I don't see the partitions that are stored inside the disk nvme. And I'm > not able to mount the NTFS partition that's mapped as nvd0p2 under FreeBSD. > > Thank you, I understand the question now. As an experiment, I created > a zvol, copied a FreeBSD disk image to it, and verified that fdisk > showed what I expected on the guest. I.e. : > > # zfs create -V 20G zroot/vmvol/gptdisk > # dd if=/vms/.img/FreeBSD-14.0-CURRENT-amd64.raw > of=/dev/zvol/zroot/vmvol/gptdisk bs=1m > # gpart recover zvol/zroot/vmvol/gptdisk > < add /dev/zvol/zroot/vmvol/gptdisk to test-vm configuration > > # vm start test-vm > # ssh root@test-vm lsb_release -a > Distributor ID: Ubuntu > Description: Ubuntu 21.04 > Release: 21.04 > Codename: hirsute > No LSB modules are available. > # ssh root@test-vm fdisk -l /dev/nvme1n1 > Disk /dev/nvme1n1: 20 GiB, 21474836480 bytes, 41943040 sectors > Disk model: bhyve-NVMe > Units: sectors of 1 * 512 = 512 bytes > Sector size (logical/physical): 512 bytes / 512 bytes > I/O size (minimum/optimal): 512 bytes / 512 bytes > Disklabel type: gpt > Disk identifier: 1B73327C-EAE2-11EB-90A0-002590EC5BF2 > > Device Start End Sectors Size Type > /dev/nvme1n1p1 3 129 127 63.5K FreeBSD boot > /dev/nvme1n1p2 130 66713 66584 32.5M EFI System > /dev/nvme1n1p3 66714 2163865 2097152 1G FreeBSD swap > /dev/nvme1n1p4 2163866 10552473 8388608 4G FreeBSD UFS > > The bhyve invocation is: > bhyve -c 2 -m 2G -Hw \ > -s 0,hostbridge \ > -s 4:0,virtio-blk,/dev/zvol/zroot/vms/test-vm/disk0 \ > -s 5:0,nvme,/dev/zvol/zroot/vmvol/disk0 \ > -s 6:0,nvme,/dev/zvol/zroot/vmvol/gptdisk \ > -s 7:0,virtio-net,tap0,mac=58:9c:fc:0b:ed:d6 > -s 31,lpc \ > -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \ > -l com1,/dev/nmdm-test-vm.1A \ > test-vm > > Note that my experiment is different from what you are doing. But I > would expect a Zvol and raw block device to behave the same as a > backing-store for an emulated NVMe drive in bhyve. My experiment ran > on -current, but the behavior in this area should be identical to the > 13.0-p8 version you are using. > > Does the output on the host of > # hd -n 256 /dev/nvd0 > match the output on the guest of > # hd -n 256 /dev/nvme0n1 > ? > -- Mario. --00000000000052d23605daab0835 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>on FreeBSD :</div><div><br></div><div><span style=3D"= font-family:monospace"><span style=3D"color:rgb(0,0,0);background-color:rgb= (255,255,255)">root@marietto:/usr/home/marietto/bhyve # hd -n 256 /dev/nvd0 </span><br></span></div><div><span style=3D"font-family:monospace"><br></sp= an></div><div><span style=3D"font-family:monospace">00000000 =C2=A033 c0 8e= d0 bc 00 7c 8e =C2=A0c0 8e d8 be 00 7c bf 00 =C2=A0|3.....|......|..| <br>00000010 =C2=A006 b9 00 02 fc f3 a4 50 =C2=A068 1c 06 cb fb b9 04 00 = =C2=A0|.......Ph.......| <br>00000020 =C2=A0bd be 07 80 7e 00 00 7c =C2=A00b 0f 85 0e 01 83 c5 10 = =C2=A0|....~..|........| <br>00000030 =C2=A0e2 f1 cd 18 88 56 00 55 =C2=A0c6 46 11 05 c6 46 10 00 = =C2=A0|.....V.U.F...F..| <br>00000040 =C2=A0b4 41 bb aa 55 cd 13 5d =C2=A072 0f 81 fb 55 aa 75 09 = =C2=A0|.A..U..]r...U.u.| <br>00000050 =C2=A0f7 c1 01 00 74 03 fe 46 =C2=A010 66 60 80 7e 10 00 74 = =C2=A0|....t..F.f`.~..t| <br>00000060 =C2=A026 66 68 00 00 00 00 66 =C2=A0ff 76 08 68 00 00 68 00 = =C2=A0|&fh....f.v.h..h.| <br>00000070 =C2=A07c 68 01 00 68 10 00 b4 =C2=A042 8a 56 00 8b f4 cd 13 = =C2=A0||h..h...B.V.....| <br>00000080 =C2=A09f 83 c4 10 9e eb 14 b8 =C2=A001 02 bb 00 7c 8a 56 00 = =C2=A0|............|.V.| <br>00000090 =C2=A08a 76 01 8a 4e 02 8a 6e =C2=A003 cd 13 66 61 73 1c fe = =C2=A0|.v..N..n...fas..| <br>000000a0 =C2=A04e 11 75 0c 80 7e 00 80 =C2=A00f 84 8a 00 b2 80 eb 84 = =C2=A0|N.u..~..........| <br>000000b0 =C2=A055 32 e4 8a 56 00 cd 13 =C2=A05d eb 9e 81 3e fe 7d 55 = =C2=A0|U2..V...]...>.}U| <br>000000c0 =C2=A0aa 75 6e ff 76 00 e8 8d =C2=A000 75 17 fa b0 d1 e6 64 = =C2=A0|.un.v....u.....d| <br>000000d0 =C2=A0e8 83 00 b0 df e6 60 e8 =C2=A07c 00 b0 ff e6 64 e8 75 = =C2=A0|......`.|....d.u| <br>000000e0 =C2=A000 fb b8 00 bb cd 1a 66 =C2=A023 c0 75 3b 66 81 fb 54 = =C2=A0|.......f#.u;f..T| <br>000000f0 =C2=A043 50 41 75 32 81 f9 02 =C2=A001 72 2c 66 68 07 bb 00 = =C2=A0|CPAu2....r,fh...| <br>00000100<br> </span></div><div><span style=3D"font-family:monospace"><br></span></div><d= iv><span style=3D"font-family:monospace">On Ubuntu 21.10 :</span></div><div= ><span style=3D"font-family:monospace"><br></span></div><div><span style=3D= "font-family:monospace">root@marietto-BHYVE:/home/marietto# hd -n 256 /dev/= nvme0n1<br></span></div><div><span style=3D"font-family:monospace"><br></sp= an></div><div><span style=3D"font-family:monospace">00000000 =C2=A033 c0 8e= d0 bc 00 7c 8e =C2=A0c0 8e d8 be 00 7c bf 00 =C2=A0|3.....|......|..|<br>0= 0000010 =C2=A006 b9 00 02 fc f3 a4 50 =C2=A068 1c 06 cb fb b9 04 00 =C2=A0|= .......Ph.......|<br>00000020 =C2=A0bd be 07 80 7e 00 00 7c =C2=A00b 0f 85 = 0e 01 83 c5 10 =C2=A0|....~..|........|<br>00000030 =C2=A0e2 f1 cd 18 88 56= 00 55 =C2=A0c6 46 11 05 c6 46 10 00 =C2=A0|.....V.U.F...F..|<br>00000040 = =C2=A0b4 41 bb aa 55 cd 13 5d =C2=A072 0f 81 fb 55 aa 75 09 =C2=A0|.A..U..]= r...U.u.|<br>00000050 =C2=A0f7 c1 01 00 74 03 fe 46 =C2=A010 66 60 80 7e 10= 00 74 =C2=A0|....t..F.f`.~..t|<br>00000060 =C2=A026 66 68 00 00 00 00 66 = =C2=A0ff 76 08 68 00 00 68 00 =C2=A0|&fh....f.v.h..h.|<br>00000070 =C2= =A07c 68 01 00 68 10 00 b4 =C2=A042 8a 56 00 8b f4 cd 13 =C2=A0||h..h...B.V= .....|<br>00000080 =C2=A09f 83 c4 10 9e eb 14 b8 =C2=A001 02 bb 00 7c 8a 56= 00 =C2=A0|............|.V.|<br>00000090 =C2=A08a 76 01 8a 4e 02 8a 6e =C2= =A003 cd 13 66 61 73 1c fe =C2=A0|.v..N..n...fas..|<br>000000a0 =C2=A04e 11= 75 0c 80 7e 00 80 =C2=A00f 84 8a 00 b2 80 eb 84 =C2=A0|N.u..~..........|<b= r>000000b0 =C2=A055 32 e4 8a 56 00 cd 13 =C2=A05d eb 9e 81 3e fe 7d 55 =C2= =A0|U2..V...]...>.}U|<br>000000c0 =C2=A0aa 75 6e ff 76 00 e8 8d =C2=A000= 75 17 fa b0 d1 e6 64 =C2=A0|.un.v....u.....d|<br>000000d0 =C2=A0e8 83 00 b= 0 df e6 60 e8 =C2=A07c 00 b0 ff e6 64 e8 75 =C2=A0|......`.|....d.u|<br>000= 000e0 =C2=A000 fb b8 00 bb cd 1a 66 =C2=A023 c0 75 3b 66 81 fb 54 =C2=A0|..= .....f#.u;f..T|<br>000000f0 =C2=A043 50 41 75 32 81 f9 02 =C2=A001 72 2c 66= 68 07 bb 00 =C2=A0|CPAu2....r,fh...|<br>00000100</span></div><div><span st= yle=3D"font-family:monospace"><br></span></div></div><br><div class=3D"gmai= l_quote"><div dir=3D"ltr" class=3D"gmail_attr">Il giorno dom 20 mar 2022 al= le ore 19:35 Chuck Tuffli <<a href=3D"mailto:chuck@tuffli.net">chuck@tuf= fli.net</a>> ha scritto:<br></div><blockquote class=3D"gmail_quote" styl= e=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddin= g-left:1ex">On Sun, Mar 20, 2022 at 8:13 AM Mario Marietto <<a href=3D"m= ailto:marietto2008@gmail.com" target=3D"_blank">marietto2008@gmail.com</a>&= gt; wrote:<br> ><br> > ----> Help me understand what "not recognized" means. The= device nvme0n1 :<br> ><br> > I don't see the partitions that are stored inside the disk nvme. A= nd I'm not able to mount the NTFS partition that's mapped as nvd0p2= under FreeBSD.<br> <br> Thank you, I understand the question now. As an experiment, I created<br> a zvol, copied a FreeBSD disk image to it, and verified that fdisk<br> showed what I expected on the guest. I.e. :<br> <br> #=C2=A0 zfs create -V 20G zroot/vmvol/gptdisk<br> #=C2=A0 dd if=3D/vms/.img/FreeBSD-14.0-CURRENT-amd64.raw<br> of=3D/dev/zvol/zroot/vmvol/gptdisk bs=3D1m<br> #=C2=A0 gpart recover zvol/zroot/vmvol/gptdisk<br> < add /dev/zvol/zroot/vmvol/gptdisk to test-vm configuration ><br> # vm start test-vm<br> # ssh root@test-vm lsb_release -a<br> Distributor ID: Ubuntu<br> Description:=C2=A0 =C2=A0 Ubuntu 21.04<br> Release:=C2=A0 =C2=A0 =C2=A0 =C2=A0 21.04<br> Codename:=C2=A0 =C2=A0 =C2=A0 =C2=A0hirsute<br> No LSB modules are available.<br> #=C2=A0 ssh root@test-vm fdisk -l /dev/nvme1n1<br> Disk /dev/nvme1n1: 20 GiB, 21474836480 bytes, 41943040 sectors<br> Disk model: bhyve-NVMe<br> Units: sectors of 1 * 512 =3D 512 bytes<br> Sector size (logical/physical): 512 bytes / 512 bytes<br> I/O size (minimum/optimal): 512 bytes / 512 bytes<br> Disklabel type: gpt<br> Disk identifier: 1B73327C-EAE2-11EB-90A0-002590EC5BF2<br> <br> Device=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Start=C2=A0 =C2=A0 =C2=A0 En= d Sectors=C2=A0 Size Type<br> /dev/nvme1n1p1=C2=A0 =C2=A0 =C2=A0 =C2=A03=C2=A0 =C2=A0 =C2=A0 129=C2=A0 = =C2=A0 =C2=A0127 63.5K FreeBSD boot<br> /dev/nvme1n1p2=C2=A0 =C2=A0 =C2=A0130=C2=A0 =C2=A0 66713=C2=A0 =C2=A066584 = 32.5M EFI System<br> /dev/nvme1n1p3=C2=A0 =C2=A066714=C2=A0 2163865 2097152=C2=A0 =C2=A0 1G Free= BSD swap<br> /dev/nvme1n1p4 2163866 10552473 8388608=C2=A0 =C2=A0 4G FreeBSD UFS<br> <br> The bhyve invocation is:<br> bhyve -c 2 -m 2G -Hw \<br> =C2=A0 =C2=A0 -s 0,hostbridge \<br> =C2=A0 =C2=A0 -s 4:0,virtio-blk,/dev/zvol/zroot/vms/test-vm/disk0 \<br> =C2=A0 =C2=A0 -s 5:0,nvme,/dev/zvol/zroot/vmvol/disk0 \<br> =C2=A0 =C2=A0 -s 6:0,nvme,/dev/zvol/zroot/vmvol/gptdisk \<br> =C2=A0 =C2=A0 -s 7:0,virtio-net,tap0,mac=3D58:9c:fc:0b:ed:d6<br> =C2=A0 =C2=A0 -s 31,lpc \<br> =C2=A0 =C2=A0 -l bootrom,/usr/local/share/uefi-firmware/BHYVE_UEFI.fd \<br> =C2=A0 =C2=A0 -l com1,/dev/nmdm-test-vm.1A \<br> =C2=A0 =C2=A0 test-vm<br> <br> Note that my experiment is different from what you are doing. But I<br> would expect a Zvol and raw block device to behave the same as a<br> backing-store for an emulated NVMe drive in bhyve. My experiment ran<br> on -current, but the behavior in this area should be identical to the<br> 13.0-p8 version you are using.<br> <br> Does the output on the host of<br> =C2=A0# hd -n 256 /dev/nvd0<br> match the output on the guest of<br> =C2=A0# hd -n 256 /dev/nvme0n1<br> ?<br> </blockquote></div><br clear=3D"all"><br>-- <br><div dir=3D"ltr" class=3D"g= mail_signature">Mario.<br></div> --00000000000052d23605daab0835--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CA%2B1FSijNKxNsHqHCTPE3swh_5ePc-_jtNHFtmBTbUbauEDPung>