Date: Mon, 31 Oct 2022 02:02:22 -0400 From: Paul Procacci <pprocacci@gmail.com> To: John Doherty <bsdlists@jld3.net> Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: NFS in bhyve VM mounted via bridge interface Message-ID: <CAFbbPuiHZokb_7Q=TpXy7fFBNfJGtN=9Dt3T2%2Bbx5OZUQOqjbg@mail.gmail.com> In-Reply-To: <A4F5B9EF-AA2B-4F34-8F62-A12ECE4E9566@jld3.net> References: <A4F5B9EF-AA2B-4F34-8F62-A12ECE4E9566@jld3.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--00000000000086179b05ec4e5ad2 Content-Type: text/plain; charset="UTF-8" On Mon, Oct 31, 2022 at 12:00 AM John Doherty <bsdlists@jld3.net> wrote: > I have a machine running FreeBSD 12.3-RELEASE with a zpool that consists > of 12 mirrored pairs of 14 TB disks. I'll call this the "storage > server." On that machine, I can write to ZFS file systems at around 950 > MB/s and read from them at around 1450 MB/s. I'm happy with that. > > I have another machine running Alma linux 8.6 that mounts file systems > from the storage server via NFS over a 10 GbE network. On this machine, > I can write to and read from an NFS file system at around 450 MB/s. I > wish that this were better but it's OK. > > I created a bhyve VM on the storage server that also runs Alma linux > 8.6. It has a vNIC that is bridged with the 10 GbE physical NIC and a > tap interface: > > [root@ss3] # ifconfig vm-storage > vm-storage: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 > mtu 1500 > ether 82:d3:46:17:4e:ee > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 10 priority 128 path cost 2000000 > member: ixl0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 5 priority 128 path cost 2000 > groups: bridge vm-switch viid-ddece@ > nd6 options=1<PERFORMNUD> > > I mount file systems from the storage server on this VM via NFS. I can > write to those file systems at around 250 MB/s and read from them at > around 280 MB/s. This surprised me a little: I thought that this might > perform better than or at least as well as the physical 10 GbE network > but find that it performs significantly worse. > > All my read and write tests here are stupidly simple, using dd to read > from /dev/zero and write to a file or to read from a file and write to > /dev/null. > > Is anyone else either surprised or unsurprised by these results? > > I have not yet tried passing a physical interface on the storage server > through to the VM with PCI passthrough, but the machine does have > another 10 GbE interface I could use for this. This stuff is all about > 3,200 miles away from me so I need to get someone to plug a cable in for > me. I'll be interested to see how that works out, though. > > Any comments much appreciated. Thanks. > > > I was getting geared up to help you with this and then this happened: Host: # dd if=17-04-27.mp4 of=/dev/null bs=4096 216616+1 records in 216616+1 records out 887263074 bytes transferred in 76.830892 secs (11548259 bytes/sec) VM: dd if=17-04-27.mp4 of=/dev/null bs=4096 216616+1 records in 216616+1 records out 887263074 bytes transferred in 7.430017 secs (119416016 bytes/sec) I'm totally flabbergasted. These results are consistent and not at all what I expected to see. I even ran the tests on the VM first and the host second. Call me confused. Anyways, that's a problem for me to figure out. Back to your problem, I had something typed out concerning checking rxsum's and txsum's are turned off on the interfaces, or at least see if that makes a difference, trying to use a disk type of nvme, and trying ng_bridge w/ netgraph interfaces but now I'm concluding my house is made of glass -- Hah! -- so until I get my house in order I'm going to refrain from providing details. Sorry and thanks! ~Paul --00000000000086179b05ec4e5ad2 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div><div><div><div><div dir=3D"ltr"><br></div><br><div cl= ass=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Oct 31, 2= 022 at 12:00 AM John Doherty <<a href=3D"mailto:bsdlists@jld3.net">bsdli= sts@jld3.net</a>> wrote:<br></div><blockquote class=3D"gmail_quote" styl= e=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);paddin= g-left:1ex">I have a machine running FreeBSD 12.3-RELEASE with a zpool that= consists <br> of 12 mirrored pairs of 14 TB disks.=C2=A0 I'll call this the "sto= rage <br> server." On that machine, I can write to ZFS file systems at around 95= 0 <br> MB/s and read from them at around 1450 MB/s. I'm happy with that.<br> <br> I have another machine running Alma linux 8.6 that mounts file systems <br> from the storage server via NFS over a 10 GbE network. On this machine, <br= > I can write to and read from an NFS file system at around 450 MB/s. I <br> wish that this were better but it's OK.<br> <br> I created a bhyve VM on the storage server that also runs Alma linux <br> 8.6. It has a vNIC that is bridged with the 10 GbE physical NIC and a <br> tap interface:<br> <br> [root@ss3] # ifconfig vm-storage<br> vm-storage: flags=3D8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metr= ic 0 <br> mtu 1500<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 ether 82:d3:46:17:4e:ee<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 id 00:00:00:00:00:00 priority 32768 hellotime 2= fwddelay 15<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 maxage 20 holdcnt 6 proto rstp maxaddr 2000 tim= eout 1200<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 root id 00:00:00:00:00:00 priority 32768 ifcost= 0 port 0<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 member: tap1 flags=3D143<LEARNING,DISCOVER,A= UTOEDGE,AUTOPTP><br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ifmaxaddr 0 port 10= priority 128 path cost 2000000<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 member: ixl0 flags=3D143<LEARNING,DISCOVER,A= UTOEDGE,AUTOPTP><br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ifmaxaddr 0 port 5 = priority 128 path cost 2000<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 groups: bridge vm-switch viid-ddece@<br> =C2=A0 =C2=A0 =C2=A0 =C2=A0 nd6 options=3D1<PERFORMNUD><br> <br> I mount file systems from the storage server on this VM via NFS. I can <br> write to those file systems at around 250 MB/s and read from them at <br> around 280 MB/s. This surprised me a little: I thought that this might <br> perform better than or at least as well as the physical 10 GbE network <br> but find that it performs significantly worse.<br> <br> All my read and write tests here are stupidly simple, using dd to read <br> from /dev/zero and write to a file or to read from a file and write to <br> /dev/null.<br> <br> Is anyone else either surprised or unsurprised by these results?<br> <br> I have not yet tried passing a physical interface on the storage server <br= > through to the VM with PCI passthrough, but the machine does have <br> another 10 GbE interface I could use for this. This stuff is all about <br> 3,200 miles away from me so I need to get someone to plug a cable in for <b= r> me. I'll be interested to see how that works out, though.<br> <br> Any comments much appreciated. Thanks.<br> <br> <br> </blockquote></div><br></div>I was getting geared up to help you with this = and then this happened:<br><br></div><div>Host:<br># dd if=3D17-04-27.mp4 o= f=3D/dev/null bs=3D4096<br>216616+1 records in<br>216616+1 records out<br>8= 87263074 bytes transferred in 76.830892 secs (11548259 bytes/sec)<br><br></= div><div>VM:<br>dd if=3D17-04-27.mp4 of=3D/dev/null bs=3D4096</div><div>216= 616+1 records in<br>216616+1 records out<br></div><div>887263074 bytes tran= sferred in 7.430017 secs (119416016 bytes/sec)<br><br></div><div>I'm to= tally flabbergasted.=C2=A0 These results are consistent and not at all what= I expected to see.<br></div><div>I even ran the tests on the VM first and = the host second.=C2=A0 Call me confused.<br><br></div><div>Anyways, that= 9;s a problem for me to figure out.<br></div><div><br>Back to your problem,= I had something typed out concerning checking rxsum's and txsum's = are turned off on<br>the interfaces, or at least see if that makes a differ= ence, trying to use a disk type of nvme, and trying ng_bridge<br>w/ netgrap= h interfaces but now I'm concluding my house is made of glass -- Hah! -= - so until I get my house in<br>order I'm going to refrain from providi= ng details.<br><br></div><div>Sorry and thanks!<br></div><div>~Paul<br></di= v></div></div></div> --00000000000086179b05ec4e5ad2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFbbPuiHZokb_7Q=TpXy7fFBNfJGtN=9Dt3T2%2Bbx5OZUQOqjbg>