Date: Mon, 31 Oct 2022 02:02:22 -0400 From: Paul Procacci <pprocacci@gmail.com> To: John Doherty <bsdlists@jld3.net> Cc: FreeBSD virtualization <freebsd-virtualization@freebsd.org> Subject: Re: NFS in bhyve VM mounted via bridge interface Message-ID: <CAFbbPuiHZokb_7Q=TpXy7fFBNfJGtN=9Dt3T2%2Bbx5OZUQOqjbg@mail.gmail.com> In-Reply-To: <A4F5B9EF-AA2B-4F34-8F62-A12ECE4E9566@jld3.net> References: <A4F5B9EF-AA2B-4F34-8F62-A12ECE4E9566@jld3.net>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On Mon, Oct 31, 2022 at 12:00 AM John Doherty <bsdlists@jld3.net> wrote: > I have a machine running FreeBSD 12.3-RELEASE with a zpool that consists > of 12 mirrored pairs of 14 TB disks. I'll call this the "storage > server." On that machine, I can write to ZFS file systems at around 950 > MB/s and read from them at around 1450 MB/s. I'm happy with that. > > I have another machine running Alma linux 8.6 that mounts file systems > from the storage server via NFS over a 10 GbE network. On this machine, > I can write to and read from an NFS file system at around 450 MB/s. I > wish that this were better but it's OK. > > I created a bhyve VM on the storage server that also runs Alma linux > 8.6. It has a vNIC that is bridged with the 10 GbE physical NIC and a > tap interface: > > [root@ss3] # ifconfig vm-storage > vm-storage: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 > mtu 1500 > ether 82:d3:46:17:4e:ee > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 10 priority 128 path cost 2000000 > member: ixl0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> > ifmaxaddr 0 port 5 priority 128 path cost 2000 > groups: bridge vm-switch viid-ddece@ > nd6 options=1<PERFORMNUD> > > I mount file systems from the storage server on this VM via NFS. I can > write to those file systems at around 250 MB/s and read from them at > around 280 MB/s. This surprised me a little: I thought that this might > perform better than or at least as well as the physical 10 GbE network > but find that it performs significantly worse. > > All my read and write tests here are stupidly simple, using dd to read > from /dev/zero and write to a file or to read from a file and write to > /dev/null. > > Is anyone else either surprised or unsurprised by these results? > > I have not yet tried passing a physical interface on the storage server > through to the VM with PCI passthrough, but the machine does have > another 10 GbE interface I could use for this. This stuff is all about > 3,200 miles away from me so I need to get someone to plug a cable in for > me. I'll be interested to see how that works out, though. > > Any comments much appreciated. Thanks. > > > I was getting geared up to help you with this and then this happened: Host: # dd if=17-04-27.mp4 of=/dev/null bs=4096 216616+1 records in 216616+1 records out 887263074 bytes transferred in 76.830892 secs (11548259 bytes/sec) VM: dd if=17-04-27.mp4 of=/dev/null bs=4096 216616+1 records in 216616+1 records out 887263074 bytes transferred in 7.430017 secs (119416016 bytes/sec) I'm totally flabbergasted. These results are consistent and not at all what I expected to see. I even ran the tests on the VM first and the host second. Call me confused. Anyways, that's a problem for me to figure out. Back to your problem, I had something typed out concerning checking rxsum's and txsum's are turned off on the interfaces, or at least see if that makes a difference, trying to use a disk type of nvme, and trying ng_bridge w/ netgraph interfaces but now I'm concluding my house is made of glass -- Hah! -- so until I get my house in order I'm going to refrain from providing details. Sorry and thanks! ~Paul [-- Attachment #2 --] <div dir="ltr"><div><div><div><div><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 31, 2022 at 12:00 AM John Doherty <<a href="mailto:bsdlists@jld3.net">bsdlists@jld3.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I have a machine running FreeBSD 12.3-RELEASE with a zpool that consists <br> of 12 mirrored pairs of 14 TB disks. I'll call this the "storage <br> server." On that machine, I can write to ZFS file systems at around 950 <br> MB/s and read from them at around 1450 MB/s. I'm happy with that.<br> <br> I have another machine running Alma linux 8.6 that mounts file systems <br> from the storage server via NFS over a 10 GbE network. On this machine, <br> I can write to and read from an NFS file system at around 450 MB/s. I <br> wish that this were better but it's OK.<br> <br> I created a bhyve VM on the storage server that also runs Alma linux <br> 8.6. It has a vNIC that is bridged with the 10 GbE physical NIC and a <br> tap interface:<br> <br> [root@ss3] # ifconfig vm-storage<br> vm-storage: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 <br> mtu 1500<br> ether 82:d3:46:17:4e:ee<br> id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15<br> maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200<br> root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0<br> member: tap1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP><br> ifmaxaddr 0 port 10 priority 128 path cost 2000000<br> member: ixl0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP><br> ifmaxaddr 0 port 5 priority 128 path cost 2000<br> groups: bridge vm-switch viid-ddece@<br> nd6 options=1<PERFORMNUD><br> <br> I mount file systems from the storage server on this VM via NFS. I can <br> write to those file systems at around 250 MB/s and read from them at <br> around 280 MB/s. This surprised me a little: I thought that this might <br> perform better than or at least as well as the physical 10 GbE network <br> but find that it performs significantly worse.<br> <br> All my read and write tests here are stupidly simple, using dd to read <br> from /dev/zero and write to a file or to read from a file and write to <br> /dev/null.<br> <br> Is anyone else either surprised or unsurprised by these results?<br> <br> I have not yet tried passing a physical interface on the storage server <br> through to the VM with PCI passthrough, but the machine does have <br> another 10 GbE interface I could use for this. This stuff is all about <br> 3,200 miles away from me so I need to get someone to plug a cable in for <br> me. I'll be interested to see how that works out, though.<br> <br> Any comments much appreciated. Thanks.<br> <br> <br> </blockquote></div><br></div>I was getting geared up to help you with this and then this happened:<br><br></div><div>Host:<br># dd if=17-04-27.mp4 of=/dev/null bs=4096<br>216616+1 records in<br>216616+1 records out<br>887263074 bytes transferred in 76.830892 secs (11548259 bytes/sec)<br><br></div><div>VM:<br>dd if=17-04-27.mp4 of=/dev/null bs=4096</div><div>216616+1 records in<br>216616+1 records out<br></div><div>887263074 bytes transferred in 7.430017 secs (119416016 bytes/sec)<br><br></div><div>I'm totally flabbergasted. These results are consistent and not at all what I expected to see.<br></div><div>I even ran the tests on the VM first and the host second. Call me confused.<br><br></div><div>Anyways, that's a problem for me to figure out.<br></div><div><br>Back to your problem, I had something typed out concerning checking rxsum's and txsum's are turned off on<br>the interfaces, or at least see if that makes a difference, trying to use a disk type of nvme, and trying ng_bridge<br>w/ netgraph interfaces but now I'm concluding my house is made of glass -- Hah! -- so until I get my house in<br>order I'm going to refrain from providing details.<br><br></div><div>Sorry and thanks!<br></div><div>~Paul<br></div></div></div></div>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAFbbPuiHZokb_7Q=TpXy7fFBNfJGtN=9Dt3T2%2Bbx5OZUQOqjbg>