Date: Sat, 07 Mar 2015 11:22:24 -0500 From: Allan Jude <allanjude@freebsd.org> To: freebsd-virtualization@freebsd.org Subject: Re: bhyve virtio-net MTU Message-ID: <54FB25C0.7060803@freebsd.org> In-Reply-To: <1425732590516.79490@tum.de> References: <1425732590516.79490@tum.de>
next in thread | previous in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] On 2015-03-07 07:49, Noah Bergbauer wrote: > Hi, > > I'm running FreeBSD 10.1 on a dedicated server, with a Linux VM in bhyve. The handbook tells you to bridge the tap interface to a real network interface, but that's not an option for me because I only have one IPv4 address. So instead, I assigned an internal IP address to bridge0 and used pf(4) to set up NAT routing. > All of this works without any issues, but I would like to increase the MTU of 1500. It's a virtual interface after all, why should it be so low? FreeBSD's loopback interface's MTU is 16384 and on Linux, it's even 65536. > > So I used ifconfig(8) to increase the MTU of tap0 and just like the manpage says, bridge0 had the same MTU after I added tap0. On the Linux side, I did the same with eth0 and then I sent 2000 byte pings to the host machine. > > It seems to work, but let's use tcpdump(8) to make sure: > > 02:02:46.244678 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1686, seq 3, length 2008 > 02:02:46.244983 IP 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1686, seq 3, length 1480 > 02:02:46.245061 IP 10.42.42.1 > 10.42.42.100: ip-proto-1 > 02:02:47.244953 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1686, seq 4, length 2008 > 02:02:47.245347 IP 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1686, seq 4, length 1480 > 02:02:47.245422 IP 10.42.42.1 > 10.42.42.100: ip-proto-1 > > The entire request goes through the virtio NIC, tap0, bridge0 and finally to the host's kernel without any issues. Which then sends a _fragmented_ reply because apparently it still thinks the MTU is 1500. > > A quick check with route(8) confirms this: > > # route show 10.42.42.100 > route to: 10.42.42.100 > destination: 10.42.42.0 > mask: 255.255.255.0 > fib: 0 > interface: bridge0 > flags: <UP,DONE,STATIC,PINNED> > recvpipe sendpipe ssthresh rtt,msec mtu weight expire > 0 0 0 0 1500 1 0 > > So I manually forced a bigger MTU: > > # route change -net 10.42.42.0 -mtu 15000 > change net 10.42.42.0 > > But now the reply packets get truncated instead of fragmented: > > 02:07:36.921165 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 3, length 2008 > 02:07:36.921624 IP truncated-ip - 6 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 3, length 2008 > 02:07:37.921042 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 4, length 2008 > 02:07:37.921499 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 4, length 2008 > 02:07:38.921522 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 5, length 2008 > 02:07:38.922253 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 5, length 2008 > 02:07:39.921432 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 6, length 2008 > 02:07:39.922165 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 6, length 2008 > 02:07:40.921513 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 7, length 2008 > 02:07:40.922245 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 7, length 2008 > 02:07:41.921393 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 8, length 2008 > 02:07:41.922160 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 8, length 2008 > 02:07:42.921504 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 9, length 2008 > 02:07:42.922348 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 9, length 2008 > 02:07:43.923031 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 10, length 2008 > 02:07:43.924904 IP truncated-ip - 6 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 10, length 2008 > 02:07:44.926832 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 11, length 2008 > 02:07:44.928511 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 11, length 2008 > 02:07:45.936968 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 12, length 2008 > 02:07:45.937722 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 12, length 2008 > 02:07:46.937453 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 13, length 2008 > 02:07:46.938161 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 13, length 2008 > ^C > 22 packets captured > 24 packets received by filter > 0 packets dropped by kernel > > This last dump was on the Linux side. Dumping at tap0 shows that the reply packets are still okay when they reach bhyve. Apparently they get truncated by bhyve's virtio-net as increasing the MTU works just fine with VirtualBox's virtio-net (at least on my Linux machine). > > Any ideas on how I can fix this? I had a quick look at the code and while a comment indicates that Ethernet-sized packets are assumed (https://svnweb.freebsd.org/base/release/10.1.0/usr.sbin/bhyve/pci_virtio_net.c?revision=274417&view=markup#l257), I was unable to find code that confirms this. > > Noah > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freebsd.org" > There is a 'do not fragment' flag you can sent with ping to help debug this. -- Allan Jude [-- Attachment #2 --] -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJU+yXDAAoJEJrBFpNRJZKf/h0QAKaPsajrY+rRPV48CXU+CSqP ktIxTACtXFKfYCxqEHRyspLCBC3qVOJ05G5VkW1cl0wOuGD1p28FtEOtb9xjypeS 8/boChdQuU9H1pkjWdVtkc71CbMKJ0KNvbuU5C3fw8sn73u6TS7D9wQ8Nd0dI2Yp XWRNWRa9P+uQaJ0hy2lyToPGgvbjIQYvZnfFOdn3M2BMsz11DZQ2758U1+zvazJ4 rkKCBNInCUZzb+tWBaXOeAi8qaBPRAaqV9FUvuo/pf14QhmzAtJPxXfe3hTZInoX +f77M4dY5Q1dXtONcDwXpauinTZhso77vN6FwOweG/cV6AU39uEZVELRUehLBTgT 1rKU5robneZh8vE37jWF/mE2faTAv2VqmTt7Zcvvg7fbhjKgJjjVKYAWVPvGOREZ WiiWP7mHGoqlsKY3G0cZi0yU7ZUY5GpcqCrerxkxLC4+Z3Y6zvG21+B/28fdTHYt gjptkA337TfxwzqGVAKeABxcYLevfWEM0omUTV/ZnrAbDxJu9Te2iGHn4NRhIAgP WoEgOJj2tII9P4f57ZbVRjSe0nPiS2NV7UBFOzLQZjGcfXQ/9Kwd7KAYh354Xj0h yLxQSUU45l8Uhtyers9/a0rBxLDz97tOEm94vTT2Nn9Vf0raWYXh7IkHq6Wrl2jI L+Sh8BEF8rUfo4PocJTW =l9n5 -----END PGP SIGNATURE-----
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54FB25C0.7060803>