Date: Sat, 7 Mar 2015 12:49:50 +0000 From: Noah Bergbauer <noah.bergbauer@tum.de> To: "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org> Subject: bhyve virtio-net MTU Message-ID: <1425732590516.79490@tum.de>
next in thread | raw e-mail | index | archive | help
Hi, I'm running FreeBSD 10.1 on a dedicated server, with a Linux VM in bhyve. The handbook tells you to bridge the tap interface to a real network interface, but that's not an option for me because I only have one IPv4 address. So instead, I assigned an internal IP address to bridge0 and used pf(4) to set up NAT routing. All of this works without any issues, but I would like to increase the MTU of 1500. It's a virtual interface after all, why should it be so low? FreeBSD's loopback interface's MTU is 16384 and on Linux, it's even 65536. So I used ifconfig(8) to increase the MTU of tap0 and just like the manpage says, bridge0 had the same MTU after I added tap0. On the Linux side, I did the same with eth0 and then I sent 2000 byte pings to the host machine. It seems to work, but let's use tcpdump(8) to make sure: 02:02:46.244678 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1686, seq 3, length 2008 02:02:46.244983 IP 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1686, seq 3, length 1480 02:02:46.245061 IP 10.42.42.1 > 10.42.42.100: ip-proto-1 02:02:47.244953 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1686, seq 4, length 2008 02:02:47.245347 IP 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1686, seq 4, length 1480 02:02:47.245422 IP 10.42.42.1 > 10.42.42.100: ip-proto-1 The entire request goes through the virtio NIC, tap0, bridge0 and finally to the host's kernel without any issues. Which then sends a _fragmented_ reply because apparently it still thinks the MTU is 1500. A quick check with route(8) confirms this: # route show 10.42.42.100 route to: 10.42.42.100 destination: 10.42.42.0 mask: 255.255.255.0 fib: 0 interface: bridge0 flags: <UP,DONE,STATIC,PINNED> recvpipe sendpipe ssthresh rtt,msec mtu weight expire 0 0 0 0 1500 1 0 So I manually forced a bigger MTU: # route change -net 10.42.42.0 -mtu 15000 change net 10.42.42.0 But now the reply packets get truncated instead of fragmented: 02:07:36.921165 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 3, length 2008 02:07:36.921624 IP truncated-ip - 6 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 3, length 2008 02:07:37.921042 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 4, length 2008 02:07:37.921499 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 4, length 2008 02:07:38.921522 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 5, length 2008 02:07:38.922253 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 5, length 2008 02:07:39.921432 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 6, length 2008 02:07:39.922165 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 6, length 2008 02:07:40.921513 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 7, length 2008 02:07:40.922245 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 7, length 2008 02:07:41.921393 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 8, length 2008 02:07:41.922160 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 8, length 2008 02:07:42.921504 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 9, length 2008 02:07:42.922348 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 9, length 2008 02:07:43.923031 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 10, length 2008 02:07:43.924904 IP truncated-ip - 6 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 10, length 2008 02:07:44.926832 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 11, length 2008 02:07:44.928511 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 11, length 2008 02:07:45.936968 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 12, length 2008 02:07:45.937722 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 12, length 2008 02:07:46.937453 IP 10.42.42.100 > 10.42.42.1: ICMP echo request, id 1689, seq 13, length 2008 02:07:46.938161 IP truncated-ip - 518 bytes missing! 10.42.42.1 > 10.42.42.100: ICMP echo reply, id 1689, seq 13, length 2008 ^C 22 packets captured 24 packets received by filter 0 packets dropped by kernel This last dump was on the Linux side. Dumping at tap0 shows that the reply packets are still okay when they reach bhyve. Apparently they get truncated by bhyve's virtio-net as increasing the MTU works just fine with VirtualBox's virtio-net (at least on my Linux machine). Any ideas on how I can fix this? I had a quick look at the code and while a comment indicates that Ethernet-sized packets are assumed (https://svnweb.freebsd.org/base/release/10.1.0/usr.sbin/bhyve/pci_virtio_net.c?revision=274417&view=markup#l257), I was unable to find code that confirms this. Noah
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1425732590516.79490>