From owner-freebsd-virtualization@freebsd.org Tue Dec 6 18:46:34 2016 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AF3C7C6A3BE for ; Tue, 6 Dec 2016 18:46:34 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 67D58F53 for ; Tue, 6 Dec 2016 18:46:34 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 73D0320AE98B for ; Wed, 7 Dec 2016 04:46:14 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 64B69280996 for ; Wed, 7 Dec 2016 04:46:14 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id tPqMjQTkpDWC for ; Wed, 7 Dec 2016 04:46:14 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (96-82-80-65-static.hfc.comcastbusiness.net [96.82.80.65]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 52EA928095F; Wed, 7 Dec 2016 04:46:11 +1000 (AEST) Subject: Re: bhyve: cannot send jumbo frames from linux or freebsd guest To: =?UTF-8?B?Sm9zw6kgR2FyY8OtYSBKdWFuaW5v?= References: From: Peter Grehan Cc: freebsd-virtualization@freebsd.org Message-ID: <414db7f0-ea86-2080-4179-eb77a18a67a5@freebsd.org> Date: Tue, 6 Dec 2016 10:46:25 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=YJDv8VOx c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=mwgbnDbW7alINpy3vhoKyg==:17 a=N659UExz7-8A:10 a=n5n_aSjo0skA:10 a=syzSIFLCVYJZK2B39fUA:9 a=ncJBWkJFt_E_2-gn:21 a=uSEb3iXFj9whLTb5:21 a=pILNOxqGKmIA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Dec 2016 18:46:34 -0000 Hi, > * From FreeBSD guest: > # tcpdump -vvn -i vtnet1 & > tcpdump: listening on vtnet1, link-type EN10MB (Ethernet), capture > size 262144 bytes > # ping -c 1 -D -s 1995 192.168.253.1 > PING 192.168.253.1 (192.168.253.1): 1995 data bytes > 23:08:46.808187 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto > ICMP (1), length 2023) > 192.168.253.4 > 192.168.253.1: ICMP echo request, id 58116, seq 0, > length 2003 > 23:08:46.808430 IP truncated-ip - 1 bytes missing! (tos 0x0, ttl 64, > id 0, offset 0, flags [DF], proto ICMP (1), length 2023) > 192.168.253.1 > 192.168.253.4: ICMP echo reply, id 58116, seq 0, length 2003 This looks like a bug in the FreeBSD virtio-net driver bpf code, since the host dump shows those sized packets going out correctly. > Could anyone explain why I cannot send jumbo frames from Linux and FreeBSD > guests, but I *can* send them from Windows guest? Am I missing o > missunderstanding something? Thanks in advanced, any comment o suggestion will > be wellcome. There is currently a bug in bhyve with handling of received frames. The virtio implementation advertises 'rx-merge' capability, which allows frames to span buffers that have been queued by the guest. However, bhyve currently doesn't copy the buffer across multiple buffers. I suspect that the Windows virtio guest driver doesn't use rx-merge but instead uses 'indirect descriptors', which is why it works. A short-term workaround for FreeBSD and Linux guests is to modify bhyve to not advertise the rx-merge capability: usr.sbin/bhyve/pci_virtio_net.c #define VTNET_S_HOSTCAPS \ - ( VIRTIO_NET_F_MAC | VIRTIO_NET_F_MRG_RXBUF | VIRTIO_NET_F_STATUS | \ + ( VIRTIO_NET_F_MAC | VIRTIO_NET_F_STATUS | \ (Note that this doesn't work with Net/OpenBSD, where jumbo frames are only supported via rx-merge.) I hope to have a fix for the rx-merge issue in the near future - it's a more efficient way of handling jumbo frames than indirect descriptors. later, Peter.