From owner-freebsd-virtualization@freebsd.org Sun Apr 1 21:00:06 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5BE2F7AFAE for ; Sun, 1 Apr 2018 21:00:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 76BE17CFA9 for ; Sun, 1 Apr 2018 21:00:06 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id B5C582CB50 for ; Sun, 1 Apr 2018 21:00:05 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w31L05qi006131 for ; Sun, 1 Apr 2018 21:00:05 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w31L05ui006130 for freebsd-virtualization@FreeBSD.org; Sun, 1 Apr 2018 21:00:05 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <201804012100.w31L05ui006130@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: freebsd-virtualization@FreeBSD.org Subject: Problem reports for freebsd-virtualization@FreeBSD.org that need special attention Date: Sun, 1 Apr 2018 21:00:05 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Apr 2018 21:00:07 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 212820 | FreeBSD 10-STABLE from latest HEAD and 11-RELEASE Open | 226583 | FreeBSD VM on Hyper-V/Azure can't properly detect 2 problems total for which you should take action. From owner-freebsd-virtualization@freebsd.org Sun Apr 1 23:39:50 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1B24F83921 for ; Sun, 1 Apr 2018 23:39:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E4C9833E1 for ; Sun, 1 Apr 2018 23:39:50 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 4C0332E251 for ; Sun, 1 Apr 2018 23:39:49 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w31Ndn6H076891 for ; Sun, 1 Apr 2018 23:39:49 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w31NdnRJ076890 for freebsd-virtualization@FreeBSD.org; Sun, 1 Apr 2018 23:39:49 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 176694] [kvm] KVM Guest Crash at Boot - kernel trap 12 with interrupts disabled [regression] Date: Sun, 01 Apr 2018 23:39:48 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: uefi X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: wheelcomplex@gmail.com X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 01 Apr 2018 23:39:50 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D176694 --- Comment #5 from David NewHamlet --- Fixed in: FreeBSD-12.0-CURRENT-amd64-20180329-r331740-disc1.iso --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Mon Apr 2 00:04:01 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 30CEAF851BC for ; Mon, 2 Apr 2018 00:04:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BEB9884888 for ; Mon, 2 Apr 2018 00:04:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 0D93C2E671 for ; Mon, 2 Apr 2018 00:04:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w3203xTC076500 for ; Mon, 2 Apr 2018 00:03:59 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w3203xQe076495 for freebsd-virtualization@FreeBSD.org; Mon, 2 Apr 2018 00:03:59 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 176694] [kvm] KVM Guest Crash at Boot - kernel trap 12 with interrupts disabled [regression] Date: Mon, 02 Apr 2018 00:04:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: uefi X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: kevans@freebsd.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: kevans@freebsd.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: assigned_to cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 00:04:01 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D176694 Kyle Evans changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|freebsd-virtualization@Free |kevans@freebsd.org |BSD.org | CC| |kevans@freebsd.org --- Comment #6 from Kyle Evans --- I guess I'll take this one, then. =3D) It'd be nice to know exactly what ch= ange we made to unbreak this, but I don't know that I care enough to ask you to bissecting it. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Mon Apr 2 03:24:02 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0EC69F6DD9F for ; Mon, 2 Apr 2018 03:24:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9641C6AB76 for ; Mon, 2 Apr 2018 03:24:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id E1370257 for ; Mon, 2 Apr 2018 03:24:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w323O0IF008482 for ; Mon, 2 Apr 2018 03:24:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w323O0dc008481 for freebsd-virtualization@FreeBSD.org; Mon, 2 Apr 2018 03:24:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 176694] [kvm] KVM Guest Crash at Boot - kernel trap 12 with interrupts disabled [regression] Date: Mon, 02 Apr 2018 03:24:00 +0000 X-Bugzilla-Reason: CC X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: uefi X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: rgrimes@FreeBSD.org X-Bugzilla-Status: Open X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: kevans@freebsd.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 03:24:02 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D176694 Rodney W. Grimes changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |freebsd-virtualization@Free | |BSD.org --=20 You are receiving this mail because: You are on the CC list for the bug.= From owner-freebsd-virtualization@freebsd.org Mon Apr 2 12:33:56 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E3432F5390E for ; Mon, 2 Apr 2018 12:33:55 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6A9E37C83D for ; Mon, 2 Apr 2018 12:33:55 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id w32CXiJi045698; Mon, 2 Apr 2018 14:33:45 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id BCF5F3A3; Mon, 2 Apr 2018 14:33:44 +0200 (CEST) Message-ID: <5AC22328.8060306@omnilan.de> Date: Mon, 02 Apr 2018 14:33:44 +0200 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Daniel Braniss CC: freebsd-virtualization@freebsd.org Subject: Re: bhyve and arp problem References: In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Mon, 02 Apr 2018 14:33:45 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 12:33:56 -0000 Bezüglich Daniel Braniss's Nachricht vom 30.03.2018 13:16 (localtime): > hi, > this is my first attempt at bhyve, and so far all seems ok, except > in my guest, the mac address of the hosting keeps flipping, ie, every 20 minutes > i see a message : > … arp: nnn (the hosting ip) moved from xxxx to yyyy > on both the host and guest I’m running a very resent -stable. > the yyyy is the mac of the host nic, while the xxxx is the tap0 > > i know this looks harmless, but it’s annoying You can calm it with 'sysctl net.link.ether.inet.log_arp_movements=0' There's also "net.link.ether.inet.log_arp_wrong_iface" and "net.inet.ip.check_interface" which influence related behaviour. You also posted (documentationized IP-addresses): > I think the problem starts with the host seeing the client/guest on 2 interfaces, the nic (mlnxen0) and the tap(tap0) > on the host: > > arp -a > ... > bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on tap0 expires in 1001 seconds [ethernet] > bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on mlxen0 expires in 644 seconds [ethernet] Initially, you reference two MAC-addresses with xxxx and yyyy. The recent post indicates non-different MAC-addresses. If xxxx and yyyy - resp. xx.xx.xx.xx.xx - are equal (but seen on different interfaces), this wouldn't get logged I think. But it was the only harmless case for straight forward setups. Even with STP/LACP/CARP/etc. in place, "arp: IP-address moved" always indicates a misconfiguration and I don't know any example where the two different MAC-Addresses for one IP-address were harmless. While using a single (locally administrated?) MAC address more than once sitewide _can_ make sense, having two interfaces on one host which both are on the same ethernet segment like the two interfaces with the same MAC address, looks like an unintended setup. So I strongly suggest to analyze your setup before altering the mentioned sysctl!!! -harry From owner-freebsd-virtualization@freebsd.org Mon Apr 2 13:08:02 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0E430F57B4A for ; Mon, 2 Apr 2018 13:08:02 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8F7317D8E1 for ; Mon, 2 Apr 2018 13:08:00 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from imac.bk.cs.huji.ac.il ([132.65.179.42]) by kabab.cs.huji.ac.il with esmtp id 1f2zBd-000914-UD; Mon, 02 Apr 2018 16:07:49 +0300 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: bhyve and arp problem From: Daniel Braniss In-Reply-To: <5AC22328.8060306@omnilan.de> Date: Mon, 2 Apr 2018 16:07:49 +0300 Cc: freebsd-virtualization@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <5AC22328.8060306@omnilan.de> To: Harry Schmalzbauer X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 13:08:02 -0000 > On 2 Apr 2018, at 15:33, Harry Schmalzbauer = wrote: >=20 > Bez=C3=BCglich Daniel Braniss's Nachricht vom 30.03.2018 13:16 = (localtime): >> hi, >> this is my first attempt at bhyve, and so far all seems ok, except >> in my guest, the mac address of the hosting keeps flipping, ie, every = 20 minutes >> i see a message : >> =E2=80=A6 arp: nnn (the hosting ip) moved from xxxx to yyyy >> on both the host and guest I=E2=80=99m running a very resent -stable. >> the yyyy is the mac of the host nic, while the xxxx is the tap0 >>=20 >> i know this looks harmless, but it=E2=80=99s annoying >=20 > You can calm it with > 'sysctl net.link.ether.inet.log_arp_movements=3D0' >=20 > There's also "net.link.ether.inet.log_arp_wrong_iface" and > "net.inet.ip.check_interface" which influence related behaviour. >=20 > You also posted (documentationized IP-addresses): >> I think the problem starts with the host seeing the client/guest on 2 = interfaces, the nic (mlnxen0) and the tap(tap0) >> on the host: >>=20 >> arp -a >> ... >> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on tap0 expires = in 1001 seconds [ethernet] >> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on mlxen0 = expires in 644 seconds [ethernet] >=20 the above 2 lines are on the host running bhyve (server?) and the MACs = belong to the client, and they are identical, there is no complaints. (BTW, did you change the ip=E2=80=99s?) > Initially, you reference two MAC-addresses with xxxx and yyyy. this is on the client, where the MAC are different (it=E2=80=99s of the = hosting computer). > The recent post indicates non-different MAC-addresses. >=20 > If xxxx and yyyy - resp. xx.xx.xx.xx.xx - are equal (but seen on > different interfaces), this wouldn't get logged I think. > But it was the only harmless case for straight forward setups. > Even with STP/LACP/CARP/etc. in place, "arp: IP-address moved" always > indicates a misconfiguration and I don't know any example where the = two > different MAC-Addresses for one IP-address were harmless. > While using a single (locally administrated?) MAC address more than = once > sitewide _can_ make sense, having two interfaces on one host which = both > are on the same ethernet segment like the two interfaces with the same > MAC address, looks like an unintended setup. >=20 > So I strongly suggest to analyze your setup before altering the > mentioned sysctl!!! >=20 I do want to know if there are ip/mac issues, it usually happens when = more than one host has the same ip, which is not the case here :-( thanks, danny > -harry From owner-freebsd-virtualization@freebsd.org Mon Apr 2 13:52:09 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59B94F6B60F for ; Mon, 2 Apr 2018 13:52:09 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BC8CC7FBB8 for ; Mon, 2 Apr 2018 13:52:08 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w32Dpt3n082638; Mon, 2 Apr 2018 06:51:55 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w32DptK9082637; Mon, 2 Apr 2018 06:51:55 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201804021351.w32DptK9082637@pdx.rh.CN85.dnsmgr.net> Subject: Re: bhyve and arp problem In-Reply-To: To: Daniel Braniss Date: Mon, 2 Apr 2018 06:51:55 -0700 (PDT) CC: Harry Schmalzbauer , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 13:52:09 -0000 > > On 2 Apr 2018, at 15:33, Harry Schmalzbauer wrote: > > > > Bez?glich Daniel Braniss's Nachricht vom 30.03.2018 13:16 (localtime): > >> hi, > >> this is my first attempt at bhyve, and so far all seems ok, except > >> in my guest, the mac address of the hosting keeps flipping, ie, every 20 minutes > >> i see a message : > >> ? arp: nnn (the hosting ip) moved from xxxx to yyyy > >> on both the host and guest I?m running a very resent -stable. > >> the yyyy is the mac of the host nic, while the xxxx is the tap0 > >> > >> i know this looks harmless, but it?s annoying > > > > You can calm it with > > 'sysctl net.link.ether.inet.log_arp_movements=0' > > > > There's also "net.link.ether.inet.log_arp_wrong_iface" and > > "net.inet.ip.check_interface" which influence related behaviour. > > > > You also posted (documentationized IP-addresses): > >> I think the problem starts with the host seeing the client/guest on 2 interfaces, the nic (mlnxen0) and the tap(tap0) > >> on the host: > >> > >> arp -a > >> ... > >> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on tap0 expires in 1001 seconds [ethernet] > >> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on mlxen0 expires in 644 seconds [ethernet] > > > > the above 2 lines are on the host running bhyve (server?) and the MACs belong to the client, and they are identical, > there is no complaints. > (BTW, did you change the ip?s?) > > > Initially, you reference two MAC-addresses with xxxx and yyyy. > this is on the client, where the MAC are different (it?s of the hosting computer). > > > The recent post indicates non-different MAC-addresses. > > > > If xxxx and yyyy - resp. xx.xx.xx.xx.xx - are equal (but seen on > > different interfaces), this wouldn't get logged I think. > > But it was the only harmless case for straight forward setups. > > Even with STP/LACP/CARP/etc. in place, "arp: IP-address moved" always > > indicates a misconfiguration and I don't know any example where the two > > different MAC-Addresses for one IP-address were harmless. > > While using a single (locally administrated?) MAC address more than once > > sitewide _can_ make sense, having two interfaces on one host which both > > are on the same ethernet segment like the two interfaces with the same > > MAC address, looks like an unintended setup. > > > > So I strongly suggest to analyze your setup before altering the > > mentioned sysctl!!! > > > I do want to know if there are ip/mac issues, it usually happens when more than one host has the same ip, > which is not the case here :-( Are you trying to use the HOSTS ip address in the GUEST? And how do you have an mlxen interface in a GUEST? Is this being done with PCI passthrough? -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Mon Apr 2 14:58:21 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2055DF6FC50 for ; Mon, 2 Apr 2018 14:58:21 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A13C081EFE for ; Mon, 2 Apr 2018 14:58:20 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from imac.bk.cs.huji.ac.il ([132.65.179.42]) by kabab.cs.huji.ac.il with esmtp id 1f30uT-000L8O-IV; Mon, 02 Apr 2018 17:58:13 +0300 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: bhyve and arp problem From: Daniel Braniss In-Reply-To: <201804021351.w32DptK9082637@pdx.rh.CN85.dnsmgr.net> Date: Mon, 2 Apr 2018 17:58:13 +0300 Cc: Harry Schmalzbauer , freebsd-virtualization@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <201804021351.w32DptK9082637@pdx.rh.CN85.dnsmgr.net> To: "Rodney W. Grimes" X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 14:58:21 -0000 > On 2 Apr 2018, at 16:51, Rodney W. Grimes = wrote: >=20 >>> On 2 Apr 2018, at 15:33, Harry Schmalzbauer = wrote: >>>=20 >>> Bez?glich Daniel Braniss's Nachricht vom 30.03.2018 13:16 = (localtime): >>>> hi, >>>> this is my first attempt at bhyve, and so far all seems ok, except >>>> in my guest, the mac address of the hosting keeps flipping, ie, = every 20 minutes >>>> i see a message : >>>> ? arp: nnn (the hosting ip) moved from xxxx to yyyy >>>> on both the host and guest I?m running a very resent -stable. >>>> the yyyy is the mac of the host nic, while the xxxx is the tap0 >>>>=20 >>>> i know this looks harmless, but it?s annoying >>>=20 >>> You can calm it with >>> 'sysctl net.link.ether.inet.log_arp_movements=3D0' >>>=20 >>> There's also "net.link.ether.inet.log_arp_wrong_iface" and >>> "net.inet.ip.check_interface" which influence related behaviour. >>>=20 >>> You also posted (documentationized IP-addresses): >>>> I think the problem starts with the host seeing the client/guest on = 2 interfaces, the nic (mlnxen0) and the tap(tap0) >>>> on the host: >>>>=20 >>>> arp -a >>>> ... >>>> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on tap0 = expires in 1001 seconds [ethernet] >>>> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on mlxen0 = expires in 644 seconds [ethernet] >>>=20 >>=20 >> the above 2 lines are on the host running bhyve (server?) and the = MACs belong to the client, and they are identical, >> there is no complaints. >> (BTW, did you change the ip?s?) >>=20 >>> Initially, you reference two MAC-addresses with xxxx and yyyy. >> this is on the client, where the MAC are different (it?s of the = hosting computer). >>=20 >>> The recent post indicates non-different MAC-addresses. >>>=20 >>> If xxxx and yyyy - resp. xx.xx.xx.xx.xx - are equal (but seen on >>> different interfaces), this wouldn't get logged I think. >>> But it was the only harmless case for straight forward setups. >>> Even with STP/LACP/CARP/etc. in place, "arp: IP-address moved" = always >>> indicates a misconfiguration and I don't know any example where the = two >>> different MAC-Addresses for one IP-address were harmless. >>> While using a single (locally administrated?) MAC address more than = once >>> sitewide _can_ make sense, having two interfaces on one host which = both >>> are on the same ethernet segment like the two interfaces with the = same >>> MAC address, looks like an unintended setup. >>>=20 >>> So I strongly suggest to analyze your setup before altering the >>> mentioned sysctl!!! >>>=20 >> I do want to know if there are ip/mac issues, it usually happens when = more than one host has the same ip, >> which is not the case here :-( >=20 > Are you trying to use the HOSTS ip address in the GUEST? the client is using the server=E2=80=99s /usr/local, which is mounted = via nfs. so I guess the answer is yes. > And how do you have an mlxen interface in a GUEST? no the guest has only vtnet0 and lo0 the ip of the client is obtained via dhcp on the server, there is a bridge, bridge0 and it bridges between the = taps and the mxlen0 > Is this being done with PCI passthrough? again, no. cheers, danny >=20 >=20 > --=20 > Rod Grimes = rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Mon Apr 2 15:39:27 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9FB56F7269C for ; Mon, 2 Apr 2018 15:39:27 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 149C883D85 for ; Mon, 2 Apr 2018 15:39:26 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id w32FdJRF047374; Mon, 2 Apr 2018 17:39:19 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id EEBFE3EA; Mon, 2 Apr 2018 17:39:18 +0200 (CEST) Message-ID: <5AC24EA6.9010207@omnilan.de> Date: Mon, 02 Apr 2018 17:39:18 +0200 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Daniel Braniss CC: "Rodney W. Grimes" , freebsd-virtualization@freebsd.org Subject: Re: bhyve and arp problem References: <201804021351.w32DptK9082637@pdx.rh.CN85.dnsmgr.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Mon, 02 Apr 2018 17:39:19 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 15:39:27 -0000 Bezüglich Daniel Braniss's Nachricht vom 02.04.2018 16:58 (localtime): … >> >> Are you trying to use the HOSTS ip address in the GUEST? > > the client is using the server’s /usr/local, which is mounted via nfs. I guess you have reasons to prefer nfs over nullfs. > so I guess the answer is yes. I don't see how /usr/local is related to IP address assigning... … > the guest has only vtnet0 and lo0 > the ip of the client is obtained via dhcp > on the server, there is a bridge, bridge0 and it bridges between the taps and the mxlen0 Hmm, do you assing an IP address to your if_tap(4) interfaces at your bhyve-host? if_tap(4) is used for interconnecting bhyve(8)'s virtio-net "backend" with the host's host-IP stack via the character device /dev/tapX. The if_tap(4) interfaces on the host mustn't get the guest's (clients) IP address! The address must be assigned to the if_vtnet(4) interface _inside_ the guest (I call it the front-end of bhyve's virtio-net) – done usually in /etc/rc.conf – no relation to /usr/local… But chances are good I misunderstood something, either your setup or your results... Else I can't explain what you see other than the above scenario. -harry From owner-freebsd-virtualization@freebsd.org Mon Apr 2 16:08:07 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 46E34F745C2 for ; Mon, 2 Apr 2018 16:08:07 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9875C84F54 for ; Mon, 2 Apr 2018 16:08:06 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w32G7uLN083224; Mon, 2 Apr 2018 09:07:56 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w32G7uFa083223; Mon, 2 Apr 2018 09:07:56 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201804021607.w32G7uFa083223@pdx.rh.CN85.dnsmgr.net> Subject: Re: bhyve and arp problem In-Reply-To: To: Daniel Braniss Date: Mon, 2 Apr 2018 09:07:56 -0700 (PDT) CC: Harry Schmalzbauer , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 16:08:07 -0000 > > On 2 Apr 2018, at 16:51, Rodney W. Grimes wrote: > > > >>> On 2 Apr 2018, at 15:33, Harry Schmalzbauer wrote: > >>> > >>> Bez?glich Daniel Braniss's Nachricht vom 30.03.2018 13:16 (localtime): > >>>> hi, > >>>> this is my first attempt at bhyve, and so far all seems ok, except > >>>> in my guest, the mac address of the hosting keeps flipping, ie, every 20 minutes > >>>> i see a message : > >>>> ? arp: nnn (the hosting ip) moved from xxxx to yyyy > >>>> on both the host and guest I?m running a very resent -stable. > >>>> the yyyy is the mac of the host nic, while the xxxx is the tap0 > >>>> > >>>> i know this looks harmless, but it?s annoying > >>> > >>> You can calm it with > >>> 'sysctl net.link.ether.inet.log_arp_movements=0' > >>> > >>> There's also "net.link.ether.inet.log_arp_wrong_iface" and > >>> "net.inet.ip.check_interface" which influence related behaviour. > >>> > >>> You also posted (documentationized IP-addresses): > >>>> I think the problem starts with the host seeing the client/guest on 2 interfaces, the nic (mlnxen0) and the tap(tap0) > >>>> on the host: > >>>> > >>>> arp -a > >>>> ... > >>>> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on tap0 expires in 1001 seconds [ethernet] > >>>> bhv-00.cs.huji.ac.il (192.0.2.246) at xx.xx.xx.xx.xx on mlxen0 expires in 644 seconds [ethernet] > >>> > >> > >> the above 2 lines are on the host running bhyve (server?) and the MACs belong to the client, and they are identical, > >> there is no complaints. > >> (BTW, did you change the ip?s?) > >> > >>> Initially, you reference two MAC-addresses with xxxx and yyyy. > >> this is on the client, where the MAC are different (it?s of the hosting computer). > >> > >>> The recent post indicates non-different MAC-addresses. > >>> > >>> If xxxx and yyyy - resp. xx.xx.xx.xx.xx - are equal (but seen on > >>> different interfaces), this wouldn't get logged I think. > >>> But it was the only harmless case for straight forward setups. > >>> Even with STP/LACP/CARP/etc. in place, "arp: IP-address moved" always > >>> indicates a misconfiguration and I don't know any example where the two > >>> different MAC-Addresses for one IP-address were harmless. > >>> While using a single (locally administrated?) MAC address more than once > >>> sitewide _can_ make sense, having two interfaces on one host which both > >>> are on the same ethernet segment like the two interfaces with the same > >>> MAC address, looks like an unintended setup. > >>> > >>> So I strongly suggest to analyze your setup before altering the > >>> mentioned sysctl!!! > >>> > >> I do want to know if there are ip/mac issues, it usually happens when more than one host has the same ip, > >> which is not the case here :-( > > > > Are you trying to use the HOSTS ip address in the GUEST? > > the client is using the server?s /usr/local, which is mounted via nfs. > so I guess the answer is yes. I didnt mean to access it, I meant did you assign the same IP to the GUEST that is assigned on the HOST. I suspect the answer here is no from other context. > > And how do you have an mlxen interface in a GUEST? > no > the guest has only vtnet0 and lo0 > the ip of the client is obtained via dhcp > on the server, there is a bridge, bridge0 and it bridges between the taps and the mxlen0 Is 192.0.2.246 on the mlxen0 interface, or on the bridge0? I believe you need to move the ip from the interface to the bridge to have this work right. > > > Is this being done with PCI passthrough? > again, no. > > cheers, > danny -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Mon Apr 2 16:11:21 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E5D82F74B3E for ; Mon, 2 Apr 2018 16:11:20 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4AB1D8534B for ; Mon, 2 Apr 2018 16:11:19 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id w32GB9YB083258; Mon, 2 Apr 2018 09:11:09 -0700 (PDT) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id w32GB9G4083257; Mon, 2 Apr 2018 09:11:09 -0700 (PDT) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201804021611.w32GB9G4083257@pdx.rh.CN85.dnsmgr.net> Subject: Re: bhyve and arp problem In-Reply-To: <5AC24EA6.9010207@omnilan.de> To: Harry Schmalzbauer Date: Mon, 2 Apr 2018 09:11:09 -0700 (PDT) CC: Daniel Braniss , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 16:11:21 -0000 > Bez?glich Daniel Braniss's Nachricht vom 02.04.2018 16:58 (localtime): > > ? > >> > >> Are you trying to use the HOSTS ip address in the GUEST? > > > > the client is using the server?s /usr/local, which is mounted via nfs. > > I guess you have reasons to prefer nfs over nullfs. You can not nullfs mount from inside a bhyve guest the host file system would be his reason. > > so I guess the answer is yes. > > I don't see how /usr/local is related to IP address assigning... Yes, this is not any part of this issue, I believe the RP misunderstood my question. > ? > > the guest has only vtnet0 and lo0 > > the ip of the client is obtained via dhcp > > on the server, there is a bridge, bridge0 and it bridges between the taps and the mxlen0 > > Hmm, do you assing an IP address to your if_tap(4) interfaces at your > bhyve-host? I missed that as a possible, yes, that would cause odd issues. > if_tap(4) is used for interconnecting bhyve(8)'s virtio-net "backend" > with the host's host-IP stack via the character device /dev/tapX. > The if_tap(4) interfaces on the host mustn't get the guest's (clients) > IP address! The address must be assigned to the if_vtnet(4) interface > _inside_ the guest (I call it the front-end of bhyve's virtio-net) ? > done usually in /etc/rc.conf ? no relation to /usr/local? > > But chances are good I misunderstood something, either your setup or > your results... Else I can't explain what you see other than the above > scenario. Is it still true that if you have a bridge hooked to an interface you must move the IP assignement from the interace device to the bridge? Ie in this case here the IP address of the most must be put on bridge0, and NOT on mlxen0. > -harry > -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Mon Apr 2 16:19:29 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFEA3F751D6 for ; Mon, 2 Apr 2018 16:19:29 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3443085BDF for ; Mon, 2 Apr 2018 16:19:29 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (mh0.gentlemail.de [78.138.80.135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id w32GJMJL047680; Mon, 2 Apr 2018 18:19:22 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id EBC933F7; Mon, 2 Apr 2018 18:19:21 +0200 (CEST) Message-ID: <5AC25809.30205@omnilan.de> Date: Mon, 02 Apr 2018 18:19:21 +0200 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: "Rodney W. Grimes" CC: Daniel Braniss , freebsd-virtualization@freebsd.org Subject: Re: bhyve and arp problem References: <201804021611.w32GB9G4083257@pdx.rh.CN85.dnsmgr.net> In-Reply-To: <201804021611.w32GB9G4083257@pdx.rh.CN85.dnsmgr.net> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Greylist: ACL 130 matched, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [78.138.80.130]); Mon, 02 Apr 2018 18:19:22 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: 78.138.80.135; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 16:19:29 -0000 Bezüglich Rodney W. Grimes's Nachricht vom 02.04.2018 18:11 (localtime): >> Bez?glich Daniel Braniss's Nachricht vom 02.04.2018 16:58 (localtime): >> >> ? >>>> >>>> Are you trying to use the HOSTS ip address in the GUEST? >>> >>> the client is using the server?s /usr/local, which is mounted via nfs. >> >> I guess you have reasons to prefer nfs over nullfs. > > You can not nullfs mount from inside a bhyve guest the host file > system would be his reason. Doh, of course – I mind-mixed with jails… Almost all my FreeBSD "guests" on a bhyve(8) host are jails... Please simply ignore ;-) … >> your results... Else I can't explain what you see other than the above >> scenario. > > Is it still true that if you have a bridge hooked to an interface > you must move the IP assignement from the interace device to the > bridge? I have no idea, I haven't used if_bridge(4) for some years. Perferably I use vale(4) (if I don't need .11q) or ng_bridge(4) for bhyve(8) setups. -harry From owner-freebsd-virtualization@freebsd.org Mon Apr 2 21:35:58 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E764F86C48 for ; Mon, 2 Apr 2018 21:35:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0A7AB736EA for ; Mon, 2 Apr 2018 21:35:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 5F43B11CFC for ; Mon, 2 Apr 2018 21:35:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w32LZvZN084789 for ; Mon, 2 Apr 2018 21:35:57 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w32LZvpP084788 for freebsd-virtualization@FreeBSD.org; Mon, 2 Apr 2018 21:35:57 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 215737] [bhyve] utilizing virtio-net truncates jumbo frames at 4084 bytes length Date: Mon, 02 Apr 2018 21:35:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: des.gaufres@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Apr 2018 21:35:58 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215737 P Kern changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |des.gaufres@gmail.com --- Comment #7 from P Kern --- Hi. We are encountering the same problem as Arjan van der Velde. We also want to pass NFS traffic through our 10G/40G switches. The difference is our VMs are in VMware ESXi with vmx(8) NICs (ie. VMXNET3). Packets will not traverse our FreeBSD gateway unless they are under 4084 by= tes. We can ping jumbo packets to either of the vmx(8) NICs, but jumbo packets w= ill not pass through the gateway. thanks for any attention/pointers. P Kern --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Tue Apr 3 00:16:42 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC8B3F6FEA1 for ; Tue, 3 Apr 2018 00:16:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4402F7A75B for ; Tue, 3 Apr 2018 00:16:42 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 7CEA4133B2 for ; Tue, 3 Apr 2018 00:16:41 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w330GfCW058145 for ; Tue, 3 Apr 2018 00:16:41 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w330GfoZ058141 for freebsd-virtualization@FreeBSD.org; Tue, 3 Apr 2018 00:16:41 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 215737] [bhyve] utilizing virtio-net truncates jumbo frames at 4084 bytes length Date: Tue, 03 Apr 2018 00:16:40 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: des.gaufres@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Apr 2018 00:16:42 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215737 --- Comment #8 from P Kern --- (In reply to P Kern from comment #7) Sigh. I should have rtfm: just noticed vmx(8) does _not_ mention supporting jumbo frames. Never mind. P Kern. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Apr 4 13:09:09 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B133EF8D4A7 for ; Wed, 4 Apr 2018 13:09:08 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2B63379DD2 for ; Wed, 4 Apr 2018 13:09:07 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from imac.bk.cs.huji.ac.il ([132.65.179.42]) by kabab.cs.huji.ac.il with esmtp id 1f3i9o-0000kb-FK; Wed, 04 Apr 2018 16:08:56 +0300 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: bhyve and arp problem From: Daniel Braniss In-Reply-To: <5AC25809.30205@omnilan.de> Date: Wed, 4 Apr 2018 16:08:56 +0300 Cc: "Rodney W. Grimes" , freebsd-virtualization@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <201804021611.w32GB9G4083257@pdx.rh.CN85.dnsmgr.net> <5AC25809.30205@omnilan.de> To: Harry Schmalzbauer X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Apr 2018 13:09:09 -0000 > On 2 Apr 2018, at 19:19, Harry Schmalzbauer = wrote: >=20 > Bez=C3=BCglich Rodney W. Grimes's Nachricht vom 02.04.2018 18:11 = (localtime): >>> Bez?glich Daniel Braniss's Nachricht vom 02.04.2018 16:58 = (localtime): >>>=20 >>> ? >>>>>=20 >>>>> Are you trying to use the HOSTS ip address in the GUEST? >>>>=20 >>>> the client is using the server?s /usr/local, which is mounted via = nfs. >>>=20 >>> I guess you have reasons to prefer nfs over nullfs. >>=20 >> You can not nullfs mount from inside a bhyve guest the host file >> system would be his reason. >=20 > Doh, of course =E2=80=93 I mind-mixed with jails=E2=80=A6 Almost all = my FreeBSD > "guests" on a bhyve(8) host are jails... Please simply ignore ;-) >=20 >=20 > =E2=80=A6 >>> your results... Else I can't explain what you see other than the = above >>> scenario. >>=20 >> Is it still true that if you have a bridge hooked to an interface >> you must move the IP assignement from the interace device to the >> bridge? >=20 > I have no idea, I haven't used if_bridge(4) for some years. = Perferably > I use vale(4) (if I don't need .11q) or ng_bridge(4) for bhyve(8) = setups. >=20 > -harry maybe this could be the cause: on the server, every time a client is started iI see: kernel: bridge0: error setting interface capabilities on mlxen0 and BTW, with the latest kernel, there is only one complaint about arp = moving! danny From owner-freebsd-virtualization@freebsd.org Wed Apr 4 15:25:27 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ACFB5F96F33 for ; Wed, 4 Apr 2018 15:25:27 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 34AE6813EE for ; Wed, 4 Apr 2018 15:25:26 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id w34FPIid079530; Wed, 4 Apr 2018 17:25:18 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 2A098804; Wed, 4 Apr 2018 17:25:18 +0200 (CEST) Message-ID: <5AC4EE5C.9030906@omnilan.de> Date: Wed, 04 Apr 2018 17:25:16 +0200 From: Harry Schmalzbauer Organization: OmniLAN User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; de-DE; rv:1.9.2.8) Gecko/20100906 Lightning/1.0b2 Thunderbird/3.1.2 MIME-Version: 1.0 To: Daniel Braniss CC: "Rodney W. Grimes" , freebsd-virtualization@freebsd.org Subject: Re: bhyve and arp problem References: <201804021611.w32GB9G4083257@pdx.rh.CN85.dnsmgr.net> <5AC25809.30205@omnilan.de> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Wed, 04 Apr 2018 17:25:18 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Apr 2018 15:25:27 -0000 Bezüglich Daniel Braniss's Nachricht vom 04.04.2018 15:08 (localtime): > > >> On 2 Apr 2018, at 19:19, Harry Schmalzbauer wrote: >> >> Bezüglich Rodney W. Grimes's Nachricht vom 02.04.2018 18:11 (localtime): >>>> Bez?glich Daniel Braniss's Nachricht vom 02.04.2018 16:58 (localtime): >>>> >>>> ? >>>>>> >>>>>> Are you trying to use the HOSTS ip address in the GUEST? >>>>> >>>>> the client is using the server?s /usr/local, which is mounted via nfs. >>>> >>>> I guess you have reasons to prefer nfs over nullfs. >>> >>> You can not nullfs mount from inside a bhyve guest the host file >>> system would be his reason. >> >> Doh, of course – I mind-mixed with jails… Almost all my FreeBSD >> "guests" on a bhyve(8) host are jails... Please simply ignore ;-) >> >> >> … >>>> your results... Else I can't explain what you see other than the above >>>> scenario. >>> >>> Is it still true that if you have a bridge hooked to an interface >>> you must move the IP assignement from the interace device to the >>> bridge? >> >> I have no idea, I haven't used if_bridge(4) for some years. Perferably >> I use vale(4) (if I don't need .11q) or ng_bridge(4) for bhyve(8) setups. >> >> -harry > > maybe this could be the cause: > on the server, every time a client is started iI see: > kernel: bridge0: error setting interface capabilities on mlxen0 Members added to the bridge will automatically get some (TX) offloadings disabled, which seems to fail in your case. man 4 if_bridge: The TOE, TSO, TXCSUM and TXCSUM6 capabilities on all interfaces added to the bridge are disabled if any of the interfaces doesn't support/enable them. The LRO capability is always disabled. All the capabilities are restored when the interface is removed from bridge. Changing capabilities in run time may cause NIC reinit and the link flap. I don't think the kernel-bridge0-logs are related to your ARP issue – txcsum offloading enabled on if_bridge(4) uplinks cause other errors (dropped packets – one layer up). -harry From owner-freebsd-virtualization@freebsd.org Wed Apr 4 20:17:16 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C9A1FF7327F for ; Wed, 4 Apr 2018 20:17:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 682417195A for ; Wed, 4 Apr 2018 20:17:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id B32B720A3 for ; Wed, 4 Apr 2018 20:17:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w34KHFpL020240 for ; Wed, 4 Apr 2018 20:17:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w34KHFEC020237 for freebsd-virtualization@FreeBSD.org; Wed, 4 Apr 2018 20:17:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 215737] [bhyve] utilizing virtio-net truncates jumbo frames at 4084 bytes length Date: Wed, 04 Apr 2018 20:17:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bugzilla.freebsd@omnilan.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Apr 2018 20:17:17 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215737 --- Comment #9 from Harald Schmalzbauer --- (In reply to P Kern from comment #8) *offtopic, vmxnet3 specific only, nothing PR related in this comment*: It's correct that if_vmx(4) does not mention MTU or "jumbo" frames, but I w= as quiet sure it _does_ support 9k frames =E2=80=93 just verified positive (st= able/11 on ESXi6.5)! if_vmx(4) has been improved over the time, but it still lacks ALTQ support. And vmx3f(4) is still a bit more efficient. Else, if_vmx(4) is featurewise on par with vmx3f(4). Unfortunately vmx3f(4) isn't supported by VMware any longer. I made a patch which allows vmx3f(4) to be compiled on FreeBSD-11, and it a= lso seems to be stable _without_ ALTQ. ALTQ causes panics!!! Unfortunately my skills/time don't last to fix. Here's the compile-patch in case somebody wants to take over: ftp://ftp.omnilan.de/pub/FreeBSD/OmniLAN/vmware-esxi_kernel-modules/10.1.5/= FreeBSD-11_vmxnet3-Tools_10.1.5-source.patch -harry --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Apr 6 16:16:25 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D821F92202 for ; Fri, 6 Apr 2018 16:16:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 02C86841D6 for ; Fri, 6 Apr 2018 16:16:25 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id B13E2F921FC; Fri, 6 Apr 2018 16:16:24 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F139F921FB for ; Fri, 6 Apr 2018 16:16:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 3D034841D1 for ; Fri, 6 Apr 2018 16:16:24 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 88F6B21112 for ; Fri, 6 Apr 2018 16:16:23 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w36GGNQo069966 for ; Fri, 6 Apr 2018 16:16:23 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w36GGNIk069963 for virtualization@FreeBSD.org; Fri, 6 Apr 2018 16:16:23 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 215737] [bhyve] utilizing virtio-net truncates jumbo frames at 4084 bytes length Date: Fri, 06 Apr 2018 16:16:23 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: des.gaufres@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2018 16:16:25 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215737 --- Comment #10 from P Kern --- (In reply to Harald Schmalzbauer from comment #9) Thanks for the code! Yes vmx(8) does support recv/xmit of 9k frames but in a case where the VM has 2 vmx(8) NICs, the 9k frames do not seem to be able to transit in one NIC and the out the other. So 9k frames only seem to "work" when the traffic terminates at the VM (...?). Just tested this scenario on the same VM with 2 Intel em(8) NICs: 9k frames seem to pass through the VM via em0<-->em1 (mtu 9k on both) without trouble. Under the same setup but with vmx0<-->vmx1, the 9k frames cannot seem to fl= ow thru: traffic will transit only after MTUs are set to 4096. I'd love to tweak the vmx3f driver but then the VM could not be used for anything we put into production (small group here. no other BSD kernel dive= rs). thx again --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Apr 6 16:33:37 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7B924F93DE2 for ; Fri, 6 Apr 2018 16:33:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id EC3B486CA2 for ; Fri, 6 Apr 2018 16:33:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id ABA99F93DDA; Fri, 6 Apr 2018 16:33:36 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 97A2FF93DD9 for ; Fri, 6 Apr 2018 16:33:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2E57A86C99 for ; Fri, 6 Apr 2018 16:33:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 54461213E3 for ; Fri, 6 Apr 2018 16:33:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w36GXZG0013821 for ; Fri, 6 Apr 2018 16:33:35 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w36GXZGA013820 for virtualization@FreeBSD.org; Fri, 6 Apr 2018 16:33:35 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 215737] [bhyve] utilizing virtio-net truncates jumbo frames at 4084 bytes length Date: Fri, 06 Apr 2018 16:33:35 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: rgrimes@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2018 16:33:37 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215737 Rodney W. Grimes changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |rgrimes@FreeBSD.org --- Comment #11 from Rodney W. Grimes --- (In reply to P Kern from comment #10) I believe that issue of not being able to forward pacekts through a VM using vmx(4) with MTU >4K is that on the receive side the incoming packets are chunked up into n * 4k pages and these do not pass through the forwarding c= ode correctly. This in effect frag's the jumbo frame as it tries to traverse the router, and I do not think the code is up to that task, nor is that a desirable situation. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Apr 6 16:53:10 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B281AF955CE for ; Fri, 6 Apr 2018 16:53:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3CDDE68104 for ; Fri, 6 Apr 2018 16:53:10 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 0167AF955C9; Fri, 6 Apr 2018 16:53:10 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E175FF955C8 for ; Fri, 6 Apr 2018 16:53:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7FA9A68102 for ; Fri, 6 Apr 2018 16:53:09 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 73627216C4 for ; Fri, 6 Apr 2018 16:53:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w36Gr8gx059877 for ; Fri, 6 Apr 2018 16:53:08 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w36Gr8nK059875 for virtualization@FreeBSD.org; Fri, 6 Apr 2018 16:53:08 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 215737] [bhyve] utilizing virtio-net truncates jumbo frames at 4084 bytes length Date: Fri, 06 Apr 2018 16:53:07 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: des.gaufres@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2018 16:53:10 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215737 --- Comment #12 from P Kern --- (In reply to Rodney W. Grimes from comment #11) [ doh! vmx(4)/em(4) -- not ..(8)! sigh, brain rot. ] yup, I was suspecting vmx(_4_) was doing something like that. With our limited resources, our options for now are ... - live with vmx(4) NICs with 4K MTU or - switch to em(4) NICs with 9k MTU. Unless there's some other benefit to using em(4) NICs in our ESXi VMs, we'll probably stick with using vmx NICs. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Apr 6 19:56:36 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59772F81566 for ; Fri, 6 Apr 2018 19:56:36 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DF97D733ED for ; Fri, 6 Apr 2018 19:56:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 950FBF81565; Fri, 6 Apr 2018 19:56:35 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E80EF81564 for ; Fri, 6 Apr 2018 19:56:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0ADD2733E6 for ; Fri, 6 Apr 2018 19:56:35 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id 4413C22F67 for ; Fri, 6 Apr 2018 19:56:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w36JuYHi085672 for ; Fri, 6 Apr 2018 19:56:34 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w36JuYuc085671 for virtualization@FreeBSD.org; Fri, 6 Apr 2018 19:56:34 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 215737] [bhyve] utilizing virtio-net truncates jumbo frames at 4084 bytes length Date: Fri, 06 Apr 2018 19:56:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bugzilla.freebsd@omnilan.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2018 19:56:36 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215737 --- Comment #13 from Harald Schmalzbauer --- (In reply to P Kern from comment #12) In case you end up switching from "vmxnet3"/[vmx(4)|vmx3f(4)] to "e1000"/[em(4)], depending on your workload, you can save lots of overhead = if you switch to "e1000e" instead, since it utilizes MSI(-X). To make use of, you need to set 'hw.pci.honor_msi_blacklist=3D0' in loader.= conf. And then, there's a negotiaten mismatch between FreeBSD/ESXi (ESX is select= ing MSI while FreeBSD MSI-X =E2=80=93 as far as I remember). You can circumven= t by simply re-loading the kernel module! "e1000e"/[if_em(4)] works fine in MSI-X mode. Since FreeBSD-11, there's also devctl(8), which could take care of the driv= er re-initialization, but when I wrote my rc(8) script to automatically re-load kernel modules on ESXi guests, it was not available. Happy to share the rc(8) script on request. -harry --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Apr 6 20:01:59 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 320D8F81D11 for ; Fri, 6 Apr 2018 20:01:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id AA74573910 for ; Fri, 6 Apr 2018 20:01:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 5B0FCF81D0F; Fri, 6 Apr 2018 20:01:58 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43A93F81D0E for ; Fri, 6 Apr 2018 20:01:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AEC8673907 for ; Fri, 6 Apr 2018 20:01:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id D98EE230AA for ; Fri, 6 Apr 2018 20:01:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w36K1uRh014604 for ; Fri, 6 Apr 2018 20:01:56 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w36K1u6O014603 for virtualization@FreeBSD.org; Fri, 6 Apr 2018 20:01:56 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 215737] [bhyve] utilizing virtio-net truncates jumbo frames at 4084 bytes length Date: Fri, 06 Apr 2018 20:01:56 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: des.gaufres@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 06 Apr 2018 20:01:59 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215737 --- Comment #14 from P Kern --- (In reply to Harald Schmalzbauer from comment #13) > Happy to share the rc(8) script on request. yup, "request" please. thx. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Sat Apr 7 10:08:03 2018 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1CE12F9E1AD for ; Sat, 7 Apr 2018 10:08:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 98F1A70032 for ; Sat, 7 Apr 2018 10:08:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 5766DF9E1A9; Sat, 7 Apr 2018 10:08:02 +0000 (UTC) Delivered-To: virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 43B3EF9E1A8 for ; Sat, 7 Apr 2018 10:08:02 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.ysv.freebsd.org (mxrelay.ysv.freebsd.org [IPv6:2001:1900:2254:206a::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.ysv.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C9D7F70017 for ; Sat, 7 Apr 2018 10:08:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.ysv.freebsd.org (Postfix) with ESMTPS id F12BC242E for ; Sat, 7 Apr 2018 10:08:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id w37A80SJ012846 for ; Sat, 7 Apr 2018 10:08:00 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id w37A80e2012845 for virtualization@FreeBSD.org; Sat, 7 Apr 2018 10:08:00 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 215737] [bhyve] utilizing virtio-net truncates jumbo frames at 4084 bytes length Date: Sat, 07 Apr 2018 10:08:00 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: misc X-Bugzilla-Version: 11.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: bugzilla.freebsd@omnilan.de X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 07 Apr 2018 10:08:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D215737 --- Comment #15 from Harald Schmalzbauer --- (In reply to P Kern from comment #14) Sorry for so much nonsense and off-topic comments; But to correct myself in case anybody else wonders.... You do _not_ need the driver reload hack for the "e1000e" _virtual_ 82574 (Intel Hartwell, if_em(4))!!! [e1000 =3D 82545, which doesn't support MSI, = just to mention] I just read with one eye and confused passthrough interfaces, which is what= I prefere to have on ESXi for my FreeBSD guests (most often with 82574 or 825= 76). Only the passthru hardware needs the MSI-X negotiation driver reload workaround. But you probably need the 'hw.pci.honor_msi_blacklist=3D0' in loader.conf = =E2=80=93 don't remember well, so please check yourself if you want to avoid unnecessary co= nfig options, even if they don't do any harm. -harry --=20 You are receiving this mail because: You are the assignee for the bug.=