From owner-freebsd-virtualization@freebsd.org Tue Oct 20 11:39:05 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8D8EB42E2AD for ; Tue, 20 Oct 2020 11:39:05 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from mail.punkt.de (mail.punkt.de [IPv6:2a00:b580:8000:11:1c6b:7032:35e9:5616]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CFs8m5Zmlz4Cp1 for ; Tue, 20 Oct 2020 11:39:04 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from [217.29.46.75] (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.punkt.de (Postfix) with ESMTPSA id 9DF0310B0A; Tue, 20 Oct 2020 13:39:01 +0200 (CEST) From: "Patrick M. Hausen" Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_E71AA62E-07B7-4771-9C33-60105090E42C"; protocol="application/pgp-signature"; micalg=pgp-sha256 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.17\)) Subject: Re: When is a switch not a switch? Date: Tue, 20 Oct 2020 13:39:00 +0200 In-Reply-To: <973b1b56-817f-6976-e5d3-34cfbc373b13@druid.net> Cc: freebsd-virtualization@freebsd.org To: D'Arcy Cain References: <57c32e6d-5572-3d3b-1a57-f3064bee7dc2@druid.net> <20201020065630.GE8272@funkthat.com> <3ed627e2-d99a-107e-4135-8aef1ad4ec71@druid.net> <30A67F82-312E-4651-A5E7-2E2AD926FF24@punkt.de> <973b1b56-817f-6976-e5d3-34cfbc373b13@druid.net> X-Mailer: Apple Mail (2.3445.104.17) X-Rspamd-Queue-Id: 4CFs8m5Zmlz4Cp1 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 2a00:b580:8000:11:1c6b:7032:35e9:5616 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-3.61 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:b580::/32:c]; MV_CASE(0.50)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; HAS_ATTACHMENT(0.00)[]; DMARC_NA(0.00)[punkt.de]; MID_RHS_MATCH_FROM(0.00)[]; NEURAL_HAM_LONG(-1.01)[-1.007]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.72)[-0.718]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.98)[-0.984]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:16188, ipnet:2a00:b580::/32, country:DE]; SUBJECT_ENDS_QUESTION(1.00)[]; MAILMAN_DEST(0.00)[freebsd-virtualization]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Oct 2020 11:39:05 -0000 --Apple-Mail=_E71AA62E-07B7-4771-9C33-60105090E42C Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi all, > Am 20.10.2020 um 12:50 schrieb D'Arcy Cain : >=20 > On 10/20/20 5:36 AM, Patrick M. Hausen wrote: >>> I did see that. Does that mean that I don't even need to create = switches at all? >> What is a switch in this context? I use bridge interfaces to connect = jails via epair >> and VMs via tap. >=20 > When I started I thought of a switch as analogous to a physical = switch. If I am in an office with one ethernet jack but I have multiple = devices I might connect a switch (or hub) to the jack and plug my = devices into the switch. I don't need to create a separate network for = my office. All of my devices are on the company network. OK, the "switch" interface in FreeBSD is bridge(4). Or to cite Radia Perlman: A bridge is a network device making forwarding decisions based on layer = 2 addresses. A router is a network device making forwarding decisions based on layer = 3 addresses. "Switch" is a marketing term meaning "faster or cheaper than the = competition". > cloned_interfaces=3D"bridge0" > ifconfig_bridge0=3D"a.b.c.d.1 addm bge0 addm switch0 up" > Except that switch0 doesn't get created until vm-bhyve starts so it = probably doesn't exist at that time. What is "switch0"? I suspect it is just a bridge interface that gets = renamed by your VM management software. In that case manually creating bridge0 and all the things we discussed will not get you anywhere. >> If em0 does not have an IP address on the host and should be used >> exclusively for VMs, then the bridge does not need an IP address, = either. >> Still you need to configure em0 "up". >=20 > I can't imagine a scenario like that. You probably always need access = to the host for maintenance. Well, there could be a second hardware interface for host communication = ... And if one of the two is member of the bridge and the other one isn't it = is perfectly valid to plug them into the same broadcast domain and get e.g. 1Gbit/s for the host and 1Gbit/s for all the jails or VMs. >> And additionally ... >> - you should disable all hardware acceleration features on the = physical interface >=20 > Like ASF? Real life example from our environment: ifconfig_igb0=3D"-rxcsum -rxcsum6 -txcsum -txcsum6 -tso -vlanhwtag = -vlanhwtso up" cloned_interfaces=3D"bridge0" ifconfig_bridge0_name=3D"inet0" ifconfig_inet0=3D"addm igb0 up" ifconfig_inet0_alias0=3D"inet 1.2.3.4/24" Then we configure iocage to attach the jails to bridge0. In your case you would have to tell your VM management tool to attach = the VM tap interfaces to bridge0 instead of creating its own "switch0" - = which I suspect is a bridge interface in disguise. As you can see above we = rename all our Internet facing interfaces to "inet0" on all hosts. Then there = are more like "mgmt0", "priv0", ... like that. So probably the bridge is renamed = to "switch0". Tell the tool not to do that and use the preconfigured bridge0 instead. Another useful sysctl to get reproduceable static MAC addresses for the = bridge itself accross reboots is: loader.conf: if_bridge_load=3D"YES" sysctl.conf: net.link.bridge.inherit_mac=3D1 HTH, Patrick -- punkt.de GmbH Patrick M. Hausen .infrastructure Kaiserallee 13a 76133 Karlsruhe Tel. +49 721 9109500 https://infrastructure.punkt.de info@punkt.de AG Mannheim 108285 Gesch=C3=A4ftsf=C3=BChrer: J=C3=BCrgen Egeling, Daniel Lienert, Fabian = Stein --Apple-Mail=_E71AA62E-07B7-4771-9C33-60105090E42C Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- iQEzBAEBCAAdFiEEgzqrjO/mj9CSsTg2kG8u4u3aiVwFAl+OzFQACgkQkG8u4u3a iVwDrggAk1Ah6hgV4QhtC+Vt+pU8AWuZdazV9gsnIC/KwkJeoPb9VaIIgiFuWeAn qpUAZKQ/talEYkXiTaVqHc5ZZXqibUShjWdh2Bx9swCih/HakFgzxw3Jspwq1gI7 665ZtSC1YPnpbcECZ+/gDvw1z8wVHVgrbpa9WrVm9Wt1omPn2wSeQcHMGmlGShWF 49KfHqVWt/FjzReM1wAOqsDFuHzfltIqQPQV4ptvhbgtvnE3bdCIbEjMba2f7zmx A0tbHWR8SnnW2NnrqWLDHP8V8wZ9a/62J3wuatU3tBXS9jmYVRVytwS0Tvrosart EQmtmq8/rx4uZ1X1vM4at9zhAXEHzA== =gdBj -----END PGP SIGNATURE----- --Apple-Mail=_E71AA62E-07B7-4771-9C33-60105090E42C--