From owner-freebsd-virtualization@freebsd.org Sun Aug 9 00:14:46 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C1B7D3A8AEE for ; Sun, 9 Aug 2020 00:14:46 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (gate2.funkthat.com [208.87.223.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gate2.funkthat.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BPKNP0Y80z4TZD for ; Sun, 9 Aug 2020 00:14:44 +0000 (UTC) (envelope-from jmg@gold.funkthat.com) Received: from gold.funkthat.com (localhost [127.0.0.1]) by gold.funkthat.com (8.15.2/8.15.2) with ESMTPS id 0790Eal8081174 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 8 Aug 2020 17:14:37 -0700 (PDT) (envelope-from jmg@gold.funkthat.com) Received: (from jmg@localhost) by gold.funkthat.com (8.15.2/8.15.2/Submit) id 0790EaxT081173; Sat, 8 Aug 2020 17:14:36 -0700 (PDT) (envelope-from jmg) Date: Sat, 8 Aug 2020 17:14:36 -0700 From: John-Mark Gurney To: Mark Raynsford Cc: freebsd-virtualization@freebsd.org Subject: Re: Restricting IP ranges for guests over tap devices Message-ID: <20200809001436.GK4213@funkthat.com> References: <20200801145144.7bf342d9@sunflower.int.arc7.info> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="iig7nzZQzi/oiJm2" Content-Disposition: inline In-Reply-To: <20200801145144.7bf342d9@sunflower.int.arc7.info> X-Operating-System: FreeBSD 11.3-STABLE amd64 X-PGP-Fingerprint: D87A 235F FB71 1F3F 55B7 ED9B D5FF 5A51 C0AC 3D65 X-Files: The truth is out there X-URL: https://www.funkthat.com/ X-Resume: https://www.funkthat.com/~jmg/resume.html X-TipJar: bitcoin:13Qmb6AeTgQecazTWph4XasEsP7nGRbAPE X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? User-Agent: Mutt/1.6.1 (2016-04-27) X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.4.3 (gold.funkthat.com [127.0.0.1]); Sat, 08 Aug 2020 17:14:37 -0700 (PDT) X-Rspamd-Queue-Id: 4BPKNP0Y80z4TZD X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of jmg@gold.funkthat.com has no SPF policy when checking 208.87.223.18) smtp.mailfrom=jmg@gold.funkthat.com X-Spamd-Result: default: False [-2.54 / 15.00]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.68)[-0.677]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.96)[-0.960]; TAGGED_RCPT(0.00)[org.freebsd.virtualization]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[funkthat.com]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.01)[-0.007]; RCPT_COUNT_TWO(0.00)[2]; SIGNED_PGP(-2.00)[]; FORGED_SENDER(0.30)[jmg@funkthat.com,jmg@gold.funkthat.com]; R_SPF_NA(0.00)[no SPF record]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:32354, ipnet:208.87.216.0/21, country:US]; FROM_NEQ_ENVFROM(0.00)[jmg@funkthat.com,jmg@gold.funkthat.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 09 Aug 2020 00:14:46 -0000 --iig7nzZQzi/oiJm2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Mark Raynsford via freebsd-virtualization wrote this message on Sat, Aug 01= , 2020 at 14:51 +0000: > Let's say I have a machine running a few dozen bhyve guests. Each bhyve > guest gets its own tap device, and all of the tap devices are connected > to a bridge. >=20 > Everything works fine. I can write pf rules that control access between > each guest, and between each guest and the world. I can't directly > observe the IP addresses that the guests have assigned to the tap > devices I gave them, but if I know the addresses beforehand, I can for > example write pf rules that say things like: >=20 > block log all > pass in on tap23 proto tcp \ > from any to $guest_23_ip port ssh modulate state >=20 > That then means that even if the guest is compromised and tries to bind > a server to another address, the pf rules won't allow anyone else to > actually connect to it. >=20 > The good thing about this is also the bad thing about this; I have to > write specific rules that say "only allow access to this specific IP > via this specific tap device". Over dozens of guests, that can multiply > to hundreds of laboriously maintained rules. >=20 > Is there some more general way I can supply a mapping between tap > devices and allowed addresses? Remember that pf can't see the guest > addresses on the host sides of the tap devices, so I can't use the > (device) syntax to expand to "the address of a NIC called 'device'". >=20 > I can generate rule sets, but perhaps there's something "better"[0]? The > documentation isn't suggesting much. >=20 > [0] Better in the sense that, for example, a table is usually better > than a massive list of macros. :) Don't think there is anything better... bridge does have sticky that binds the mac address to an interface, but that doesn't deal w/ IP ARP. One issue w/ this is how do you know the difference between one machine that's been down for a long time, and an attacking machine that takes over the down'd machine's IP address? I assume that these addresses are assigned via DHCP server, otherwise if you are launching the VM's w/ known static IP's, you could use pf's anchor directive, and each start/stop of a VM, update the rule for that tap's anchor. --=20 John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." --iig7nzZQzi/oiJm2 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJfLz/rXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ2MEI1RTRGMTNDNzYyMDZDNjEyMDBCNjAy MDVGMEIzM0REMDA2QURBAAoJECBfCzPdAGrazVYP/iiny6XQQdgJBmyde95I/NNe ZYFVPRe+PW5a2683qqhDLnQy9LJb4j0yNAut13YM1wIc4IQjJGPhs/iPrQ5GelFi 86ti5epue+r42Ln82yURz/KCrsFa46UXmpgOc42N20zBq0OgvROPfPvuZEScS6iT EDrET0Gt9PJjxMeipm84MADNv1IVrMfaA450T9US8FluH9xrLozu3of09Ml9/OqS xMdAxgwj15fzsB8dSIqqlWpIGrYcNrUbsgDKRX5xhVXiO11wSegZMbfu/bQtWdcF jAB0xD4bTlvOXHa7ACePau3plbKQ/CPy7S8VgkUJJjfWNx0BKTldb/pjaA0DCxht xxfrYx/6WXA37Wdhg28jRL5i1CL7tESgloLe0Lyxri5ahSHhJYGzv/nJpdBf4W/G 84iAtryDXkNGhJq9vvvJ8d/qTmMREehg+abxdxrUDvOl3QkyxBiwhGBsrdsxf4Yz g4XT7JG/UMy+miA06CufQqRhwt5RLuxxlqEmHTjF8DtRYLa7O2TRrZVysQ64VAWv 600amFEYMZq+8QUKjuXUOMKlszUdVNZN61pFOzYwvkAEkxuK/Yq+ssvj2BBcZDzE yYA/R394zZ4S5IaLcXWwjCBwPuWqe0VR3cEplH4hRstJq90xVfwJ8iaUxlHQERW5 vzSWvLLVQa2r3yq/n2Pm =4ATH -----END PGP SIGNATURE----- --iig7nzZQzi/oiJm2--