Date: Wed, 10 Dec 2025 19:30:53 +0100 From: =?UTF-8?Q?=C5=81ukasz_Moska=C5=82a?= <lmoskala@devology.pl> To: Anthony Pankov <anthony.pankov@yahoo.com>, freebsd-jail@freebsd.org Subject: Re: shared (between jails) local host interface Message-ID: <c5c6d863-3f9b-4e52-9c9d-e7294fbfef14@devology.pl> In-Reply-To: <2397748.20251210141029@yahoo.com> References: <2397748.20251210141029.ref@yahoo.com> <2397748.20251210141029@yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
So, what I have done is that I have /31 subnets for jails (every jail has it's own subnet, with host on lower address and jail on higher).
I just put IP addresses directly on epair interfaces and use host as a router. No bridge interfaces at all.
So every jail can reach any other jail via it's address, and if I transfer said jail to another host, it retains it's address.
The jail's /31 subnets are announced via BIRD running on host to the rest of my infrastructure.
I also have DNS at address 10.0.0.0 and 10.0.0.255 - these two addresses are as /32 on loopback in one jail, and host has route set to them (again via bird with BGP).
Each host has the same jail with dns at the same anycast address 10.0.0.0 and 10.0.0.255 (and of course different unicast address so that I can actually ssh into them) and BGP takes care of figuring out which one is the closest (usually the one on the same host), and if one goes down, then BIRD recalculates route to the next best one.
That's basically anycast setup with host as router. And the host would have to peer with another host(s) and probably upstream router/firewall as well - so that the rest of the network knows routes to your jails.
This also works well with ipv6 however I stopped using it due to unrelated problems with tunnelbroker.net (and my ISP doesn't support it)
And if you wouldn't want to have the DNS address accessible outside of host - you can just set up BIRD so that it doesn't redistribute that prefix
Keep in mind that this setup makes it so that the jail's network is NOT the same network that is on the wire - it's completely virtual, and traffic from it only appears on wire if it leaves host. And thus jails don't see broadcast traffic from real network so that would for example prevent you from running DHCP server inside of jail. Or basically anything that requires L2 connectivity for one reason or another.
This may be very much overkill for most use cases (mine included) but it has been rock solid for the past 3 years that I had this setup. Currently it's scaled down to one host but if I would need it, I could add another one in a couple of hours.
Best regards
Łukasz
On 12/10/25 12:10, Anthony Pankov wrote:
> Hello,
>
> I'm again facing the problem of providing host-wide interface to all jails in a host.
> I want (for example) that there is a jail with unbound (DNS resolver/cacher) inside, binded to a host-wide interface. Host-wide interface has a well known address (IPv6 1::1/128 for example).
> All other jails simply put "nameserver 1::1" in its resolv.conf.
>
> The problem that jails can't share one ip address ("address clashes") and can't "inherit" one interface only. Inheriting is for a whole network as I see.
>
> May be someone have some suggestions?
>
>
> P.S. Sometime ago I solve this in a not a clean manner by supplying own loopback interface and unique address (127.0.X.1) to every jail in a host. Then jail with unbound bind to its 127.0.U.1, other jail must set "nameserver 127.0.U.1". This solution has administrative problems because of a dynamic nature of addresses (127.0.X.1). Thing get worse if jails transferred to a different host.
>
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?c5c6d863-3f9b-4e52-9c9d-e7294fbfef14>