Date: Wed, 16 Oct 2024 16:19:40 +0200 From: Palle Girgensohn <girgen@FreeBSD.org> To: "freebsd-net@freebsd.org" <freebsd-net@FreeBSD.org> Subject: pf for netgraph jails? Message-ID: <7D5BD9CC-8A08-4C74-B2E6-E0437235F3B1@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
--Apple-Mail=_AC6D5B8C-A6E6-413A-BF25-E00942506C27 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi! Using FreeBSD-14.1, I have a rather simple setup with jails using = netgraph (using the `/usr/share/examples/jails/jng` script and "model"). The host machine has two interfaces: bnxt0: (external, has no IP#) bnxt1: 192.168.1.79/24 jail.conf: -- host.hostname =3D "$name.example.com <http://name.example.com/>"; # = hostname path =3D "/jails/$name"; exec.clean; exec.system_user =3D "root"; exec.jail_user =3D "root"; vnet; # netgraph vnet.interface =3D ng0_$name, ng1_$name; # vnet interface(s) exec.prestart +=3D "jng bridge $name bnxt0 bnxt1"; # bridge = interface(s) exec.poststop +=3D "jng shutdown $name"; # destroy interface(s) exec.start +=3D "/bin/sh /etc/rc"; exec.stop =3D "/bin/sh /etc/rc.shutdown jail"; exec.consolelog =3D "/var/log/jail_$name.log"; mount.devfs; # mount devfs mount.fdescfs; devfs_ruleset=3D5; allow.mlock=3D1; mount.fstab=3D"/etc/fstab.$name"; fw {} -- which creates a single jail `fw'. /jails/fw/etc/rc.conf: -- hostname=3Dfw.example.com <http://fw.example.com/>; ifconfig_ng0_fw=3D"inet 1.2.3.4/26" ifconfig_ng1_fw=3D"inet 192.168.1.212/24" defaultrouter=3D"1.2.3.1" sshd_enable=3D"yes" -- $ sudo ngctl list There are 8 total nodes: Name: ngctl69965 Type: socket ID: 00000021 Num hooks: 0 Name: bnxt0 Type: ether ID: 00000001 Num hooks: 2 Name: bnxt1 Type: ether ID: 00000002 Num hooks: 2 Name: ue0 Type: ether ID: 00000003 Num hooks: 0 Name: bnxt0bridge Type: bridge ID: 00000009 Num hooks: 3 Name: ng0_fw Type: eiface ID: 0000000e Num hooks: 1 Name: bnxt1bridge Type: bridge ID: 00000016 Num hooks: 3 Name: ng1_fw Type: eiface ID: 0000001b Num hooks: 1 I plan to create a reasonably large number of jails this way, by just = adding jname {} to the jail.conf file. Now, I would like to have a simple generic setup with pf filtering out = unwanted ports from incoming traffic. I tried this simplistic setup: -- ext_if =3D "bnxt0" int_if =3D "bnxt1" block in on $ext_if dns_servers =3D "{ 192.168.1.194, 1.2.3.9, 8.8.8.8, 1.1.1.1 }" pass in on $ext_if proto { tcp udp } from $dns_servers to any port 53 pass in on $ext_if proto tcp from any to any port { 80 443 22 } -- but nothing happens, everything is passed directly into the jail: nc -l 4444 (inside the jail) and I can just telnet 1.2.3.4 4444 I assume I'm doing some simple mistake here, but find very little = information wrt the combo of netgraph, pf and jails. Any tips? I tried = configuring pf to work on the bridge interface but no difference. What = am I missing here? Palle= --Apple-Mail=_AC6D5B8C-A6E6-413A-BF25-E00942506C27 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html><head><meta http-equiv=3D"content-type" content=3D"text/html; = charset=3Dus-ascii"></head><body style=3D"overflow-wrap: break-word; = -webkit-nbsp-mode: space; line-break: after-white-space;"><meta = http-equiv=3D"content-type" content=3D"text/html; charset=3Dus-ascii"><div= style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; = line-break: after-white-space;"><font face=3D"Menlo-Regular">Hi!</font><br= style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">Using = FreeBSD-14.1, </span><span style=3D"font-family: Menlo-Regular;">I = have a rather simple setup with jails using netgraph (using the = `/usr/share/examples/jails/jng` script and "model").</span><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">The host = machine has two interfaces:</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">bnxt0: = (external, has no IP#)</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">bnxt1: = 192.168.1.79/24</span><br style=3D"font-family: Menlo-Regular;"><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">jail.conf:</span><br style=3D"font-family: = Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">--</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">host.hostname =3D "$</span><a = href=3D"http://name.example.com/" style=3D"font-family: = Menlo-Regular;">name.example.com</a><span style=3D"font-family: = Menlo-Regular;">"; # hostname</span><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">path =3D = "/jails/$name";</span><br style=3D"font-family: Menlo-Regular;"><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">exec.clean;</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">exec.system_user =3D "root";</span><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">exec.jail_user =3D "root";</span><br style=3D"font-family:= Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">vnet;</span><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;"># = netgraph</span><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">vnet.interface =3D ng0_$name, = ng1_$name; # vnet = interface(s)</span><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">exec.prestart +=3D "jng bridge = $name bnxt0 bnxt1"; # bridge = interface(s)</span><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">exec.poststop +=3D "jng shutdown = $name"; # destroy interface(s)</span><br style=3D"font-family: = Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">exec.start +=3D "/bin/sh = /etc/rc";</span><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">exec.stop =3D "/bin/sh = /etc/rc.shutdown jail";</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">exec.consolelog =3D "/var/log/jail_$name.log";</span><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">mount.devfs; # mount devfs</span><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">mount.fdescfs;</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">devfs_ruleset=3D5;</span><br style=3D"font-family: = Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">allow.mlock=3D1;</span><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">mount.fstab=3D"/etc/fstab.$name";</span><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">fw = {}</span><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">--</span><br style=3D"font-family: = Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">which creates a single jail = `fw'.</span><br style=3D"font-family: Menlo-Regular;"><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">/jails/fw/etc/rc.conf:</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">--</span><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">hostname=3D</span><a href=3D"http://fw.example.com/" = style=3D"font-family: Menlo-Regular;">fw.example.com</a><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">ifconfig_ng0_fw=3D"inet 1.2.3.4/26"</span><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">ifconfig_ng1_fw=3D"inet 192.168.1.212/24"</span><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">defaultrouter=3D"1.2.3.1"</span><br style=3D"font-family: = Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">sshd_enable=3D"yes"</span><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">--</span><br style=3D"font-family: Menlo-Regular;"><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">$ sudo ngctl = list</span><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">There are 8 total nodes:</span><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;"> Name: ngctl69965 = Type: socket = ID: 00000021 = Num hooks: 0</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;"> Name: = bnxt0 Type: = ether ID: = 00000001 Num hooks: 2</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;"> Name: = bnxt1 Type: = ether ID: = 00000002 Num hooks: 2</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;"> Name: = ue0 = Ty= pe: ether = ID: 00000003 = Num hooks: 0</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;"> Name: = bnxt0bridge Type: bridge = ID: 00000009 = Num hooks: 3</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;"> Name: = ng0_fw Type: = eiface ID: = 0000000e Num hooks: 1</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;"> Name: = bnxt1bridge Type: bridge = ID: 00000016 = Num hooks: 3</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;"> Name: = ng1_fw Type: = eiface ID: = 0000001b Num hooks: 1</span><br style=3D"font-family: = Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">I plan to create a reasonably = large number of jails this way, by just adding jname {} to the jail.conf = file.</span><br style=3D"font-family: Menlo-Regular;"><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">Now, I would like to have a simple generic setup with pf = filtering out unwanted ports from incoming traffic.</span><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">I tried this = simplistic setup:</span><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">--</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">ext_if =3D = "bnxt0"</span><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">int_if =3D "bnxt1"</span><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">block in on $ext_if</span><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">dns_servers =3D "{ 192.168.1.194, = 1.2.3.9, 8.8.8.8, 1.1.1.1 }"</span><br style=3D"font-family: = Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">pass in on $ext_if proto { tcp udp } from $dns_servers = to any port 53</span><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">pass in on $ext_if proto tcp from = any to any port { 80 443 22 }</span><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">--</span><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">but nothing = happens, everything is passed directly into the jail:</span><br = style=3D"font-family: Menlo-Regular;"><br style=3D"font-family: = Menlo-Regular;"><span style=3D"font-family: Menlo-Regular;">nc -l 4444 = (inside the jail)</span><br style=3D"font-family: = Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">and I can just telnet 1.2.3.4 = 4444</span><br style=3D"font-family: Menlo-Regular;"><br = style=3D"font-family: Menlo-Regular;"><span style=3D"font-family: = Menlo-Regular;">I assume I'm doing some simple mistake here, but find = very little information wrt the combo of netgraph, pf and jails. Any = tips? I tried configuring pf to work on the bridge interface but = no difference. What am I missing here?</span><br style=3D"font-family: = Menlo-Regular;"><br style=3D"font-family: Menlo-Regular;"><span = style=3D"font-family: Menlo-Regular;">Palle</span></div></body></html>= --Apple-Mail=_AC6D5B8C-A6E6-413A-BF25-E00942506C27--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7D5BD9CC-8A08-4C74-B2E6-E0437235F3B1>