Date: Wed, 16 Oct 2024 16:19:40 +0200 From: Palle Girgensohn <girgen@FreeBSD.org> To: "freebsd-net@freebsd.org" <freebsd-net@FreeBSD.org> Subject: pf for netgraph jails? Message-ID: <7D5BD9CC-8A08-4C74-B2E6-E0437235F3B1@FreeBSD.org>
next in thread | raw e-mail | index | archive | help
[-- Attachment #1 --] Hi! Using FreeBSD-14.1, I have a rather simple setup with jails using netgraph (using the `/usr/share/examples/jails/jng` script and "model"). The host machine has two interfaces: bnxt0: (external, has no IP#) bnxt1: 192.168.1.79/24 jail.conf: -- host.hostname = "$name.example.com <http://name.example.com/>"; # hostname path = "/jails/$name"; exec.clean; exec.system_user = "root"; exec.jail_user = "root"; vnet; # netgraph vnet.interface = ng0_$name, ng1_$name; # vnet interface(s) exec.prestart += "jng bridge $name bnxt0 bnxt1"; # bridge interface(s) exec.poststop += "jng shutdown $name"; # destroy interface(s) exec.start += "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown jail"; exec.consolelog = "/var/log/jail_$name.log"; mount.devfs; # mount devfs mount.fdescfs; devfs_ruleset=5; allow.mlock=1; mount.fstab="/etc/fstab.$name"; fw {} -- which creates a single jail `fw'. /jails/fw/etc/rc.conf: -- hostname=fw.example.com <http://fw.example.com/>; ifconfig_ng0_fw="inet 1.2.3.4/26" ifconfig_ng1_fw="inet 192.168.1.212/24" defaultrouter="1.2.3.1" sshd_enable="yes" -- $ sudo ngctl list There are 8 total nodes: Name: ngctl69965 Type: socket ID: 00000021 Num hooks: 0 Name: bnxt0 Type: ether ID: 00000001 Num hooks: 2 Name: bnxt1 Type: ether ID: 00000002 Num hooks: 2 Name: ue0 Type: ether ID: 00000003 Num hooks: 0 Name: bnxt0bridge Type: bridge ID: 00000009 Num hooks: 3 Name: ng0_fw Type: eiface ID: 0000000e Num hooks: 1 Name: bnxt1bridge Type: bridge ID: 00000016 Num hooks: 3 Name: ng1_fw Type: eiface ID: 0000001b Num hooks: 1 I plan to create a reasonably large number of jails this way, by just adding jname {} to the jail.conf file. Now, I would like to have a simple generic setup with pf filtering out unwanted ports from incoming traffic. I tried this simplistic setup: -- ext_if = "bnxt0" int_if = "bnxt1" block in on $ext_if dns_servers = "{ 192.168.1.194, 1.2.3.9, 8.8.8.8, 1.1.1.1 }" pass in on $ext_if proto { tcp udp } from $dns_servers to any port 53 pass in on $ext_if proto tcp from any to any port { 80 443 22 } -- but nothing happens, everything is passed directly into the jail: nc -l 4444 (inside the jail) and I can just telnet 1.2.3.4 4444 I assume I'm doing some simple mistake here, but find very little information wrt the combo of netgraph, pf and jails. Any tips? I tried configuring pf to work on the bridge interface but no difference. What am I missing here? Palle [-- Attachment #2 --] <html><head><meta http-equiv="content-type" content="text/html; charset=us-ascii"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><meta http-equiv="content-type" content="text/html; charset=us-ascii"><div style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><font face="Menlo-Regular">Hi!</font><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">Using FreeBSD-14.1, </span><span style="font-family: Menlo-Regular;">I have a rather simple setup with jails using netgraph (using the `/usr/share/examples/jails/jng` script and "model").</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">The host machine has two interfaces:</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">bnxt0: (external, has no IP#)</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">bnxt1: 192.168.1.79/24</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">jail.conf:</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">--</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">host.hostname = "$</span><a href="http://name.example.com/" style="font-family: Menlo-Regular;">name.example.com</a><span style="font-family: Menlo-Regular;">"; # hostname</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">path = "/jails/$name";</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">exec.clean;</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">exec.system_user = "root";</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">exec.jail_user = "root";</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">vnet;</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;"># netgraph</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">vnet.interface = ng0_$name, ng1_$name; # vnet interface(s)</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">exec.prestart += "jng bridge $name bnxt0 bnxt1"; # bridge interface(s)</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">exec.poststop += "jng shutdown $name"; # destroy interface(s)</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">exec.start += "/bin/sh /etc/rc";</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">exec.stop = "/bin/sh /etc/rc.shutdown jail";</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">exec.consolelog = "/var/log/jail_$name.log";</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">mount.devfs; # mount devfs</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">mount.fdescfs;</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">devfs_ruleset=5;</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">allow.mlock=1;</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">mount.fstab="/etc/fstab.$name";</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">fw {}</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">--</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">which creates a single jail `fw'.</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">/jails/fw/etc/rc.conf:</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">--</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">hostname=</span><a href="http://fw.example.com/" style="font-family: Menlo-Regular;">fw.example.com</a><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">ifconfig_ng0_fw="inet 1.2.3.4/26"</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">ifconfig_ng1_fw="inet 192.168.1.212/24"</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">defaultrouter="1.2.3.1"</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">sshd_enable="yes"</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">--</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">$ sudo ngctl list</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">There are 8 total nodes:</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;"> Name: ngctl69965 Type: socket ID: 00000021 Num hooks: 0</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;"> Name: bnxt0 Type: ether ID: 00000001 Num hooks: 2</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;"> Name: bnxt1 Type: ether ID: 00000002 Num hooks: 2</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;"> Name: ue0 Type: ether ID: 00000003 Num hooks: 0</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;"> Name: bnxt0bridge Type: bridge ID: 00000009 Num hooks: 3</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;"> Name: ng0_fw Type: eiface ID: 0000000e Num hooks: 1</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;"> Name: bnxt1bridge Type: bridge ID: 00000016 Num hooks: 3</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;"> Name: ng1_fw Type: eiface ID: 0000001b Num hooks: 1</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">I plan to create a reasonably large number of jails this way, by just adding jname {} to the jail.conf file.</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">Now, I would like to have a simple generic setup with pf filtering out unwanted ports from incoming traffic.</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">I tried this simplistic setup:</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">--</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">ext_if = "bnxt0"</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">int_if = "bnxt1"</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">block in on $ext_if</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">dns_servers = "{ 192.168.1.194, 1.2.3.9, 8.8.8.8, 1.1.1.1 }"</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">pass in on $ext_if proto { tcp udp } from $dns_servers to any port 53</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">pass in on $ext_if proto tcp from any to any port { 80 443 22 }</span><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">--</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">but nothing happens, everything is passed directly into the jail:</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">nc -l 4444 (inside the jail)</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">and I can just telnet 1.2.3.4 4444</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">I assume I'm doing some simple mistake here, but find very little information wrt the combo of netgraph, pf and jails. Any tips? I tried configuring pf to work on the bridge interface but no difference. What am I missing here?</span><br style="font-family: Menlo-Regular;"><br style="font-family: Menlo-Regular;"><span style="font-family: Menlo-Regular;">Palle</span></div></body></html>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7D5BD9CC-8A08-4C74-B2E6-E0437235F3B1>