Date: Thu, 06 Jan 2022 09:14:12 +0000 From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 260973] pf: firewall rules stop matching when vnet jails share interface names with the host Message-ID: <bug-260973-227@https.bugs.freebsd.org/bugzilla/>
next in thread | raw e-mail | index | archive | help
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D260973 Bug ID: 260973 Summary: pf: firewall rules stop matching when vnet jails share interface names with the host Product: Base System Version: 13.0-STABLE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: thomas@gibfest.dk Hello, I've been building a new vnet jailhost on 13 and I am hitting a weird issue where pf stops permitting traffic it clearly has rules to allow after interfaces inside vnet jails are renamed to the same name as the host inter= face with the pf rule. This is on FreeBSD nuc1.servers.bornhack.org 13.0-STABLE FreeBSD 13.0-STABL= E #1 stable/13-d208638c5: Wed Jan 5 13:32:08 UTC 2022=20=20=20=20 root@nuc1.servers.bornhack.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC am= d64 The complete ruleset is pretty complex but I've managed to cook it down to a few lines: [tykling@nuc1 ~]$ cat testpf.conf=20 block log all set skip on lo0 pass in quick on { em0 } proto { tcp } from { 85.235.250.87 } to { (em0) } = port { 22 } [tykling@nuc1 ~]$=20 The host has an em0 interface: [tykling@nuc1 ~]$ ifconfig em0 em0: flags=3D8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 =20=20=20=20=20=20=20 options=3D481049b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,VLA= N_HWFILTER,NOMAP> ether 1c:69:7a:ab:fe:be inet 85.209.118.130/28 broadcast 85.209.118.143 inet6 fe80::1e69:7aff:feab:febe%em0/64 scopeid 0x1 inet6 2a09:94c4:55d1:7680::82/64 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=3D21<PERFORMNUD,AUTO_LINKLOCAL> [tykling@nuc1 ~]$=20 The issue seems to be triggered by renaming epair interfaces inside vnet ja= ils to the same name as an interface on the host. The above pf ruleset works and keeps working if I don't start any vnet jail= s. It also keeps working if I start vnet jails but don't rename interfaces. It also keeps working if I start vnet jails but rename the interfaces to somet= hing other than em0. Existing states established before the issue happens keep working (I am wor= king remote via ssh on the server), but new states seem to just ignore the permit rule on em0, and the traffic gets blocked even though a rule should permit = it: 06:08:46.357935 rule 0/0(match): block in on em0: 85.235.250.87.40108 > 85.209.118.130.22: Flags [S], seq 909787121, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 799486870 ecr 0], length 0 06:08:47.358590 rule 0/0(match): block in on em0: 85.235.250.87.40108 > 85.209.118.130.22: Flags [S], seq 909787121, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 799487870 ecr 0], length 0 06:08:49.557897 rule 0/0(match): block in on em0: 85.235.250.87.40108 > 85.209.118.130.22: Flags [S], seq 909787121, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 799490070 ecr 0], length 0 A wild guess as to the reason might be a race leading to some confusion over which em0 interface is which? Some more observations: - It didn't seem to happen with just one vnet jail when I tried narrowing it down. Enabling and starting three more made the problem occur almost instan= tly. - Rebooting with four jails plus the above ruleset enabled means never gett= ing any contact to the server at all (ie. the problem manifests from boot). - Results with two jails were less consistent. The number of jails/interface renames seem to play a role in whether or not the issue is triggered. - A "service jail restart" will trigger it almost instantly if it doesn't happen right away. - Renaming interfaces to something other than "em0" also works without any issues. I hope reproducing will be possible, I've included the jail.conf file for o= ne of the jails below: [tykling@nuc1 ~]$ cat /var/run/jail.syslog1_servers_bornhack_org.conf # Generated by rc.d/jail at 2022-01-06 08:19:08 syslog1_servers_bornhack_org { host.hostname =3D "syslog1.servers.bornhack.org"; path =3D "/usr/jails/syslog1.servers.bornhack.org"; vnet; vnet.interface =3D "epair2b"; exec.clean; exec.system_user =3D "root"; exec.jail_user =3D "root"; exec.prestart +=3D "ifconfig epair2a destroy 2>/dev/null || true && ifconfig epair2 create up && ifconfig epair2a up && ifconfig bridge1 addm epair2a up"; exec.start +=3D "/sbin/ifconfig epair2b name em0 && ifconfig em0 10.1.0.3/24 && ifconfig em0 inet6 2a09:94c4:55d1:76A0::3/64"; exec.start +=3D "route add -inet default 10.1.0.1"; exec.start +=3D "route add -inet6 default 2a09:94c4:55d1:76A0::1"; exec.poststop +=3D "ifconfig bridge1 deletem epair2a && ifconfig ep= air2a destroy"; exec.start +=3D "/bin/sh /etc/rc"; exec.stop =3D "/bin/sh /etc/rc.shutdown jail"; exec.consolelog =3D "/var/log/jail_syslog1_servers_bornhack_org_console.log"; mount.fstab =3D "/etc/fstab.syslog1_servers_bornhack_org"; allow.set_hostname =3D 0; allow.sysvipc =3D 0; enforce_statfs =3D "2"; } [tykling@nuc1 ~]$ The interesting sections I guess are: - in exec.prestart (on the host) where the epair interface is destroyed, recreated and added to a bridge - and in exec.start (inside the jail) where the interface is renamed to em0= and then configured with v4 and v6. I've included the whole thing in case it is useful to someone. I hope someone is able to reproduce, if not then I will have to narrow it d= own further, please let me know. I have run out of time for now. Thanks! :) --=20 You are receiving this mail because: You are the assignee for the bug.=
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?bug-260973-227>