From nobody Thu Jan 6 09:14:12 2022 X-Original-To: bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id B756D1933131 for ; Thu, 6 Jan 2022 09:14:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4JV0z835Njz3CL4 for ; Thu, 6 Jan 2022 09:14:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 4A2821D48C for ; Thu, 6 Jan 2022 09:14:12 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 2069ECG5088251 for ; Thu, 6 Jan 2022 09:14:12 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 2069ECw7088250 for bugs@FreeBSD.org; Thu, 6 Jan 2022 09:14:12 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: bugs@FreeBSD.org Subject: [Bug 260973] pf: firewall rules stop matching when vnet jails share interface names with the host Date: Thu, 06 Jan 2022 09:14:12 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 13.0-STABLE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: thomas@gibfest.dk X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: bugs@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-bugs@freebsd.org MIME-Version: 1.0 ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1641460452; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=PMOj2SGD1Or5v5NzD1Q0Vj7FZonDzwFac3GtBe94aH4=; b=akuPFNlT9JwKIo9YsTFLq2fEzD+Q4PDW96Pixzi8JmhoOSsSn/UBam4K58hNxvqjIjZ7bT 9AjQMrcrHPTPTXKbFC3Fr/gv83gKR+GPcbOVTLcMlP5GrDx7yysqwzLpTG/cTJiOo2WykL aefowLBK4iI6xtGfe1udOnYQOimbvx+Y40UfQnrA2rORDDN/4z5y2R0DOw0OoPn511s6WL OpUITnQ4Z9OR1mtKQzoqx28R2qm+MW4dBJZHwirubQS3hiW/iFAtM4KZRPZUBI0Nhx83+u q933ZPTPhjFleBI1NyN3rOochN2Ma2QC4j81IKMWDOEfaUYDUMgh1oOqtJ1ZWw== ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1641460452; a=rsa-sha256; cv=none; b=ymk2HRQvTx+UwKpizXpcLwtypXROF4HVz+oVSX+FmoUtUJEvepPIxBYiYXV6gq/4a+6iYV N0JS5VvNXuHwjDxRRaMOTGO3qADVrCUzHGW83yUCmrF7hfFujLEz5W2DAjJFjYn1pn8bpR nuxLmXPZDKpIF8HDjiBfFo4Mc5V25QN5NOrgl0cvitE2ItUxg3hbudtF0QcV1GFtP4zORE XZ/WWlK1NSG12TKfTS0ZfbWTIsTiB9/zVhFcOOtpziA2IKWSrv9yhpjfgTKyRD1q7Go1tN RalqpxvKZk6GkqGfvYmctkfu44Ma0PvGfOpAEyJWaEcNCQuybrURm2udziK5UQ== ARC-Authentication-Results: i=1; mx1.freebsd.org; none X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D260973 Bug ID: 260973 Summary: pf: firewall rules stop matching when vnet jails share interface names with the host Product: Base System Version: 13.0-STABLE Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: kern Assignee: bugs@FreeBSD.org Reporter: thomas@gibfest.dk Hello, I've been building a new vnet jailhost on 13 and I am hitting a weird issue where pf stops permitting traffic it clearly has rules to allow after interfaces inside vnet jails are renamed to the same name as the host inter= face with the pf rule. This is on FreeBSD nuc1.servers.bornhack.org 13.0-STABLE FreeBSD 13.0-STABL= E #1 stable/13-d208638c5: Wed Jan 5 13:32:08 UTC 2022=20=20=20=20 root@nuc1.servers.bornhack.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC am= d64 The complete ruleset is pretty complex but I've managed to cook it down to a few lines: [tykling@nuc1 ~]$ cat testpf.conf=20 block log all set skip on lo0 pass in quick on { em0 } proto { tcp } from { 85.235.250.87 } to { (em0) } = port { 22 } [tykling@nuc1 ~]$=20 The host has an em0 interface: [tykling@nuc1 ~]$ ifconfig em0 em0: flags=3D8863 metric 0 mtu 1500 =20=20=20=20=20=20=20 options=3D481049b ether 1c:69:7a:ab:fe:be inet 85.209.118.130/28 broadcast 85.209.118.143 inet6 fe80::1e69:7aff:feab:febe%em0/64 scopeid 0x1 inet6 2a09:94c4:55d1:7680::82/64 media: Ethernet autoselect (1000baseT ) status: active nd6 options=3D21 [tykling@nuc1 ~]$=20 The issue seems to be triggered by renaming epair interfaces inside vnet ja= ils to the same name as an interface on the host. The above pf ruleset works and keeps working if I don't start any vnet jail= s. It also keeps working if I start vnet jails but don't rename interfaces. It also keeps working if I start vnet jails but rename the interfaces to somet= hing other than em0. Existing states established before the issue happens keep working (I am wor= king remote via ssh on the server), but new states seem to just ignore the permit rule on em0, and the traffic gets blocked even though a rule should permit = it: 06:08:46.357935 rule 0/0(match): block in on em0: 85.235.250.87.40108 > 85.209.118.130.22: Flags [S], seq 909787121, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 799486870 ecr 0], length 0 06:08:47.358590 rule 0/0(match): block in on em0: 85.235.250.87.40108 > 85.209.118.130.22: Flags [S], seq 909787121, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 799487870 ecr 0], length 0 06:08:49.557897 rule 0/0(match): block in on em0: 85.235.250.87.40108 > 85.209.118.130.22: Flags [S], seq 909787121, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 799490070 ecr 0], length 0 A wild guess as to the reason might be a race leading to some confusion over which em0 interface is which? Some more observations: - It didn't seem to happen with just one vnet jail when I tried narrowing it down. Enabling and starting three more made the problem occur almost instan= tly. - Rebooting with four jails plus the above ruleset enabled means never gett= ing any contact to the server at all (ie. the problem manifests from boot). - Results with two jails were less consistent. The number of jails/interface renames seem to play a role in whether or not the issue is triggered. - A "service jail restart" will trigger it almost instantly if it doesn't happen right away. - Renaming interfaces to something other than "em0" also works without any issues. I hope reproducing will be possible, I've included the jail.conf file for o= ne of the jails below: [tykling@nuc1 ~]$ cat /var/run/jail.syslog1_servers_bornhack_org.conf # Generated by rc.d/jail at 2022-01-06 08:19:08 syslog1_servers_bornhack_org { host.hostname =3D "syslog1.servers.bornhack.org"; path =3D "/usr/jails/syslog1.servers.bornhack.org"; vnet; vnet.interface =3D "epair2b"; exec.clean; exec.system_user =3D "root"; exec.jail_user =3D "root"; exec.prestart +=3D "ifconfig epair2a destroy 2>/dev/null || true && ifconfig epair2 create up && ifconfig epair2a up && ifconfig bridge1 addm epair2a up"; exec.start +=3D "/sbin/ifconfig epair2b name em0 && ifconfig em0 10.1.0.3/24 && ifconfig em0 inet6 2a09:94c4:55d1:76A0::3/64"; exec.start +=3D "route add -inet default 10.1.0.1"; exec.start +=3D "route add -inet6 default 2a09:94c4:55d1:76A0::1"; exec.poststop +=3D "ifconfig bridge1 deletem epair2a && ifconfig ep= air2a destroy"; exec.start +=3D "/bin/sh /etc/rc"; exec.stop =3D "/bin/sh /etc/rc.shutdown jail"; exec.consolelog =3D "/var/log/jail_syslog1_servers_bornhack_org_console.log"; mount.fstab =3D "/etc/fstab.syslog1_servers_bornhack_org"; allow.set_hostname =3D 0; allow.sysvipc =3D 0; enforce_statfs =3D "2"; } [tykling@nuc1 ~]$ The interesting sections I guess are: - in exec.prestart (on the host) where the epair interface is destroyed, recreated and added to a bridge - and in exec.start (inside the jail) where the interface is renamed to em0= and then configured with v4 and v6. I've included the whole thing in case it is useful to someone. I hope someone is able to reproduce, if not then I will have to narrow it d= own further, please let me know. I have run out of time for now. Thanks! :) --=20 You are receiving this mail because: You are the assignee for the bug.=