From owner-freebsd-pf@freebsd.org Sun Oct 11 16:46:08 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 747543F85C5 for ; Sun, 11 Oct 2020 16:46:08 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4C8SPC4bJTz4Nxs for ; Sun, 11 Oct 2020 16:46:07 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id 783D63AD65; Sun, 11 Oct 2020 18:46:00 +0200 (CEST) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id TGTvRZ-LiPdG; Sun, 11 Oct 2020 18:45:59 +0200 (CEST) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id 787C73AD58; Sun, 11 Oct 2020 18:45:59 +0200 (CEST) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.local.incore (Postfix) with ESMTP id 5C57A105; Sun, 11 Oct 2020 18:45:59 +0200 (CEST) Message-ID: <5F8336C7.5020709@incore.de> Date: Sun, 11 Oct 2020 18:45:59 +0200 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: J David CC: freebsd-pf@freebsd.org Subject: Re: Packets passed by pf don't make it out? References: In-Reply-To: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4C8SPC4bJTz4Nxs X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of longwitz@incore.de designates 195.145.1.138 as permitted sender) smtp.mailfrom=longwitz@incore.de X-Spamd-Result: default: False [-1.48 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.02)[-1.018]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[incore.de]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.17)[-0.169]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-1.00)[-0.996]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:3320, ipnet:195.145.0.0/16, country:DE]; MIME_TRACE(0.00)[0:+]; MAILMAN_DEST(0.00)[freebsd-pf] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2020 16:46:08 -0000 > To investigate this issue, I've created a greatly simplified and > reproducible test case. The code is available at: > > https://github.com/jdavidlists/pfudpbug A similar setup works for me without any problems, so there may be something special in your environment. Please look at the output of "pfctl -vsn" on fb2 during your test. With "netstat -ss | grep drop" you can check for packets dropped by the kernel for what reason ever. It seems your routing table on fb2 is empty, please try to set a defaultroute, e.g.: "route add default 10.0.0.NN" with any NN. From owner-freebsd-pf@freebsd.org Sun Oct 11 18:00:12 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 529633F8FCE; Sun, 11 Oct 2020 18:00:12 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lj1-x243.google.com (mail-lj1-x243.google.com [IPv6:2a00:1450:4864:20::243]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C8V2g48f8z4RNN; Sun, 11 Oct 2020 18:00:11 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lj1-x243.google.com with SMTP id a23so13857303ljp.5; Sun, 11 Oct 2020 11:00:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=W+McX9gN4ZsVhAh1mAeYLyWPIfPcja4gwBZZhAdLQZs=; b=KmKEM9cmubBDN9mS9YESOA3LMuSJyQMiL3AqQJELAqHCG/7e53c/wnKN4hyzDrJ223 9t2ZdCFhu6EIeldGL80m5G85BVnECrHJBAhy38FFpJkKP0cQt9Q7F1zAzHJXrrXm2axj VRmqWfKr2B9JxQyul6Ypuxwb6p+o0NdVmRw7LeLwteYL2XtZx39iGNlbRmLOhAmKV5A9 OG0GfTwRwk6vEmkKfg6+lSS1XGy/zEP4nU645acB50dbIH0wJxxBsZV/ZPZn56g+S93T MguClpUciZOWTq3PWgYO8yg7wztTKbJncMio9Sdmt1WAfsbfCrEYJYe27/aUJPSNd1Wb EgOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=W+McX9gN4ZsVhAh1mAeYLyWPIfPcja4gwBZZhAdLQZs=; b=ZOY4pYKgepISHivLwHIHDv4YcKJ1ITC+6gZqy3cQKVDyTKv1LdjCkXZuVACFNhVuc1 m8BH1MRg+7G/ahzXmcadp63YoydjC2f08F0WFQ58YcFTSnA2ng2VZnDnHnlRqQ+2pT05 w+cTY813raHXf2pnviVUHwt6KBzfRVXojGhVSTEj14pTvVHI9XzLLWDTz95oc/Wo4TQp Gc2UMT73AaLhzXR12ToMG7vA3jnvxdjU0vHW4lgjyiF3T9EHMmWpOR1T6nT3opKZCBRe 1+PBe8MraGb6v/c+HmPQVUa6meCKJXHSQ2V5ZfRoWDOtf/36thcrPAfUtsxet8I28OTX djMQ== X-Gm-Message-State: AOAM532NDO6gBF2jWrzFnaC/aP+lgQrkfEI6sTyKxahlTa/QZmdIQmTq 2lnQFdnzPeL75JzKa+iTZfuhnfVDfg8YVhngYNh9ovJpkuI= X-Google-Smtp-Source: ABdhPJwM4kP8nEoFjJM59lYtZGbGGHc/Pig0JRVpoclaQgjlQlmVoOtU/dMH1Ek9NoplgdUyTlLN4nb/SrLAA6ZS/jg= X-Received: by 2002:a2e:88c2:: with SMTP id a2mr6464503ljk.438.1602439208952; Sun, 11 Oct 2020 11:00:08 -0700 (PDT) MIME-Version: 1.0 References: <5F8336C7.5020709@incore.de> In-Reply-To: <5F8336C7.5020709@incore.de> From: J David Date: Sun, 11 Oct 2020 13:59:58 -0400 Message-ID: Subject: Re: Packets passed by pf don't make it out? To: Andreas Longwitz Cc: freebsd-pf@freebsd.org, freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4C8V2g48f8z4RNN X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=KmKEM9cm; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2a00:1450:4864:20::243 as permitted sender) smtp.mailfrom=jdavidlists@gmail.com X-Spamd-Result: default: False [-2.45 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-0.99)[-0.990]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; NEURAL_HAM_LONG(-0.99)[-0.985]; RCVD_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::243:from]; NEURAL_HAM_SHORT(-0.48)[-0.476]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; MAILMAN_DEST(0.00)[freebsd-pf,freebsd-net] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2020 18:00:12 -0000 On Sun, Oct 11, 2020 at 12:46 PM Andreas Longwitz wrote: > Please look at the output of "pfctl -vsn" on fb2 during your test. > With "netstat -ss | grep drop" you can check for packets dropped by the > kernel for what reason ever. Here's the complete diff of the output from netstat -ss from before to after running the test: --- nss.pre 2020-10-11 17:10:19.932002000 +0000 +++ nss.post 2020-10-11 17:10:21.999823000 +0000 @@ -48,9 +48,9 @@ Packet drop statistics: Timeouts: ip: - 66578 total packets received + 66582 total packets received 66531 packets for this host - 16 packets forwarded + 17 packets forwarded 1 packet not forwardable 31675 packets sent from this host 10 packets sent with fabricated ip header No drops of any kind (nor anything else) recorded during the test. 4 packets in, 1 packet forwarded, which exactly matches the observed behavior of only one packet reaching the server. The results of "pfctl -vsn" are a bit more interesting, and also inconsistent. Before, after a full flush to zero states and counters: rdr inet proto udp from any to 172.16.0.0/12 port = 12345 -> 10.255.255.3 [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 1044 State Creations: 0 ] After: rdr inet proto udp from any to 172.16.0.0/12 port = 12345 -> 10.255.255.3 [ Evaluations: 4 Packets: 1 Bytes: 44 States: 1 ] [ Inserted: uid 0 pid 1044 State Creations: 4 ] So it says it created four states, but only matched one packet out of the four it evaluated. And it didn't create 4 states, either. "pfctl -s state" shows only 1: all udp 10.255.255.3:12345 (172.16.0.1:12345) <- 10.0.0.1:23456 NO_TRAFFIC:SINGLE and pflog0 reported all four packets as matching the pass rule which, important, is based on the destination address after redirection: 17:23:39.039641 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 17:23:39.039751 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 17:23:39.039769 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 17:23:39.039780 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 If I repeat the test, this happens: rdr inet proto udp from any to 172.16.0.0/12 port = 12345 -> 10.255.255.3 [ Evaluations: 7 Packets: 2 Bytes: 88 States: 1 ] [ Inserted: uid 0 pid 1044 State Creations: 7 ] But still just the one state: all udp 10.255.255.3:12345 (172.16.0.1:12345) <- 10.0.0.1:23456 NO_TRAFFIC:SINGLE But only three passes appear in pflog0: 17:29:19.857174 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 17:29:19.857193 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 17:29:19.857226 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 And, to confirm, the first packet from each of these tests did reach the server, the remaining three from each test did not. > A similar setup works for me without any problems, so there may be > something special in your environment. This has been tested on fresh installs of both FreeBSD 12.1 and 11.4 on both physical hardware and virtual machines including both Xeons and AMD Epyc. So it seems like most environmental factors have been controlled for. > It seems your routing table on fb2 is empty, please try to set a > defaultroute, e.g.: "route add default 10.0.0.NN" with any NN. fb2 does have a default route, it is obtained from DHCP on the first interface. But that is not relevant; the client machine (fb1) is directly connected to fb2's second interface, and the server (fb3) is directly connected to fb2's third interface. No additional routes are necessary for this test, and the default route is never consulted. Perhaps there's some detail of the scenario that I have omitted without which it's not clear? Thanks! From owner-freebsd-pf@freebsd.org Mon Oct 12 21:48:18 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A3C8B429382 for ; Mon, 12 Oct 2020 21:48:18 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from dss.incore.de (dss.incore.de [195.145.1.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4C9C3P5zGXz3Sxw for ; Mon, 12 Oct 2020 21:48:17 +0000 (UTC) (envelope-from longwitz@incore.de) Received: from inetmail.dmz (inetmail.dmz [10.3.0.3]) by dss.incore.de (Postfix) with ESMTP id C7FA73AD9F; Mon, 12 Oct 2020 23:48:09 +0200 (CEST) X-Virus-Scanned: amavisd-new at incore.de Received: from dss.incore.de ([10.3.0.3]) by inetmail.dmz (inetmail.dmz [10.3.0.3]) (amavisd-new, port 10024) with LMTP id Ws_DP83_77qj; Mon, 12 Oct 2020 23:48:09 +0200 (CEST) Received: from mail.local.incore (fwintern.dmz [10.0.0.253]) by dss.incore.de (Postfix) with ESMTP id 1E2DB3AD7F; Mon, 12 Oct 2020 23:48:09 +0200 (CEST) Received: from bsdmhs.longwitz (unknown [192.168.99.6]) by mail.local.incore (Postfix) with ESMTP id 0319D11F; Mon, 12 Oct 2020 23:48:08 +0200 (CEST) Message-ID: <5F84CF18.1040905@incore.de> Date: Mon, 12 Oct 2020 23:48:08 +0200 From: Andreas Longwitz User-Agent: Thunderbird 2.0.0.19 (X11/20090113) MIME-Version: 1.0 To: J David CC: freebsd-pf@freebsd.org Subject: Re: Packets passed by pf don't make it out? References: <5F8336C7.5020709@incore.de> In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4C9C3P5zGXz3Sxw X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of longwitz@incore.de designates 195.145.1.138 as permitted sender) smtp.mailfrom=longwitz@incore.de X-Spamd-Result: default: False [-1.49 / 15.00]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-0.97)[-0.971]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[incore.de]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.25)[-0.251]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.97)[-0.972]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:3320, ipnet:195.145.0.0/16, country:DE]; SUBJECT_ENDS_QUESTION(1.00)[]; MAILMAN_DEST(0.00)[freebsd-pf] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Oct 2020 21:48:18 -0000 Hello, now I can confirm (on FreeBSD 10 Stable) what you see on fb2 when your program udp_client is running on fb1. pf creates a state for the first packet only, for the other packets pf failes to create a state with messages like pf: stack key attach failed on re0: UDP in wire: 192.168.14.10:23456 172.16.0.2:12345 stack: 192.168.14.10:23456 192.168.14.100:12345 1:0, existing: UDP in wire: 192.168.14.10:23456 172.16.0.1:12345 stack: 192.168.14.10:23456 192.168.14.100:12345 1:0 pf gives this messages in debug mode (pfctl -x loud). I do not know if we see a bug in pf or if your program udp_client does something illegal, I think Kristof can tell us. Regards Andreas From owner-freebsd-pf@freebsd.org Tue Oct 13 16:07:45 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3C2D344559D for ; Tue, 13 Oct 2020 16:07:45 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from wout3-smtp.messagingengine.com (wout3-smtp.messagingengine.com [64.147.123.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4C9gS02x1bz3dDn for ; Tue, 13 Oct 2020 16:07:44 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.west.internal (Postfix) with ESMTP id C3221B27 for ; Tue, 13 Oct 2020 12:07:41 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Tue, 13 Oct 2020 12:07:41 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:mime-version:content-type; s= fm3; bh=MH/VoUtQ+TuUgi+2He6/9woFqZqhk4yZC6FbMkadMY0=; b=NuqllaSI aS7jKtAyJ9fDKTLOz/7ihZQvPlLkSOSKYbiqWQcZFNf/GXGEf6enLjbQ9847tcNA s5XZwzNWhmDalfg/Fz7guSzzpbKPSwPOVw0g4JlA5d8QAwNlPSVxInnY5y/7+LBQ UH/3xcr9+TofCDkYeGQJXXhH/nne0Lj0hMqyhHhMgoC/loPdWY/QqhE9euWhD1vp VRuRa7qCN0ZGLFHGKM/r3wPB3yWccBe0H5Tmz57Usnz/rX70kjMSZfCcxkJls9Ql 7k4MCMT2mxbu9edxNtLt8MoDgQFmBhvMbNQPcL7jAu4FT4bQoeUXkEZGJYhQ6P32 fYdN59wiYXsfog== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm1; bh=MH/VoUtQ+TuUgi+2He6/9woFqZqhk 4yZC6FbMkadMY0=; b=lwD3LlATG79NxZsysyPLh+Zqj4lNdZTqcco5JCyrZ2PSB tm9bbpboVbltxTKTdnvqTtR4QqSBPUxkstpHVj9Oeg8SSFgSok1mob/e0w/hEI40 GaSTpiHJ2o7XCmw913epWS3Ep0Ej4yzJFHF/cQRpUp1ngax5bIOp+tVy0ryPVQw/ Y0nHEg0GxrTGMNvqsfItTogkcdXKvhmQo4LyyT3kP+wpdGDwbUunbkZUasLnOeTV K0qMaaVjfOzVj2oO12xODq/mhRzcYeiRPBe/Oy85RX/EatRv7m+PHdWsbCBSFRoX SMpRMEdRMb+HoyFoy+Es43cjlkR5UbuwOZYujpOaA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedrheelgdelkecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfggtggusehgtderredttd dvnecuhfhrohhmpehtvggthhdqlhhishhtshcuoehtvggthhdqlhhishhtshesiiihgihs thdrnhgvtheqnecuggftrfgrthhtvghrnhepvefghffftdefkeelleehtdejledvhfdvge eijeevfffguddvhfetgeejueejueeinecukfhppeekvddrjedtrdeluddruddtvdenucev lhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehtvggthhdqlh hishhtshesiiihgihsthdrnhgvth X-ME-Proxy: Received: from rpi4.gilescoppice.lan (axs-0-ipv4.zyxst.net [82.70.91.102]) by mail.messagingengine.com (Postfix) with ESMTPA id EE4BA328005A for ; Tue, 13 Oct 2020 12:07:40 -0400 (EDT) Date: Tue, 13 Oct 2020 17:07:38 +0100 From: tech-lists To: freebsd-pf@freebsd.org Subject: pf and tap(4) interfaces Message-ID: <20201013160738.GD30207@rpi4.gilescoppice.lan> Mail-Followup-To: freebsd-pf@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="OROCMA9jn6tkzFBc" Content-Disposition: inline X-Rspamd-Queue-Id: 4C9gS02x1bz3dDn X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm3 header.b=NuqllaSI; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=lwD3LlAT; dmarc=none; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 64.147.123.19 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-4.74 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm3,messagingengine.com:s=fm1]; NEURAL_HAM_MEDIUM(-0.98)[-0.977]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[64.147.123.19:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.19]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-pf@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.96)[-0.956]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[zyxst.net]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-0.11)[-0.106]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; MAILMAN_DEST(0.00)[freebsd-pf]; RCVD_IN_DNSWL_LOW(-0.10)[64.147.123.19:from] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2020 16:07:45 -0000 --OROCMA9jn6tkzFBc Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi, Is it possible to have a ruleset allowing unfiltered access to a tap interface, but filtered on the real interface it's bridged to? Let's say there are these: ext_if=3D"ix0" # real external ip, on a /29=20 int_if=3D"igb0" # internal ip 10.0.0.2/8 tap_if=3D"tap0" # this services a vm on this machine, also with a real ip bridge0 has ix0 and tap0 as members tap0 needs unfiltered access. it has its own firewall. ix0 wants to block everything apart from ssh. This doesn't work (it blocks everything apart from ssh to the vm as well): [snip] block all pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 22 pass in quick on $tap_if inet proto tcp from any to ($tap_if) thanks, --=20 J. --OROCMA9jn6tkzFBc Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAl+F0MEACgkQs8o7QhFz NAVylhAAj+Xyug+6+kJGzdE+7Df+hOBoryXIHoFk2REDIXPgEUx1nZACfDY5FFv9 J+esUISzz6tdAeBCP0ODGISwZ0Gk4p5AjR3G0zF7bGt0W4iUvOfAKec9NduVMjS+ ZLFHSFhQtORpIlPfIzv7edJLK9lkhsczJ7H4rxnGRfRBych7KFb/3JfksFVbSdoQ UybS9b/282oOHFJZa8TWvDj4j17WQg/7a+TRyPRItoZ47I1tdOVRWCPEW4Yo4C6b H8bs/irY3C/bopXfEpz28wi6HpflzdntpWpYp/ClSNHT+TnU8McpH8uNhaPmvPmO d9V2oVUmYJ9oHbdfRL+IEWw2I7eQtB/Wy6W99CZK3NPEzIGCZ783/Gg5qAAimp7G 6NjTqvdJ4/RQDimWXr5TboFbDiTYZ1XoCrLBVlw86/WiGBJnAJsCE7GkxUP8rFst RVOFJoYtR0BhRn/Cqe3ZZl8XeKFmzwVQL3GTQKHWhOarXyWo+2OkrMFHNtae5pC/ M+/dx8Nn1ssjikaQ8KPlQl8cVcRrTtw9hN7EgH02vcLOTQUX4D01eBHsx2h5qoMS Buw7vw3eg0PDdDz6Snbs4gAQVSOMbe0EfX8i/TGiC+KuOMU++VtZCxbexYcnafE3 lyeoDEPmuaEDZeDTxtHYNm2mZKVtWiDgKt7wYGPuc6h1udKhOso= =eU3l -----END PGP SIGNATURE----- --OROCMA9jn6tkzFBc-- From owner-freebsd-pf@freebsd.org Tue Oct 13 17:26:39 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0AF164469C0 for ; Tue, 13 Oct 2020 17:26:39 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Received: from mail.flex-it.com.ua (mail.flex-it.com.ua [193.239.74.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4C9jC208gRz3yDK for ; Tue, 13 Oct 2020 17:26:37 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Received: from [91.243.208.110] (helo=thinkpad.flex-it.com.ua) by mail.flex-it.com.ua with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.94 (FreeBSD)) (envelope-from ) id 1kSO4C-000Mb0-Qm; Tue, 13 Oct 2020 20:26:28 +0300 Subject: Re: pf and tap(4) interfaces To: freebsd-pf@freebsd.org References: <20201013160738.GD30207@rpi4.gilescoppice.lan> From: Oleksandr Kryvulia Message-ID: <41851719-8e17-d5d6-4abb-0c4221df70ef@shurik.kiev.ua> Date: Tue, 13 Oct 2020 20:26:23 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.3.1 MIME-Version: 1.0 In-Reply-To: <20201013160738.GD30207@rpi4.gilescoppice.lan> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit Content-Language: ru X-Rspamd-Queue-Id: 4C9jC208gRz3yDK X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of shuriku@shurik.kiev.ua designates 193.239.74.7 as permitted sender) smtp.mailfrom=shuriku@shurik.kiev.ua X-Spamd-Result: default: False [-2.24 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-0.95)[-0.947]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[shurik.kiev.ua]; RECEIVED_SPAMHAUS_PBL(0.00)[91.243.208.110:received]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.14)[-0.143]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.85)[-0.850]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:35297, ipnet:193.239.72.0/22, country:UA]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-pf]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2020 17:26:39 -0000 On 13.10.20 19:07, tech-lists wrote: > Hi, > > Is it possible to have a ruleset allowing unfiltered access to a tap > interface, but filtered on the real interface it's bridged to? > > Let's say there are these: > > ext_if="ix0" # real external ip, on a /29 int_if="igb0" # internal ip > 10.0.0.2/8 > tap_if="tap0" # this services a vm on this machine, also with a real ip > > bridge0 has ix0 and tap0 as members > > tap0 needs unfiltered access. it has its own firewall. > ix0 wants to block everything apart from ssh. > > This doesn't work (it blocks everything apart from ssh to the vm as > well): > > [snip] > block all > pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 22 > pass in quick on $tap_if inet proto tcp from any to ($tap_if) > > thanks, External traffic to your tap interface arrives through ix0. So you need to change a third rule: block all pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 22 pass in quick on $ext_if inet proto tcp from any to ($tap_if) Also check net.link.bridge.pfil_member=1 As for me I prefer to have  all IPs and filter it on bridge interface and not on members. From owner-freebsd-pf@freebsd.org Tue Oct 13 21:35:36 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1A05944C122 for ; Tue, 13 Oct 2020 21:35:36 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C9pkH6z5Cz4JXP; Tue, 13 Oct 2020 21:35:35 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id B866B1A929; Tue, 13 Oct 2020 21:35:35 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id CB6EF38795; Tue, 13 Oct 2020 23:35:33 +0200 (CEST) From: "Kristof Provost" To: "Andreas Longwitz" Cc: "J David" , freebsd-pf@freebsd.org Subject: Re: Packets passed by pf don't make it out? Date: Tue, 13 Oct 2020 23:35:29 +0200 X-Mailer: MailMate (1.13.2r5673) Message-ID: <0072D8A9-6ACE-47D0-AE94-124C4F955735@FreeBSD.org> In-Reply-To: <5F84CF18.1040905@incore.de> References: <5F8336C7.5020709@incore.de> <5F84CF18.1040905@incore.de> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 13 Oct 2020 21:35:36 -0000 On 12 Oct 2020, at 23:48, Andreas Longwitz wrote: > Hello, > > now I can confirm (on FreeBSD 10 Stable) what you see on fb2 when your > program udp_client is running on fb1. pf creates a state for the first > packet only, for the other packets pf failes to create a state with > messages like > > pf: stack key attach failed on re0: UDP in wire: 192.168.14.10:23456 > 172.16.0.2:12345 stack: 192.168.14.10:23456 > 192.168.14.100:12345 1:0, existing: UDP in wire: 192.168.14.10:23456 > 172.16.0.1:12345 stack: 192.168.14.10:23456 192.168.14.100:12345 1:0 > > pf gives this messages in debug mode (pfctl -x loud). > > I do not know if we see a bug in pf or if your program udp_client does > something illegal, I think Kristof can tell us. > Your confidence is both flattering and misplaced :) I think I can reproduce the problem on CURRENT and with VNET jails, which is convenient. I see the same ‘stack key attach failed’ error message. My current thinking is that we’re hitting a state collision, because post-RDR our connection information is the same (192.168.14.10:23456 192.168.14.100:12345). That means we can’t create a new state, and the packet gets dropped. It’s a little unusual for a client to keep re-using src ports like that, but it’s not actually wrong. I’m not sure how we can fix this. Best, Kristof From owner-freebsd-pf@freebsd.org Wed Oct 14 01:37:45 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 01BE54286BB for ; Wed, 14 Oct 2020 01:37:45 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4C9w5g6cHDz4WXt for ; Wed, 14 Oct 2020 01:37:43 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 08B175C0109 for ; Tue, 13 Oct 2020 21:37:43 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute1.internal (MEProxy); Tue, 13 Oct 2020 21:37:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm3; bh=frL1ORrJjd0xt7JA5aRlrZUZS/m yEPn3OyXM+Syis7Y=; b=JpMEGAF5D0bfN7LGs1+1kFdtRAVHC8vXJRxFPu2oMRd fuBV7qe7b7jUEvJb1uGZncU1WzxMdzLX/2b9eRNPR9bwklwQmoAtp3E/kRNrYz6x 7R7yuyHtKK/HjOrl81J+nXbqOLYBNJ7XY0jy/9nPJ9JcBduQycwwQ7M2cGKGieWT x65KsKPgcxkG1OgoEJjULHgHbOKsYPani4lcXLjGrUotlXc9EK1bf5MMTfHEz3oo vw3kHBN58ZwYQCVdjgFTRC4hUAg5Yr+9MSuPVvIT2RFoPWNa0W0+x29afUCp3NTN lguoP3p7MH7L2xbBZ9unxs3DU0xMiVPiljCTSWwHcIQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm1; bh=frL1OR rJjd0xt7JA5aRlrZUZS/myEPn3OyXM+Syis7Y=; b=TiaqfahvMWBCO9MqSyRxtO rPzvz0DHuSMO74vqozKhELnkAzTvVyfPj4VkKkejk5GpYJlDtfkMuIU+atyqAMY8 Bct5v+bBupsfKNZEP9s2BViDZaCsXS0oDLLut6nuraP8KFvJbGGEM1vh3Yq+zybX ehuIXNFqoiFj5AKR+/AgOFdEbZ6hH+zAVbzaDxheY9cwBxvscIcbiE3GK5C9t/F0 ce47z9Ldy4tJAWnv2BVGUy7pSLXT9iKmhNjA/PvopBd9YxRNcT4+v4h3wJVPZQKP HKonDJd/MfnYXF1keCs3CxZfJOz7WpAgggosn7vUjJykOBcfwSvuCNlPShuRe/Sw == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedujedriedtgdeglecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjsehgtderre dttddunecuhfhrohhmpehtvggthhdqlhhishhtshcuoehtvggthhdqlhhishhtshesiiih gihsthdrnhgvtheqnecuggftrfgrthhtvghrnhepveffueejgfeghfefkedutedvhedthe dugfffieegfeevfeehgedvvddtheffleejnecukfhppeekvddrjedtrdeluddruddtvden ucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehtvggthh dqlhhishhtshesiiihgihsthdrnhgvth X-ME-Proxy: Received: from rpi4.gilescoppice.lan (axs-0-ipv4.zyxst.net [82.70.91.102]) by mail.messagingengine.com (Postfix) with ESMTPA id 67225306467E for ; Tue, 13 Oct 2020 21:37:42 -0400 (EDT) Date: Wed, 14 Oct 2020 02:37:40 +0100 From: tech-lists To: freebsd-pf@freebsd.org Subject: Re: pf and tap(4) interfaces Message-ID: <20201014013740.GA69661@rpi4.gilescoppice.lan> Mail-Followup-To: freebsd-pf@freebsd.org References: <20201013160738.GD30207@rpi4.gilescoppice.lan> <41851719-8e17-d5d6-4abb-0c4221df70ef@shurik.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="IJpNTDwzlM2Ie8A6" Content-Disposition: inline In-Reply-To: <41851719-8e17-d5d6-4abb-0c4221df70ef@shurik.kiev.ua> X-Rspamd-Queue-Id: 4C9w5g6cHDz4WXt X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm3 header.b=JpMEGAF5; dkim=pass header.d=messagingengine.com header.s=fm1 header.b=Tiaqfahv; dmarc=none; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 66.111.4.27 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-5.03 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm3,messagingengine.com:s=fm1]; NEURAL_HAM_MEDIUM(-1.03)[-1.026]; FROM_HAS_DN(0.00)[]; RWL_MAILSPIKE_GOOD(0.00)[66.111.4.27:from]; TO_MATCH_ENVRCPT_ALL(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.27]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-pf@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.04)[-1.040]; RCVD_COUNT_THREE(0.00)[4]; DMARC_NA(0.00)[zyxst.net]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; NEURAL_HAM_SHORT(-0.26)[-0.264]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; MAILMAN_DEST(0.00)[freebsd-pf]; RCVD_IN_DNSWL_LOW(-0.10)[66.111.4.27:from] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 01:37:45 -0000 --IJpNTDwzlM2Ie8A6 Content-Type: text/plain; charset=iso-8859-1; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, On Tue, Oct 13, 2020 at 08:26:23PM +0300, Oleksandr Kryvulia wrote: >> >> [snip] >> block all >> pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 22 >> pass in quick on $tap_if inet proto tcp from any to ($tap_if) >> >> thanks, > >External traffic to your tap interface arrives through ix0. So you need >to change a third rule: > >block all >pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 22 >pass in quick on $ext_if inet proto tcp from any to ($tap_if) > >Also check net.link.bridge.pfil_member=3D1 Unfortunately this suggestion didn't work for me, but thanks for suggesting. It ends up blocking everything to the vm.=20 I should also have mentioned my full context originally:=20 What I have in this instance is a freebsd host running a freebsd=20 vm through bhyve. Both the host and the vm have real ips.=20 The vm wants full access as it has its own pf within itself.=20 The host wants ssh open and no more. I can lock down the ssh server on the host with sshd_config plus some additions to sysctl.conf, without involving pf at all. I just wondered if I can do it with pf on the=20 host. I'm surprised there's no mention of this type of config in=20 the handbook. I would have thought it was common? I've also tried set skip on $tap_if to no effect, in that if I apply this (but have the allow only ssh to $ext_if), then I can't access the vm on the vm's open ports. Clearly I'm doing something wrong. >As for me I prefer to have=A0 all IPs and filter it on bridge interface and >not on members. How do you do that? It's probably (if I understand correctly) not for me because I'm using bhyve, and $ext_if and $tap_if are both members and they need different access. But I'd be interested how you're filtering on the bridge interface. --=20 J. --IJpNTDwzlM2Ie8A6 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAl+GVlsACgkQs8o7QhFz NAWc+Q/8D/rwjrPS90qb6Qc2y5ybUC+La2Hnbge5xr5NgwJHk+oRaG6EkxhcHCND CmeyZ+btvEN8v3c0wAnAUYN8Fj7qroN14/odUcHLNes8Wro268DqMQVd/0Jvd+5Y PWO7sI8bcjzl4ePCO9ibftNX4gzH2fuphK5cTmvflpsdstp2+LVhTezJGHJS/b0g 4+mKHlv5kb8tCMZwc3jkgfCoY5wVcmtfprYJp/A36SEUkwz7Y9dLnuFAezHj9hcJ h1HjWMvxccfZM4qccyK4jFOPfyes97CYAeZq8zVO5Hn0feEbpf42SaFG6SdGWa2i RJgt3NZY8q/gg2guDHYoi5eGMY4hcD/rrQMOKbhu/5ijWfp3NrvZDMHNGZ3AIHbk 8p/RdKVXl5ycV5acb5xU0RpupLZaaC7K7xlZbcSK3y7XEKIUpnlyQJdp/6XJWTeA JisEND17iSkL/0Itqsl6Ch5lK/rq5p9/BUyFdDKHEGrreyJ6jEr7tMTwxHeXsVWq UgpSSQ8CvxFINj2Mqggfw2/OCiAUpNFJf+0M4hsyKY6kshdIMCloKtTCOxKo4QWG wI2e5vzc408ghAVZAVmALCEtr7Jt1VBgeyyQypBN7Kz5HYnfKbmWurIWy3lAXzJd dnRAno7O/adx4w+wcYDTu4U94wP+WWv2zOxowPdXvv6nv2DiKr4= =CsxS -----END PGP SIGNATURE----- --IJpNTDwzlM2Ie8A6-- From owner-freebsd-pf@freebsd.org Wed Oct 14 06:55:29 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2D95542EF44 for ; Wed, 14 Oct 2020 06:55:29 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Received: from mail.flex-it.com.ua (mail.flex-it.com.ua [193.239.74.7]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CB38J0Zdkz4mRN for ; Wed, 14 Oct 2020 06:55:27 +0000 (UTC) (envelope-from shuriku@shurik.kiev.ua) Received: from [91.243.208.110] (helo=thinkpad.flex-it.com.ua) by mail.flex-it.com.ua with esmtpsa (TLS1.3) tls TLS_AES_128_GCM_SHA256 (Exim 4.94 (FreeBSD)) (envelope-from ) id 1kSah2-0003xa-Lu for freebsd-pf@freebsd.org; Wed, 14 Oct 2020 09:55:24 +0300 Subject: Re: pf and tap(4) interfaces To: freebsd-pf@freebsd.org References: <20201013160738.GD30207@rpi4.gilescoppice.lan> <41851719-8e17-d5d6-4abb-0c4221df70ef@shurik.kiev.ua> <20201014013740.GA69661@rpi4.gilescoppice.lan> From: Oleksandr Kryvulia Message-ID: <785bef6b-3500-7f54-3d25-a0700e0b9678@shurik.kiev.ua> Date: Wed, 14 Oct 2020 09:55:19 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:78.0) Gecko/20100101 Thunderbird/78.3.1 MIME-Version: 1.0 In-Reply-To: <20201014013740.GA69661@rpi4.gilescoppice.lan> Content-Language: en-US X-Rspamd-Queue-Id: 4CB38J0Zdkz4mRN X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of shuriku@shurik.kiev.ua designates 193.239.74.7 as permitted sender) smtp.mailfrom=shuriku@shurik.kiev.ua X-Spamd-Result: default: False [-0.89 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+mx:c]; TO_DN_NONE(0.00)[]; NEURAL_HAM_SHORT(-0.14)[-0.137]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:35297, ipnet:193.239.72.0/22, country:UA]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[91.243.208.110:received]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.04)[-0.039]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.41)[-0.411]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[shurik.kiev.ua]; RCPT_COUNT_ONE(0.00)[1]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-pf] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 06:55:29 -0000 On 14.10.20 04:37, tech-lists wrote: > > Hello, > > On Tue, Oct 13, 2020 at 08:26:23PM +0300, Oleksandr Kryvulia wrote: >>> >>> [snip] >>> block all >>> pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 22 >>> pass in quick on $tap_if inet proto tcp from any to ($tap_if) >>> >>> thanks, >> >> External traffic to your tap interface arrives through ix0. So you need >> to change a third rule: >> >> block all >> pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 22 >> pass in quick on $ext_if inet proto tcp from any to ($tap_if) >> >> Also check net.link.bridge.pfil_member=1 > > Unfortunately this suggestion didn't work for me, but thanks for > suggesting. It ends up blocking everything to the vm. > I should also have mentioned my full context originally: What I have > in this instance is a freebsd host running a freebsd vm through bhyve. > Both the host and the vm have real ips. The vm wants full access as it > has its own pf within itself. > The host wants ssh open and no more. I can lock down the ssh server on > the host with sshd_config plus some additions to sysctl.conf, without > involving pf at all. I just wondered if I can do it with pf on the > host. I'm surprised there's no mention of this type of config in the > handbook. I would have thought it was common? > > I've also tried > set skip on $tap_if > > to no effect, in that if I apply this (but have the allow only ssh to > $ext_if), then I can't access the vm on the vm's open ports. Clearly I'm > doing something wrong. > >> As for me I prefer to have  all IPs and filter it on bridge interface >> and >> not on members. > > How do you do that? It's probably (if I understand correctly) not for me > because I'm using bhyve, and $ext_if and $tap_if are both members and > they need different access. But I'd be interested how you're filtering > on the bridge interface. > Your VM IP is assigned on VM's internal interface, not on tap0. This rule may does not make any sense: pass in quick on $ext_if inet proto tcp from any to ($tap_if) Try to try to specify real VM IP instead of interface name: pass in quick on $ext_if inet proto tcp from any to a.b.c.d In my setup for example, ifconfig bridge0 create addm ix0 addm tap0 ifconfig bridge0 a.b.c.d/24 (your external ip) Assign your VM ip (1.2.3.4) on VM internel interface (not on tap0). Set in /etc/sysctl.conf and apply it: net.link.bridge.pfil_bridge=1 net.link.bridge.pfil_member=1 net.link.bridge.pfil_local_phys=1 Your pf rules will look like this: ext_if="bridge0" vm_ip="1.2.3.4" block all pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 22 pass in quick on $ext_if inet proto tcp from any to $vm_ip or pass in quick on $ext_if inet proto tcp from any to ($ext_if) port 22 block in quick on $ext_if inet proto tcp from any to ($ext_if) pass in quick on $ext_if From owner-freebsd-pf@freebsd.org Wed Oct 14 16:53:08 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AF1B443E138 for ; Wed, 14 Oct 2020 16:53:08 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lf1-x141.google.com (mail-lf1-x141.google.com [IPv6:2a00:1450:4864:20::141]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CBJPw2gzvz4Fdl; Wed, 14 Oct 2020 16:53:08 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lf1-x141.google.com with SMTP id v6so164307lfa.13; Wed, 14 Oct 2020 09:53:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=+BSzQt6mi4HlpVhj9HaQiyFtMmc9RwGcsY8hcksuqcw=; b=cP7ExcPoy2meoJmRbUvaqVmMpSJepOCgf5ViEmfxqYDVbENVdjDW1iC5WW5C8usOJM /sHhktMDD6mbukXD7fqL6UuzNa93xEE/y5+w3EB4VrUxEYUDVtN2ZXQ0jDqp0aJmvB4x jaZMOmaj7nD703CHTC6z8Pi1iQjGl5AAuEpYotsPGDyzDCtWkpwjH669JeldITGZ0fVu EI/1LKn/P5YIFHwqiTqyODAUl+mh/CICD8Hkb1yBiMl1us/nZwbg08zeeSVrvW9Kx+S1 3+sjObSVjyk1ziIZbAZgaxnITI6kAaPg0vc3MG/dLdg8nOaan4Z9XDRZdczVOdnJd0u7 VMVA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=+BSzQt6mi4HlpVhj9HaQiyFtMmc9RwGcsY8hcksuqcw=; b=lcW+EPpuM40OeN2DBwGDIQnCiLP3F57T4ljmDFU2Fy9V268TTHbtLMRC+TDvwNzSMo DZSgU8SXWFozRrSu268geXV92bi8oXYx8P/3fFj3YkxOz6dpQGlkSMwmxCmA/rBIqlv6 EqmEfhXkxPKIyF4Mj6rKXs65eywFg/OFAklwTQPL44CVFduIVopH+hk0GYx4jmrYbPr2 SX/tWF3ewgHouzg8Qujd1ZqVg6W69G39sHGMWy7nqRw79/ZP3QOFy6V2h2B3ujjMUkNt mY2w16wVzWxswRE9LvzvEQBXHLfqad5xWtaaE9mc7GC4nK5Zdq7+9QVw6jhL3q06xvbI 4HPw== X-Gm-Message-State: AOAM532Ak/UabcACJB1PKTucJMq2zeAi9mklRwBaBxqh4Gq36rp1ab55 Gwc5ERezJMUFg5coUwBJaoMh+L89Yx9HiaJ+gZryESnM7NI= X-Google-Smtp-Source: ABdhPJxGUl9HpwMbpYgXljbDi8E/JS7zbiY8srjAkg7lmXcZlvqZb+Qw3rdjMAYK5IA6bdSnchtToNtS0kzmwqavKOs= X-Received: by 2002:ac2:5048:: with SMTP id a8mr82417lfm.60.1602694386328; Wed, 14 Oct 2020 09:53:06 -0700 (PDT) MIME-Version: 1.0 References: <5F8336C7.5020709@incore.de> <5F84CF18.1040905@incore.de> <0072D8A9-6ACE-47D0-AE94-124C4F955735@FreeBSD.org> In-Reply-To: <0072D8A9-6ACE-47D0-AE94-124C4F955735@FreeBSD.org> From: J David Date: Wed, 14 Oct 2020 12:52:55 -0400 Message-ID: Subject: Re: Packets passed by pf don't make it out? To: Kristof Provost Cc: Andreas Longwitz , freebsd-pf@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4CBJPw2gzvz4Fdl X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; REPLY(-4.00)[] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 16:53:08 -0000 On 12 Oct 2020, at 23:48, Andreas Longwitz wrote: > pf gives this messages in debug mode (pfctl -x loud). Yes, with that setting I'm also seeing those messages. On Tue, Oct 13, 2020 at 5:35 PM Kristof Provost wrote: > I see the same =E2=80=98stack key attach failed=E2=80=99 error message. M= y current > thinking is that we=E2=80=99re hitting a state collision, because post-RD= R our > connection information is the same (192.168.14.10:23456 > 192.168.14.100:12345). That means we can=E2=80=99t create a new state, an= d the > packet gets dropped. This is probably a dumb question because I know less than nothing about pf internals, but why wouldn't it match the existing state? Yes, the original destination was different before the redirect, but after the redirect they're going from the same host/port to the same host/port. What's the definition of state equality, and does that satisfy it? In order to say that they're not the same state, wouldn't the packets have to bear some property that survived the redirect that would distinguish them as state-unequal? If it did, would/should that property not then be part of the computation of state entry uniqueness? Like I said, probably a dumb question. > It=E2=80=99s a little unusual for a client to keep re-using src ports lik= e > that, but it=E2=80=99s not actually wrong. After further study, I think the DNS validators used by Let's Encrypt to control TLS certificate issuance may also be doing this, which would make it a little more urgent. (But that bears confirmation.) Thanks! From owner-freebsd-pf@freebsd.org Wed Oct 14 17:59:17 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E84E743F842 for ; Wed, 14 Oct 2020 17:59:17 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CBKtF5nKBz4JZ2; Wed, 14 Oct 2020 17:59:17 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 9118D237E8; Wed, 14 Oct 2020 17:59:17 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 0A81C3B66A; Wed, 14 Oct 2020 19:59:16 +0200 (CEST) From: "Kristof Provost" To: "J David" Cc: "Andreas Longwitz" , freebsd-pf@freebsd.org Subject: Re: Packets passed by pf don't make it out? Date: Wed, 14 Oct 2020 19:59:15 +0200 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: References: <5F8336C7.5020709@incore.de> <5F84CF18.1040905@incore.de> <0072D8A9-6ACE-47D0-AE94-124C4F955735@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed; markup=markdown Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 17:59:18 -0000 On 14 Oct 2020, at 18:52, J David wrote: > On 12 Oct 2020, at 23:48, Andreas Longwitz wrote: >> pf gives this messages in debug mode (pfctl -x loud). > > Yes, with that setting I'm also seeing those messages. > > On Tue, Oct 13, 2020 at 5:35 PM Kristof Provost > wrote: >> I see the same ‘stack key attach failed’ error message. My >> current >> thinking is that we’re hitting a state collision, because post-RDR >> our >> connection information is the same (192.168.14.10:23456 >> 192.168.14.100:12345). That means we can’t create a new state, and >> the >> packet gets dropped. > > This is probably a dumb question because I know less than nothing > about pf internals, but why wouldn't it match the existing state? > “It’s complicated”. In essence, pf tracks both the pre- and post-translation tuple, so what we’re seeing here is one of those conflicting with an existing session and that’s causing the failure. There’s good reason to do this, as we have to be able to match state on both the pre-translation side (when processing LAN -> WAN traffic) and post-translation (WAN -> LAN). Best regards, Kristof From owner-freebsd-pf@freebsd.org Wed Oct 14 19:16:35 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0BC24440A8C for ; Wed, 14 Oct 2020 19:16:35 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lj1-x22b.google.com (mail-lj1-x22b.google.com [IPv6:2a00:1450:4864:20::22b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CBMbQ4XGDz4N0q; Wed, 14 Oct 2020 19:16:34 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lj1-x22b.google.com with SMTP id y16so646182ljk.1; Wed, 14 Oct 2020 12:16:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=vftaGn+tPlqwowMoe7wChVgW8Lz/FH36O5dpIp2JjjM=; b=TlO/26w7twGNZ45Ze+VhipRvr03PPhwk/w2Z/lEliOhgRolFIHFu0QLhW4tZCf9UXq Pqt5EdsfwhJh3dJ5FWBLRTPkurxnpzaDUWIgA2FazMmTS1XyT0EGso+9pP2/bU+d6EbA CuApC8203BxP5QS6dxmZ0GImDwGa5JdKIHjrqYxcRS6TqQUlCOA54gq/L3QJDRCMTnUG LuAXeeE/jJojUGBTkaGpZXOQvmoEZe9Sx0opSNio5VV096ozO4XV/hzwjBSaLDOrWFUn XNFrkb8SmQq2j4sR/hLndpyMWjmMnuvGjGSpJyCyJ4DbmrzEBmZlrPMJUfAdBVyIDULW bjqQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=vftaGn+tPlqwowMoe7wChVgW8Lz/FH36O5dpIp2JjjM=; b=r3SgR1CzfUUiYdHINdjX2fWsIphbsc+RPzzL62FxPwxAAeuew1gGpZE3BjFA4AhWhV Qnsv7yZd17bkpALY9upaIBFoOcr4OrsZx/24VdeAWajNki+bJT0eOcnDoMj9h9qlVHqM f/szeTDEe3dZFG9qVHgt2Ae6E+L9/mhHwvSzxTev2X8xYmR+ASJn3U1zGetaP0fKcKU3 IlRYsPI/AOQIvm55gdJ0eklm09r4Yw89Tpl1BFtF+fWeAcWJpsbOEZsDrfhhz2z4i9nq AOPtMxKWSN/YCkzMfyemqUcefVMSQa3jlIudGujbdFZTFNZ/6Y7w3YfGpos0z8T8NDoB 8kxw== X-Gm-Message-State: AOAM5316gGOspH8Qfbq+ziISkjj77nIBjodLCfjENzK7ENvXh2SVx4lF udKwM7afWT+qGpNMt64PWZHE4YzFAZSI27Nlv/RxR6l3 X-Google-Smtp-Source: ABdhPJySjyEWzJ8efL+3x7xH1U+/JhB1XBss445qJpDhExDoftqbqqrct6HujJ78qGQPLpcCS2FjLj9QvIhCL39Nawo= X-Received: by 2002:a2e:8599:: with SMTP id b25mr56034lji.107.1602702992554; Wed, 14 Oct 2020 12:16:32 -0700 (PDT) MIME-Version: 1.0 References: <5F8336C7.5020709@incore.de> <5F84CF18.1040905@incore.de> <0072D8A9-6ACE-47D0-AE94-124C4F955735@FreeBSD.org> In-Reply-To: From: J David Date: Wed, 14 Oct 2020 15:16:21 -0400 Message-ID: Subject: Re: Packets passed by pf don't make it out? To: Kristof Provost Cc: Andreas Longwitz , freebsd-pf@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4CBMbQ4XGDz4N0q X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; REPLY(-4.00)[] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 19:16:35 -0000 On Wed, Oct 14, 2020 at 1:59 PM Kristof Provost wrote: > There=E2=80=99s good reason to do this, as we have to be able to match st= ate > on both the pre-translation side (when processing LAN -> WAN traffic) > and post-translation (WAN -> LAN). So, basically, pf would need separate states for each pre-redirect destination address in order to have the information needed to map the reply packet back to the original destination address. But even if pf did that, the problem does not go away. It just moves to the reply packet coming back with only the post-redirect info. That info matches multiple states, leaving pf no way to pick the right one. Is that about right? Thanks! From owner-freebsd-pf@freebsd.org Wed Oct 14 19:20:26 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4C017440A2F for ; Wed, 14 Oct 2020 19:20:26 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CBMgt1JVbz4N8M; Wed, 14 Oct 2020 19:20:26 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id EB9E424419; Wed, 14 Oct 2020 19:20:25 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 814943B920; Wed, 14 Oct 2020 21:20:24 +0200 (CEST) From: "Kristof Provost" To: "J David" Cc: "Andreas Longwitz" , freebsd-pf@freebsd.org Subject: Re: Packets passed by pf don't make it out? Date: Wed, 14 Oct 2020 21:20:23 +0200 X-Mailer: MailMate (1.13.2r5673) Message-ID: <66EA3FE1-598F-4D42-8464-5A3A5C75CD07@FreeBSD.org> In-Reply-To: References: <5F8336C7.5020709@incore.de> <5F84CF18.1040905@incore.de> <0072D8A9-6ACE-47D0-AE94-124C4F955735@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed; markup=markdown Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 19:20:26 -0000 On 14 Oct 2020, at 21:16, J David wrote: > On Wed, Oct 14, 2020 at 1:59 PM Kristof Provost > wrote: >> There’s good reason to do this, as we have to be able to match >> state >> on both the pre-translation side (when processing LAN -> WAN traffic) >> and post-translation (WAN -> LAN). > > So, basically, pf would need separate states for each pre-redirect > destination address in order to have the information needed to map the > reply packet back to the original destination address. > > But even if pf did that, the problem does not go away. It just moves > to the reply packet coming back with only the post-redirect info. > That info matches multiple states, leaving pf no way to pick the right > one. > > Is that about right? > Pretty much, I think. I’ve not dug very deep yet, but I wonder if we shouldn’t have to teach pf to change the source port to avoid conflicting states in the first place. It’s a non-trivial problem in any case. Regards, Kristof From owner-freebsd-pf@freebsd.org Wed Oct 14 19:36:02 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 82B66440E4E for ; Wed, 14 Oct 2020 19:36:02 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lf1-x12e.google.com (mail-lf1-x12e.google.com [IPv6:2a00:1450:4864:20::12e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CBN1t01qtz4NYw; Wed, 14 Oct 2020 19:36:01 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lf1-x12e.google.com with SMTP id l28so738322lfp.10; Wed, 14 Oct 2020 12:36:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=93Ox0yhdq/61aDhpD5tix6OiV283yJgsEbglhBBtKL0=; b=NDHv6B1foJgZOdcLOQnopZodj21PmW2m8gEC5zvuEIA0hi7Ajk6OW0ukddLACUdtLu o3aUJMrCszBqG4Rc8N1VcVvm3MCat7nvdbtcevP6eQBrEtVTVA7Qt3jPgBB5i5E3xmgi RMyVLp7w9wTWld54WD7BLjyr+SYxjbQY6lXV58YZ27oykzBv0kztPS3C8U06GLO4kWuI zmMemeNMQ1gLBSpkwyl7bgrTXb5n66PX3prET9W1L9HYPD7rfyEJBtcEf/6HGWWAg0hB VbhEbMyPBkiB9nfRBmCzUVS9C6LeZ0/ipLjhFb/TPlqleyDC6t6rCSqjxlnsmbI+C4zY TvJQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=93Ox0yhdq/61aDhpD5tix6OiV283yJgsEbglhBBtKL0=; b=s9K2tgZpeaL4WzBKg5v2UR3KC/sVJ7LWATFrNfJ7eTB5/H5zmJxi2DIvY5oB66vrJZ e81yOExK3y2d6w6TocLJuPMDMYwAYagkJV0mLAl7v9wtvo1YeD+giSXJZ5gzvADXwr0+ r4MQSnNbY2sDEp+SmAI5/aNK4xlxmF8/l7VxHGMU2Qba20BXDXMjQUIEMfsTGrBqRYyY kvIetaRyp0REX8xew1dOxgjHFqQ+Wp9y0Vgi+OThDdpaBoxXmUw+6EG8XXBUZAqr7VBR 5aChoAiHeGtwlByuiNF6iqEq9BOhiaoUJ8J5IfU0N3XTsLPpkrHsubl9t5/8gtBbUxXi d3Ew== X-Gm-Message-State: AOAM5302PsjANS3fPdUEw8iQSZBvxmp4VRL78lBqhiAVy6r5lqR82oUT axjACcGBq4lyqn/aD+pIKoU3JRWbxJEUVQ/z+lphMyJN X-Google-Smtp-Source: ABdhPJzywDzQQ/aS3Zy69vtIdK8i/tbMTezEtUk0nWXxhfj4QXRmDKJELgYrBb5ePdkMy1m1/xGXuGnlE2v7+ngAplk= X-Received: by 2002:a05:6512:2001:: with SMTP id a1mr239850lfb.336.1602704159322; Wed, 14 Oct 2020 12:35:59 -0700 (PDT) MIME-Version: 1.0 References: <5F8336C7.5020709@incore.de> <5F84CF18.1040905@incore.de> <0072D8A9-6ACE-47D0-AE94-124C4F955735@FreeBSD.org> <66EA3FE1-598F-4D42-8464-5A3A5C75CD07@FreeBSD.org> In-Reply-To: <66EA3FE1-598F-4D42-8464-5A3A5C75CD07@FreeBSD.org> From: J David Date: Wed, 14 Oct 2020 15:35:48 -0400 Message-ID: Subject: Re: Packets passed by pf don't make it out? To: Kristof Provost Cc: Andreas Longwitz , freebsd-pf@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 4CBN1t01qtz4NYw X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 14 Oct 2020 19:36:02 -0000 On Wed, Oct 14, 2020 at 3:20 PM Kristof Provost wrote: > I=E2=80=99ve not dug very deep yet, but I wonder if we shouldn=E2=80=99t = have to > teach pf to change the source port to avoid conflicting states in the > first place. That was my first thought as well, framed mentally as some sort of port-only Frankenstein's binat because my level of understanding is clearly more cartoonish than yours. ;-) My second thought was to wonder if my approach is architecturally wrong. Would it make sense for the many-to-many case to use route-to instead of rdr, leave the packet unmodified, and expect every machine in the server pool to catch all the public IPs? That might still be tricky. Using rdr would presumably hit the same problem. Maybe something gross like ifconfig'ing the public pool addresses as /32's on lo0, then binding on those, maybe? Thanks! From owner-freebsd-pf@freebsd.org Thu Oct 15 08:51:42 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9683B437B4B for ; Thu, 15 Oct 2020 08:51:42 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [IPv6:2610:1c1:1:606c::24b:4]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4CBjgy3Xd2z4Dgn; Thu, 15 Oct 2020 08:51:42 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 412862A1B6; Thu, 15 Oct 2020 08:51:42 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id D78523CBAE; Thu, 15 Oct 2020 10:51:40 +0200 (CEST) From: "Kristof Provost" To: "J David" Cc: "Andreas Longwitz" , freebsd-pf@freebsd.org Subject: Re: Packets passed by pf don't make it out? Date: Thu, 15 Oct 2020 10:51:37 +0200 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: References: <5F8336C7.5020709@incore.de> <5F84CF18.1040905@incore.de> <0072D8A9-6ACE-47D0-AE94-124C4F955735@FreeBSD.org> <66EA3FE1-598F-4D42-8464-5A3A5C75CD07@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed; markup=markdown Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 15 Oct 2020 08:51:42 -0000 On 14 Oct 2020, at 21:35, J David wrote: > On Wed, Oct 14, 2020 at 3:20 PM Kristof Provost > wrote: >> I’ve not dug very deep yet, but I wonder if we shouldn’t have to >> teach pf to change the source port to avoid conflicting states in the >> first place. > > That was my first thought as well, framed mentally as some sort of > port-only Frankenstein's binat because my level of understanding is > clearly more cartoonish than yours. ;-) > > My second thought was to wonder if my approach is architecturally > wrong. Would it make sense for the many-to-many case to use route-to > instead of rdr, leave the packet unmodified, and expect every machine > in the server pool to catch all the public IPs? > > That might still be tricky. Using rdr would presumably hit the same > problem. Maybe something gross like ifconfig'ing the public pool > addresses as /32's on lo0, then binding on those, maybe? > I honestly don’t know. The pf NAT/RDR/… code is complex, and I certainly don’t understand all edge cases. It may be worth experimenting with such options though, because this is unlikely to be fixed short-term. Best regards, Kristof