From owner-freebsd-pf@freebsd.org Sun Oct 4 20:07:21 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4AF51431CF7 for ; Sun, 4 Oct 2020 20:07:21 +0000 (UTC) (envelope-from SRS0=jDJA=DL=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4C4FBc0lvzz4ClF for ; Sun, 4 Oct 2020 20:07:19 +0000 (UTC) (envelope-from SRS0=jDJA=DL=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 4A56C2842F; Sun, 4 Oct 2020 22:07:11 +0200 (CEST) Received: from illbsd.quip.test (ip-94-112-144-235.net.upcbroadband.cz [94.112.144.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 4DD3F2840C; Sun, 4 Oct 2020 22:07:10 +0200 (CEST) Subject: Re: PF states limit reached To: l.m.v.breda@xs4all.nl, freebsd-pf References: <489adbd3-4400-0cf8-31f1-45509af31925@quip.cz> <9c2bc3f6-0420-fe79-ae36-8a62511f71b2@quip.cz> <000801d6996d$81b5ab20$85210160$@xs4all.nl> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: Date: Sun, 4 Oct 2020 22:07:09 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.10.0 MIME-Version: 1.0 In-Reply-To: <000801d6996d$81b5ab20$85210160$@xs4all.nl> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 4C4FBc0lvzz4ClF X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of SRS0=jDJA=DL=quip.cz=000.fbsd@elsa.codelab.cz has no SPF policy when checking 94.124.105.4) smtp.mailfrom=SRS0=jDJA=DL=quip.cz=000.fbsd@elsa.codelab.cz X-Spamd-Result: default: False [2.32 / 15.00]; RCVD_TLS_LAST(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; ARC_NA(0.00)[]; NEURAL_SPAM_SHORT(0.40)[0.402]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.55)[0.547]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_SPAM_LONG(0.17)[0.168]; R_SPF_NA(0.00)[no SPF record]; FREEMAIL_TO(0.00)[xs4all.nl,freebsd.org]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=jDJA=DL=quip.cz=000.fbsd@elsa.codelab.cz]; RECEIVED_SPAMHAUS_PBL(0.00)[94.112.144.235:received]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=jDJA=DL=quip.cz=000.fbsd@elsa.codelab.cz]; MAILMAN_DEST(0.00)[freebsd-pf] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Oct 2020 20:07:21 -0000 On 03/10/2020 12:11, l.m.v.breda@xs4all.nl wrote: > Miroslav, > > I saw your mails. First thing I thought when I dis see your mails is "** What is going on, on that network!! **". > > I can be wrong, but are you really sure that there is no malware of any kind, using your network, causing the problems !! I can never be 100% sure but as far as I can tell there is no malware on this network. We have rented 19" rack in DC with /25 IP addresses and only this VM in question had this problem. No anomalies seen on the network (no unusual traffic, Apache workers and so on) > I would never change my firewall, to cope with strange things !! > Just making things less secure! I don't think PF without state tracking would be less secure. I am not an expert in this area but as I can see it the states can be target for DoS and I do not think the state tracking is useful if we already have policy "open for all outgoing traffic". Maybe I am wrong. I was thinking about "no state" for a long time regardless of this current issue. I don't know what was causing this problem but it disappeared after VM reboot. So I think it was some issue on OS / kernel side. I hope it will not repeat again but if it will I will let you know. 3 hours after reboot everything seems fine: # pfctl -s states | wc -l 55 # pfctl -s info Status: Enabled for 0 days 03:06:21 Debug: Urgent Interface Stats for em0 IPv4 IPv6 Bytes In 180884551 0 Bytes Out 1182768426 0 Packets In Passed 685980 0 Blocked 1471 0 Packets Out Passed 1008493 0 Blocked 124 0 State Table Total Rate current entries 63 searches 1696122 151.7/s inserts 31427 2.8/s removals 31364 2.8/s Counters match 33014 3.0/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 0 0.0/s state-mismatch 8 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s map-failed 0 0.0/s Kind regards Miroslav Lachman From owner-freebsd-pf@freebsd.org Sat Oct 10 00:25:09 2020 Return-Path: Delivered-To: freebsd-pf@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D5F5B431B17; Sat, 10 Oct 2020 00:25:09 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lf1-x141.google.com (mail-lf1-x141.google.com [IPv6:2a00:1450:4864:20::141]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C7Qgm5jd7z3cMd; Sat, 10 Oct 2020 00:25:08 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lf1-x141.google.com with SMTP id c141so5844517lfg.5; Fri, 09 Oct 2020 17:25:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=s9auyOeQD/V2h6BcqQ2Q/LjkTlWLZde7d23VdK2/Gt4=; b=BB2SFbw0LihL3+k2d1PC02S2+pDApk6Jh3yNI2FFIM582iD6yDRn2F7GyVvegKE933 1m0A+/sCwuctZ0H0ewiewEsMc0lNN2O6urFNqm0aVQoTsi1SG8uhwedRJ2OZvgW8qm1j wuObs22QWCSTRjElH7Kk7i8+Dk8TKbjuydHN57MI54TdGGCjGN8ksFF79EGZhYod6tef OjHDdLeTkf6dDApfOIIyof/wGEdV4dvWEF/vyWTYWW4CtFI7/1kCSmE9yS+ps8i5AcCP vzxefrkh3GDmha64W4ePiYa3cxbrDZ+5gOBmxwWvZzFpJ9yBwuJ2QSCcvnGqUg7C4ayx fz0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=s9auyOeQD/V2h6BcqQ2Q/LjkTlWLZde7d23VdK2/Gt4=; b=YawoVd+fO0pgrnAnO+bWHLeWE3JBP+RMnUB1afdt+fFeJSI5kqyClBIPn4r2x7tDRA GvwHcpBPc3rjrv/bs/imG0vDQ7wEzUthyc8JO4pRovqxdngzzDlq+f0jFrKzpTdyQDLt Q88CG5O5sMxOnbR3IUtIh1tL7JXwJFJMLoKsoNaz88vK/YF9DlF3a7L6TatuBBREgEVV 6g7MjPN1fs3JEn5AEbCCKJuz0EVSxn7ZDzBpOOgdfDdZg4fo+MD0Ew82M7n/4OseCU7D VUOe7wRyVzvBtlLn3J6DPuAdF4P4p9SCliQ8s2mrLiIRKu3aVkf5x/ibBZRvUDJwyUpE 7IhA== X-Gm-Message-State: AOAM5310vAqZz9PN+5I/B3QKwYzntPTbcrd+uXpZ8W6LqF3kNlAmtvc1 RkflE1It3gEvQSYEFYyYCM0OiblL9ny4jTRxL/UALCN4q7o= X-Google-Smtp-Source: ABdhPJx+TukpWK3DKoF1Ng4SV7fDG9N3l94ECrhl7RkV3nr3pO0fU1aEaVy/39binEHY8Xr7GWItI21bLH6WnYJUESQ= X-Received: by 2002:a19:c617:: with SMTP id w23mr942455lff.602.1602289505801; Fri, 09 Oct 2020 17:25:05 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: J David Date: Fri, 9 Oct 2020 20:24:54 -0400 Message-ID: Subject: Re: Packets passed by pf don't make it out? To: freebsd-pf@freebsd.org Cc: freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4C7Qgm5jd7z3cMd X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=BB2SFbw0; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2a00:1450:4864:20::141 as permitted sender) smtp.mailfrom=jdavidlists@gmail.com X-Spamd-Result: default: False [-1.79 / 15.00]; SUBJECT_ENDS_QUESTION(1.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; FREEMAIL_FROM(0.00)[gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_HAM_LONG(-0.97)[-0.968]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_MEDIUM(-0.98)[-0.981]; NEURAL_SPAM_SHORT(0.16)[0.157]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::141:from]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; MAILMAN_DEST(0.00)[freebsd-pf,freebsd-net]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Oct 2020 00:25:09 -0000 To investigate this issue, I've created a greatly simplified and reproducible test case. The code is available at: https://github.com/jdavidlists/pfudpbug It includes the pf.conf, the source code for client and server, and the rc.conf from all three machines. The test uses three FreeBSD systems (client, gateway, and server) to demonstrate that if a client with a bound UDP source port sends UDP packets to multiple server addresses that a gateway running pf redirects to one backend server, only the first such packet will be delivered. The remaining packets never leave the gateway; they get lost somewhere after being logged as passing pf via pflog0 but before a tcpdump running on the gateway's server-facing interface. They also do not increase the outbound packet count on the server-facing interface. (More detail is available in the repository's README.md.) This may indicate a bug, but I'm not sure whether it is happening inside pf or farther along the output path. Nor do I know how to explore this further. Is anyone able to point me in the right direction here? Thanks! On Fri, Oct 2, 2020 at 1:35 PM J David wrote: > > We have pf running on a FreeBSD 11.4 system acting as a load balancer, > mapping a set of 8 external DNS service IP addresses to a set of > internal DNS servers, any of which can handle those requests. > > When UDP packets from one source IP/port arrive for multiple external > IPs in a short period of time, pf claims they all pass, but only the > ones for the first IP actually make it out the outbound interface. > > Redirect rule: > > rdr inet proto udp from any to { 172.17.53.1, 172.17.53.2, > 172.17.53.3, 172.17.53.4, 172.17.53.5, 172.17.53.6, 172.17.53.7, > 172.17.53.8 } port 53 -> { 10.53.0.1, 10.53.0.2, 10.53.0.3 } > round-robin sticky-address > > Pass rule: > > pass in log quick proto udp to { 10.53.0.1, 10.53.0.2, 10.53.0.3 } port 53 > > (The pass rule isn't technically necessary, it's only there to log the > packets to debug this issue.) > > With tcpdumps running simultaneously on ix1, all packets show up the > inbound interface: > > 16:32:39.183168 IP 149.20.1.48.56246 > 172.17.53.1.53: 3215 SOA? > example.com. (29) > 16:32:39.183761 IP 149.20.1.48.56246 > 172.17.53.2.53: 2934 SOA? > example.com. (29) > 16:32:39.184368 IP 149.20.1.48.56246 > 172.17.53.3.53: 52875 SOA? > example.com. (29) > 16:32:39.185618 IP 149.20.1.48.56246 > 172.17.53.4.53: 36289 SOA? > example.com. (29) > 16:32:39.186067 IP 149.20.1.48.56246 > 172.17.53.5.53: 44049 SOA? > example.com. (29) > 16:32:39.186422 IP 149.20.1.48.56246 > 172.17.53.6.53: 34410 SOA? > example.com. (29) > 16:32:39.186494 IP 149.20.1.48.56246 > 172.17.53.7.53: 30923 SOA? > example.com. (29) > 16:32:39.188541 IP 149.20.1.48.56246 > 172.17.53.8.53: 48814 SOA? > example.com. (29) > > and on pflog0: > > 16:32:39.183189 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 3215 SOA? example.com. (29) > 16:32:39.183780 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 2934 SOA? example.com. (29) > 16:32:39.184375 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 52875 SOA? example.com. (29) > 16:32:39.185625 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 36289 SOA? example.com. (29) > 16:32:39.186074 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 44049 SOA? example.com. (29) > 16:32:39.186425 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 34410 SOA? example.com. (29) > 16:32:39.186499 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 30923 SOA? example.com. (29) > 16:32:39.188548 rule 16/0(match): pass in on ix1: 149.20.1.48.56246 > > 10.53.0.1.53: 48814 SOA? example.com. (29) > > but only the first one appears on ix0, the outbound interface: > > 16:32:39.183211 IP 149.20.1.48.56246 > 10.53.0.1.53: 3215 SOA? example.com. (29) > > The actual query order is random, so if the test is repeated a minute > later, then 172.17.53.3 might be hit first, and then that one will > make it through and the rest will disappear. So it is not specific to > any destination IP. > > It also only appears to occur when the UDP source port is the same > across the connections. (This is probably why TCP is not affected.) > > It does not appear related to state entries ("no state") doesn't help. > > If "sticky-address" is removed from the rdr, then one packet will make > it through for each backend IP, instead of one total. > > What could be causing this? It seems somehow related to 5-tuple > non-uniqueness after the rdr, but that shouldn't be an issue for UDP; > it should be treated as two connectionless packets from the same > source for the same destination. > > (The query test source, 149.20.1.48 is an EDNS checker found at > https://dnsflagday.net/2020/ .) > > Thanks for any advice!