From owner-freebsd-net@freebsd.org Sun Oct 11 18:00:12 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 529633F8FCE; Sun, 11 Oct 2020 18:00:12 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: from mail-lj1-x243.google.com (mail-lj1-x243.google.com [IPv6:2a00:1450:4864:20::243]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4C8V2g48f8z4RNN; Sun, 11 Oct 2020 18:00:11 +0000 (UTC) (envelope-from jdavidlists@gmail.com) Received: by mail-lj1-x243.google.com with SMTP id a23so13857303ljp.5; Sun, 11 Oct 2020 11:00:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=W+McX9gN4ZsVhAh1mAeYLyWPIfPcja4gwBZZhAdLQZs=; b=KmKEM9cmubBDN9mS9YESOA3LMuSJyQMiL3AqQJELAqHCG/7e53c/wnKN4hyzDrJ223 9t2ZdCFhu6EIeldGL80m5G85BVnECrHJBAhy38FFpJkKP0cQt9Q7F1zAzHJXrrXm2axj VRmqWfKr2B9JxQyul6Ypuxwb6p+o0NdVmRw7LeLwteYL2XtZx39iGNlbRmLOhAmKV5A9 OG0GfTwRwk6vEmkKfg6+lSS1XGy/zEP4nU645acB50dbIH0wJxxBsZV/ZPZn56g+S93T MguClpUciZOWTq3PWgYO8yg7wztTKbJncMio9Sdmt1WAfsbfCrEYJYe27/aUJPSNd1Wb EgOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=W+McX9gN4ZsVhAh1mAeYLyWPIfPcja4gwBZZhAdLQZs=; b=ZOY4pYKgepISHivLwHIHDv4YcKJ1ITC+6gZqy3cQKVDyTKv1LdjCkXZuVACFNhVuc1 m8BH1MRg+7G/ahzXmcadp63YoydjC2f08F0WFQ58YcFTSnA2ng2VZnDnHnlRqQ+2pT05 w+cTY813raHXf2pnviVUHwt6KBzfRVXojGhVSTEj14pTvVHI9XzLLWDTz95oc/Wo4TQp Gc2UMT73AaLhzXR12ToMG7vA3jnvxdjU0vHW4lgjyiF3T9EHMmWpOR1T6nT3opKZCBRe 1+PBe8MraGb6v/c+HmPQVUa6meCKJXHSQ2V5ZfRoWDOtf/36thcrPAfUtsxet8I28OTX djMQ== X-Gm-Message-State: AOAM532NDO6gBF2jWrzFnaC/aP+lgQrkfEI6sTyKxahlTa/QZmdIQmTq 2lnQFdnzPeL75JzKa+iTZfuhnfVDfg8YVhngYNh9ovJpkuI= X-Google-Smtp-Source: ABdhPJwM4kP8nEoFjJM59lYtZGbGGHc/Pig0JRVpoclaQgjlQlmVoOtU/dMH1Ek9NoplgdUyTlLN4nb/SrLAA6ZS/jg= X-Received: by 2002:a2e:88c2:: with SMTP id a2mr6464503ljk.438.1602439208952; Sun, 11 Oct 2020 11:00:08 -0700 (PDT) MIME-Version: 1.0 References: <5F8336C7.5020709@incore.de> In-Reply-To: <5F8336C7.5020709@incore.de> From: J David Date: Sun, 11 Oct 2020 13:59:58 -0400 Message-ID: Subject: Re: Packets passed by pf don't make it out? To: Andreas Longwitz Cc: freebsd-pf@freebsd.org, freebsd-net@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4C8V2g48f8z4RNN X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=KmKEM9cm; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of jdavidlists@gmail.com designates 2a00:1450:4864:20::243 as permitted sender) smtp.mailfrom=jdavidlists@gmail.com X-Spamd-Result: default: False [-2.45 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-0.99)[-0.990]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MIME_GOOD(-0.10)[text/plain]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; NEURAL_HAM_LONG(-0.99)[-0.985]; RCVD_COUNT_TWO(0.00)[2]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::243:from]; NEURAL_HAM_SHORT(-0.48)[-0.476]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; TAGGED_FROM(0.00)[]; MAILMAN_DEST(0.00)[freebsd-pf,freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 11 Oct 2020 18:00:12 -0000 On Sun, Oct 11, 2020 at 12:46 PM Andreas Longwitz wrote: > Please look at the output of "pfctl -vsn" on fb2 during your test. > With "netstat -ss | grep drop" you can check for packets dropped by the > kernel for what reason ever. Here's the complete diff of the output from netstat -ss from before to after running the test: --- nss.pre 2020-10-11 17:10:19.932002000 +0000 +++ nss.post 2020-10-11 17:10:21.999823000 +0000 @@ -48,9 +48,9 @@ Packet drop statistics: Timeouts: ip: - 66578 total packets received + 66582 total packets received 66531 packets for this host - 16 packets forwarded + 17 packets forwarded 1 packet not forwardable 31675 packets sent from this host 10 packets sent with fabricated ip header No drops of any kind (nor anything else) recorded during the test. 4 packets in, 1 packet forwarded, which exactly matches the observed behavior of only one packet reaching the server. The results of "pfctl -vsn" are a bit more interesting, and also inconsistent. Before, after a full flush to zero states and counters: rdr inet proto udp from any to 172.16.0.0/12 port = 12345 -> 10.255.255.3 [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: uid 0 pid 1044 State Creations: 0 ] After: rdr inet proto udp from any to 172.16.0.0/12 port = 12345 -> 10.255.255.3 [ Evaluations: 4 Packets: 1 Bytes: 44 States: 1 ] [ Inserted: uid 0 pid 1044 State Creations: 4 ] So it says it created four states, but only matched one packet out of the four it evaluated. And it didn't create 4 states, either. "pfctl -s state" shows only 1: all udp 10.255.255.3:12345 (172.16.0.1:12345) <- 10.0.0.1:23456 NO_TRAFFIC:SINGLE and pflog0 reported all four packets as matching the pass rule which, important, is based on the destination address after redirection: 17:23:39.039641 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 17:23:39.039751 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 17:23:39.039769 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 17:23:39.039780 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 If I repeat the test, this happens: rdr inet proto udp from any to 172.16.0.0/12 port = 12345 -> 10.255.255.3 [ Evaluations: 7 Packets: 2 Bytes: 88 States: 1 ] [ Inserted: uid 0 pid 1044 State Creations: 7 ] But still just the one state: all udp 10.255.255.3:12345 (172.16.0.1:12345) <- 10.0.0.1:23456 NO_TRAFFIC:SINGLE But only three passes appear in pflog0: 17:29:19.857174 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 17:29:19.857193 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 17:29:19.857226 rule 0/0(match): pass in on em1: 10.0.0.1.23456 > 10.255.255.3.12345: UDP, length 16 And, to confirm, the first packet from each of these tests did reach the server, the remaining three from each test did not. > A similar setup works for me without any problems, so there may be > something special in your environment. This has been tested on fresh installs of both FreeBSD 12.1 and 11.4 on both physical hardware and virtual machines including both Xeons and AMD Epyc. So it seems like most environmental factors have been controlled for. > It seems your routing table on fb2 is empty, please try to set a > defaultroute, e.g.: "route add default 10.0.0.NN" with any NN. fb2 does have a default route, it is obtained from DHCP on the first interface. But that is not relevant; the client machine (fb1) is directly connected to fb2's second interface, and the server (fb3) is directly connected to fb2's third interface. No additional routes are necessary for this test, and the default route is never consulted. Perhaps there's some detail of the scenario that I have omitted without which it's not clear? Thanks!