From owner-freebsd-net@freebsd.org Mon Jul 24 15:14:17 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 433D2CFC368 for ; Mon, 24 Jul 2017 15:14:17 +0000 (UTC) (envelope-from m.muenz@spam-fetish.org) Received: from mailout-02.maxonline.de (mailout-02.maxonline.de [81.24.66.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F0B7E6B73B for ; Mon, 24 Jul 2017 15:14:16 +0000 (UTC) (envelope-from m.muenz@spam-fetish.org) Received: from web03-01.max-it.de (web03-01.max-it.de [81.24.64.215]) by mailout-02.maxonline.de (Postfix) with ESMTPS id 919F54B for ; Mon, 24 Jul 2017 17:14:08 +0200 (CEST) Received: from localhost (localhost [127.0.0.1]) by web03-01.max-it.de (Postfix) with ESMTP id 8196928B848 for ; Mon, 24 Jul 2017 17:14:08 +0200 (CEST) X-Virus-Scanned: Debian amavisd-new at web03-01.max-it.de Received: from web03-01.max-it.de ([127.0.0.1]) by localhost (web03-01.max-it.de [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id bytYUoLydfnO for ; Mon, 24 Jul 2017 17:14:07 +0200 (CEST) Received: from [81.24.66.132] (unknown [81.24.66.132]) (Authenticated sender: m.muenz@spam-fetish.org) by web03-01.max-it.de (Postfix) with ESMTPA id E91C628B847 for ; Mon, 24 Jul 2017 17:14:07 +0200 (CEST) Subject: Re: NAT before IPSEC - reply packets stuck at enc0 To: freebsd-net@freebsd.org References: <459d59f7-2895-8aed-d547-be46a0fbb918@spam-fetish.org> <1c0de616-91ff-a6f9-d946-f098bc1a709f@spam-fetish.org> <911903d1-f353-d5d6-d400-d86150f88136@yandex.ru> <2d607e1a-a2c0-0f85-1530-c478962a76cd@spam-fetish.org> <3344e189-cdf0-a2c9-3a2a-645460866f2d@yandex.ru> <1279753e-9ad1-2c02-304e-5001e2bbc82f@spam-fetish.org> <15e6eb38-ef0c-7bfd-5f2c-d2acc8ea1af4@yandex.ru> <63e80fcf-915e-2dd5-d8c9-1904c8261c6f@yandex.ru> <1c91cd8f-105d-e886-3126-67505c6c3900@spam-fetish.org> From: "Muenz, Michael" Message-ID: <1e889acf-49d1-b70f-7097-82e6e4dfabb6@spam-fetish.org> Date: Mon, 24 Jul 2017 17:15:12 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.2.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Jul 2017 15:14:17 -0000 Am 24.07.2017 um 13:18 schrieb Andrey V. Elsukov: > > Ok, let's try to debug the problem. Please, use 11.1-RC, it has > significantly changed IPsec stack. > > Apply attached patch to if_enc(4), it makes if_enc a bit useful for > debugging your problem. You need to rebuild and reinstall > sys/modules/if_enc. > > Now enable verbose BPF logging: > net.enc.out.ipsec_bpf_mask=3 > net.enc.in.ipsec_bpf_mask=3 > > According your tcpdump output, you need to set > net.enc.out.ipsec_filter_mask=2 > > Show what you will see in the `tcpdump -nvi enc0` with such config > options. Also, show what you have in the `sysctl net.inet.ip.fw` and > `ipfw show` output. > Great! The guys from OPNsense built me a custom 11.1 kernel with your patch. Here's one packet on enc0: root@PB-FW1-FRA:~ # tcpdump -vni enc0 tcpdump: listening on enc0, link-type ENC (OpenBSD encapsulated IP), capture size 262144 bytes 17:07:41.769313 (authentic,confidential): SPI 0x90b7ba72: IP (tos 0x0, ttl 63, id 27752, offset 0, flags [none], proto ICMP (1), length 28, bad cksum b72d (->b82d)!) 10.26.1.1 > 10.24.66.25: ICMP echo request, id 41163, seq 28416, length 8 17:07:41.777223 (authentic,confidential): SPI 0xcaae18f8: IP (tos 0x0, ttl 58, id 44180, offset 0, flags [none], proto IPIP (4), length 48) 81.24.74.3 > 213.244.192.191: IP (tos 0x0, ttl 63, id 46327, offset 0, flags [none], proto ICMP (1), length 28) 10.24.66.25 > 10.26.1.1: ICMP echo reply, id 41163, seq 28416, length 8 17:07:41.777240 (authentic,confidential): SPI 0xcaae18f8: IP (tos 0x0, ttl 63, id 46327, offset 0, flags [none], proto ICMP (1), length 28) 10.26.1.1 > 10.26.1.1: ICMP echo reply, id 33347, seq 28416, length 8 17:07:41.846588 (authentic,confidential): SPI 0x90b7ba72: IP (tos 0x0, ttl 63, id 61607, offset 0, flags [none], proto ICMP (1), length 28, bad cksum 32ee (->33ee)!) 10.26.1.1 > 10.24.66.25: ICMP echo request, id 45562, seq 58116, length 8 17:07:41.854692 (authentic,confidential): SPI 0xcaae18f8: IP (tos 0x0, ttl 58, id 44196, offset 0, flags [none], proto IPIP (4), length 48) 81.24.74.3 > 213.244.192.191: IP (tos 0x0, ttl 63, id 46335, offset 0, flags [none], proto ICMP (1), length 28) 10.24.66.25 > 10.26.1.1: ICMP echo reply, id 45562, seq 58116, length 8 17:07:41.854706 (authentic,confidential): SPI 0xcaae18f8: IP (tos 0x0, ttl 63, id 46335, offset 0, flags [none], proto ICMP (1), length 28) 10.26.1.1 > 10.26.1.1: ICMP echo reply, id 40754, seq 58116, length 8 ipfw show: root@PB-FW1-FRA:~ # ipfw show 00100 0 0 allow pfsync from any to any 00110 0 0 allow carp from any to any 00120 0 0 allow ip from any to any layer2 mac-type 0x0806,0x8035 00130 0 0 allow ip from any to any layer2 mac-type 0x888e,0x88c7 00140 0 0 allow ip from any to any layer2 mac-type 0x8863,0x8864 00150 0 0 deny ip from any to any layer2 not mac-type 0x0800,0x86dd 00179 410 11480 nat 1 log ip from 10.26.2.0/24 to 10.24.66.0/24 00179 414 11816 nat 1 log ip from 10.24.66.0/24 to 10.26.1.1 in recv enc0 00200 0 0 skipto 60000 ip6 from ::1 to any 00201 44 41006 skipto 60000 ip4 from 127.0.0.0/8 to any 00202 0 0 skipto 60000 ip6 from any to ::1 00203 0 0 skipto 60000 ip4 from any to 127.0.0.0/8 01002 0 0 skipto 60000 udp from any to 10.26.1.1 dst-port 53 keep-state 01002 4 336 skipto 60000 ip from any to { 255.255.255.255 or 10.26.1.1 } in 01002 463 14672 skipto 60000 ip from { 255.255.255.255 or 10.26.1.1 } to any out 01002 0 0 skipto 60000 icmp from { 255.255.255.255 or 10.26.1.1 } to any out icmptypes 0 01002 0 0 skipto 60000 icmp from any to { 255.255.255.255 or 10.26.1.1 } in icmptypes 8 06000 5131 4476281 skipto 60000 tcp from any to any out 06199 10768 1914882 skipto 60000 ip from any to any 30000 0 0 count ip from any to any 60000 0 0 return ip from any to any 60001 0 0 queue 10000 tcp from any to 10.24.66.0/24 via enc0 65533 16410 6447177 allow ip from any to any 65534 0 0 deny ip from any to any 65535 0 0 deny ip from any to any sysctl: net.enc.out.ipsec_bpf_mask: 3 net.enc.out.ipsec_filter_mask: 1 net.enc.in.ipsec_bpf_mask: 3 net.enc.in.ipsec_filter_mask: 2 net.enc.out.ipsec_bpf_mask: 3 net.enc.out.ipsec_filter_mask: 1 net.enc.in.ipsec_bpf_mask: 3 net.enc.in.ipsec_filter_mask: 2 root@PB-FW1-FRA:~ # sysctl net.inet.ip.fw net.inet.ip.fw.dyn_keep_states: 0 net.inet.ip.fw.dyn_keepalive: 1 net.inet.ip.fw.dyn_short_lifetime: 5 net.inet.ip.fw.dyn_udp_lifetime: 10 net.inet.ip.fw.dyn_rst_lifetime: 1 net.inet.ip.fw.dyn_fin_lifetime: 1 net.inet.ip.fw.dyn_syn_lifetime: 20 net.inet.ip.fw.dyn_ack_lifetime: 300 net.inet.ip.fw.dyn_max: 16384 net.inet.ip.fw.dyn_count: 0 net.inet.ip.fw.curr_dyn_buckets: 256 net.inet.ip.fw.dyn_buckets: 256 net.inet.ip.fw.enable: 1 net.inet.ip.fw.static_count: 25 net.inet.ip.fw.default_to_accept: 0 net.inet.ip.fw.tables_sets: 0 net.inet.ip.fw.tables_max: 128 net.inet.ip.fw.default_rule: 65535 net.inet.ip.fw.verbose_limit: 0 net.inet.ip.fw.verbose: 0 net.inet.ip.fw.autoinc_step: 100 net.inet.ip.fw.one_pass: 0 Thanks! Michael -- www.muenz-it.de - Cisco, Linux, Networks