From owner-freebsd-net@freebsd.org Mon Apr 23 12:10:41 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8DB2FB6DEF for ; Mon, 23 Apr 2018 12:10:40 +0000 (UTC) (envelope-from vit@otcnet.ru) Received: from mail.otcnet.ru (mail.otcnet.ru [194.190.78.3]) by mx1.freebsd.org (Postfix) with ESMTP id 340186ACD8 for ; Mon, 23 Apr 2018 12:10:38 +0000 (UTC) (envelope-from vit@otcnet.ru) Received: from Victors-MacBook-Air-2.local (unknown [213.33.226.214]) by mail.otcnet.ru (Postfix) with ESMTPSA id AE3A059659F for ; Mon, 23 Apr 2018 15:10:28 +0300 (MSK) Subject: Re: multiple if_ipsec To: freebsd-net@freebsd.org References: <5e36ac3f-39ce-72c5-cd97-dd3c4cf551a7@yandex.ru> <30d1c5f9-56e7-c67b-43e1-e6f0457360a8@otcnet.ru> <77c37ff9-8de3-dec0-176a-2b34db136bc5@otcnet.ru> <92930ba6-828d-ecb5-ce37-36794ec80ef7@yandex.ru> From: Victor Gamov Organization: OTCnet Message-ID: <112ea6c0-1927-5f47-24c7-6888295496cf@otcnet.ru> Date: Mon, 23 Apr 2018 15:10:29 +0300 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <92930ba6-828d-ecb5-ce37-36794ec80ef7@yandex.ru> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Apr 2018 12:10:41 -0000 On 23/04/2018 14:13, Andrey V. Elsukov wrote: > On 21.04.2018 19:16, Victor Gamov wrote: >> When I change ipsec-interfaces creation order then only last created >> interface worked fine again and previously configured interfaces does >> not work. >> >> >> And very interesting fact: when I ping from remote 10.10.98.5 for >> example to FreeBSD 10.10.98.6 then no ICMP-request coming over >> ipsec-interface but ICMP-reply outgoing via this ipsec-interface (but >> not delivered to 10.10.98.5) >> >> >> Any ideas? > > I'm lack of any ideas. For further debugging I need to see the output of > # sysctl net. | grep ipsec > # setkey -DP > # setkey -D > # ifconfig > > And probably racoon's logs. Hi Andrey! First of all -- many thanks for your responses! Configs are followed # sysctl net. | grep ipsec ===== net.inet.ipsec.def_policy: 1 net.inet.ipsec.esp_trans_deflev: 1 net.inet.ipsec.esp_net_deflev: 1 net.inet.ipsec.ah_trans_deflev: 1 net.inet.ipsec.ah_net_deflev: 1 net.inet.ipsec.ah_cleartos: 1 net.inet.ipsec.ah_offsetmask: 0 net.inet.ipsec.dfbit: 0 net.inet.ipsec.ecn: 0 net.inet.ipsec.debug: 0 net.inet.ipsec.filtertunnel: 0 net.inet.ipsec.natt_cksum_policy: 0 net.inet.ipsec.check_policy_history: 0 net.inet.ipsec.crypto_support: 50331648 net.inet6.ipsec6.def_policy: 1 net.inet6.ipsec6.esp_trans_deflev: 1 net.inet6.ipsec6.esp_net_deflev: 1 net.inet6.ipsec6.ah_trans_deflev: 1 net.inet6.ipsec6.ah_net_deflev: 1 net.inet6.ipsec6.ecn: 0 net.inet6.ipsec6.debug: 0 net.inet6.ipsec6.filtertunnel: 0 ===== # setkey -DP | grep -A 4 '^0' ===== 0.0.0.0/0[any] 0.0.0.0/0[any] any in ipsec esp/tunnel/__Cisco_30__-__FreeBSD_IP__/unique:30 spid=1 seq=11 pid=99239 scope=ifnet ifname=ipsec30 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any in ipsec esp/tunnel/__Cisco_26__-__FreeBSD_IP__/unique#16385 spid=5 seq=9 pid=99239 scope=ifnet ifname=ipsec26 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any in ipsec esp/tunnel/__Cisco_25__-__FreeBSD_IP__/unique:26 spid=9 seq=7 pid=99239 scope=ifnet ifname=ipsec25 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any out ipsec esp/tunnel/__FreeBSD_IP__-__Cisco_30__/unique:30 spid=2 seq=5 pid=99239 scope=ifnet ifname=ipsec30 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any out ipsec esp/tunnel/__FreeBSD_IP__-__Cisco_26__/unique#16385 spid=6 seq=3 pid=99239 scope=ifnet ifname=ipsec26 refcnt=1 0.0.0.0/0[any] 0.0.0.0/0[any] any out ipsec esp/tunnel/__FreeBSD_IP__-__Cisco_25__/unique:26 spid=10 seq=1 pid=99239 scope=ifnet ifname=ipsec25 refcnt=1 ===== # setkey -D ===== __FreeBSD_IP__ __Cisco_30__ esp mode=tunnel spi=2124688285(0x7ea42b9d) reqid=26(0x0000001a) E: rijndael-cbc 6ca42c3b c24ce0ec f3f676c8 c9b9e72d fde63423 3f957b0c ee5da59d dce8a66d A: hmac-sha1 2adb7dfb 26d5de00 2fdd9a21 f63701ef 59d95a1a seq=0x00000000 replay=4 flags=0x00000000 state=mature created: Apr 23 14:02:03 2018 current: Apr 23 14:17:40 2018 diff: 937(s) hard: 3600(s) soft: 2880(s) last: hard: 0(s) soft: 0(s) current: 0(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 0 hard: 0 soft: 0 sadb_seq=5 pid=95677 refcnt=1 __FreeBSD_IP__ __Cisco_25__ esp mode=tunnel spi=153891647(0x092c333f) reqid=26(0x0000001a) E: rijndael-cbc 8f9905fe 6a9cfc76 a0da354b 53a7f901 298dca43 b5feda65 3be012e7 08835553 A: hmac-sha1 aa2ec447 0e6b36e2 23ba9b27 9d0ecc05 4513af70 seq=0x00000000 replay=4 flags=0x00000000 state=mature created: Apr 23 13:40:24 2018 current: Apr 23 14:17:40 2018 diff: 2236(s) hard: 3600(s) soft: 2880(s) last: hard: 0(s) soft: 0(s) current: 0(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 0 hard: 0 soft: 0 sadb_seq=4 pid=95677 refcnt=1 __Cisco_25__ __FreeBSD_IP__ esp mode=tunnel spi=21918183(0x014e71e7) reqid=26(0x0000001a) E: rijndael-cbc 43e8f54a 0bdda6b5 41a637d5 4469973d 5b3dc8d0 37022187 43c86f0c 34054df8 A: hmac-sha1 cf08a56a beead8b8 e637a14a 5fdbde3d b8c71192 seq=0x00000000 replay=4 flags=0x00000000 state=mature created: Apr 23 13:40:24 2018 current: Apr 23 14:17:40 2018 diff: 2236(s) hard: 3600(s) soft: 2880(s) last: Apr 23 13:40:25 2018 hard: 0(s) soft: 0(s) current: 46900(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 719 hard: 0 soft: 0 sadb_seq=3 pid=95677 refcnt=1 __FreeBSD_IP__ __Cisco_26__ esp mode=tunnel spi=2471238029(0x934c198d) reqid=26(0x0000001a) E: rijndael-cbc 01b3235e 0fe554d3 6dbcb505 bb34d511 93f8ee6f b0b15f43 077c411a afdb1b3b A: hmac-sha1 29ab22bd 2c4f0ade e1478e19 0ecf423f ef155ff3 seq=0x00000000 replay=4 flags=0x00000000 state=mature created: Apr 23 13:42:29 2018 current: Apr 23 14:17:40 2018 diff: 2111(s) hard: 3600(s) soft: 2880(s) last: hard: 0(s) soft: 0(s) current: 0(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 0 hard: 0 soft: 0 sadb_seq=2 pid=95677 refcnt=1 __Cisco_26__ __FreeBSD_IP__ esp mode=tunnel spi=103689330(0x062e2c72) reqid=26(0x0000001a) E: rijndael-cbc 27936832 275a949a a156336c dbc049e1 3a88218a 1f23351f 54eb336d 8381bf0b A: hmac-sha1 8ed4e3a6 7d3d5b25 0c167123 fc8052a5 43738cf8 seq=0x00000000 replay=4 flags=0x00000000 state=mature created: Apr 23 13:42:29 2018 current: Apr 23 14:17:40 2018 diff: 2111(s) hard: 3600(s) soft: 2880(s) last: Apr 23 13:42:33 2018 hard: 0(s) soft: 0(s) current: 27360(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 456 hard: 0 soft: 0 sadb_seq=1 pid=95677 refcnt=1 __Cisco_30__ __FreeBSD_IP__ esp mode=tunnel spi=42561509(0x02896fe5) reqid=26(0x0000001a) E: rijndael-cbc a9c9d21a b09f705b fbf33201 881b27af a23ea9fa 85085847 b4b50418 54d6c739 A: hmac-sha1 7994e8dc ece0c8e7 434ac694 b0fc7952 bc1e01b0 seq=0x00000000 replay=4 flags=0x00000000 state=mature created: Apr 23 14:02:03 2018 current: Apr 23 14:17:40 2018 diff: 937(s) hard: 3600(s) soft: 2880(s) last: Apr 23 14:02:05 2018 hard: 0(s) soft: 0(s) current: 19644(bytes) hard: 0(bytes) soft: 0(bytes) allocated: 301 hard: 0 soft: 0 sadb_seq=0 pid=95677 refcnt=1 ===== # ifconfig -au ===== em0: flags=8843 metric 0 mtu 1500 description: -LAN options=9b ether 00:50:56:b0:81:ac hwaddr 00:50:56:b0:81:ac inet 192.168.10.130 netmask 0xffffff00 broadcast 192.168.10.255 nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active em1: flags=8843 metric 0 mtu 1500 description: -WAN options=9b ether 00:50:56:b0:bf:de hwaddr 00:50:56:b0:bf:de inet __FreeBSD_IP__ netmask 0xffffffe0 broadcast __FreeBSD_IP_broadcast__ nd6 options=29 media: Ethernet autoselect (1000baseT ) status: active lo0: flags=8049 metric 0 mtu 16384 options=600003 inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3 inet 127.0.0.1 netmask 0xff000000 nd6 options=21 groups: lo ipsec30: flags=8051 metric 0 mtu 1400 description: -so: Kur tunnel inet __FreeBSD_IP__ --> __Cisco_30__ inet 10.10.98.1 --> 10.10.98.2 netmask 0xfffffffc nd6 options=29 reqid: 30 groups: ipsec ipsec26: flags=8051 metric 0 mtu 1400 description: -so: Mur tunnel inet __FreeBSD_IP__ --> __Cisco_26__ inet 10.10.98.9 --> 10.10.98.10 netmask 0xfffffffc nd6 options=29 reqid: 16385 groups: ipsec ipsec25: flags=8051 metric 0 mtu 1400 description: -so: Sofy tunnel inet __FreeBSD_IP__ --> __Cisco_25__ inet 10.10.98.5 --> 10.10.98.6 netmask 0xfffffffc nd6 options=29 reqid: 26 groups: ipsec ===== Racoon launched with debug now and sometimes I've got DEBUG messages ===== racoon: DEBUG: no such a SA found: ESP/Tunnel __Cisco_30__[500]->__FreeBSD_IP__[500] spi=198258211(0xbd12e23) racoon: DEBUG: no such a SA found: ESP/Tunnel __Cisco_25__[500]->__FreeBSD_IP__[[500] spi=2471238029(0x934c198d) ===== with many FreeBSD/Cisco IP conbinations. And sometimes: ===== racoon: DEBUG: check spi(packet)=153891647 spi(db)=738738094. racoon: DEBUG: check spi(packet)=153891647 spi(db)=153891647. racoon: DEBUG: purged 1 SAs. racoon: DEBUG: purged SAs. racoon: DEBUG: pk_recv: retry[0] recv() racoon: DEBUG: DELETE message is not interesting because the message was originated by me. racoon: DEBUG: pk_recv: retry[0] recv() racoon: DEBUG: got pfkey ACQUIRE message ===== Regardless this messages ping still works fine but for last configured ipsec-interface -- CU, Victor Gamov