From owner-freebsd-net@freebsd.org Sun Dec 10 17:35:30 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D5373E91EE5 for ; Sun, 10 Dec 2017 17:35:30 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0C5A678AC3 for ; Sun, 10 Dec 2017 17:35:29 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBAHZ5rr097419 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Sun, 10 Dec 2017 18:35:06 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: trashcan@ellael.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id vBAHZ0iS007513 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Mon, 11 Dec 2017 00:35:00 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: [IPsec] Weird performance issue via IPsec/racoon tunnel To: Michael Grimm , freebsd-net@FreeBSD.org References: <7A6EF712-920E-40BF-B155-113EE6C00AEA@ellael.org> From: Eugene Grosbein Message-ID: <5A2D703F.8040004@grosbein.net> Date: Mon, 11 Dec 2017 00:34:55 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <7A6EF712-920E-40BF-B155-113EE6C00AEA@ellael.org> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Dec 2017 17:35:31 -0000 10.12.2017 23:55, Michael Grimm wrote: > Hi > > I do run an IPsec/racoon tunnel between two servers (11.1-STABLE #0 r326663). Some days ago I did migrate one of my servers from bare metal to a public cloud instance. Now I do observe weird performance issues from new to old server: > > ifconfig (OLD server, bare metal): > ix0: flags=8843 metric 0 mtu 1500 > options=e407bb TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> > > ifconfig (NEW server, cloud instance): > vtnet0: flags=8843 metric 0 mtu 1500 > options=6c07bb TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> > > Immediately after booting of NEW (test file has 10 MB) I do observe the following: > > #) scp OLD to NEW via ssh/internet: 16.7 MB/s > #) scp NEW to OLD via ssh/internet: 17.4 MB/s > #) scp NEW to OLD via IPsec tunnel: -> 65.8 KB/s ! > #) scp OLD to NEW via IPsec tunnel: 16.5 MB/s > > Now I do a "ifconfig vtnet0 mtu 1500 up" and can observe very similar performance. > > *BUT* if I do a "ifconfig vtnet0 mtu 1450 up ; ifconfig vtnet0 mtu 1500 up" I do observe: > > #) scp NEW to OLD via IPsec tunnel: 17.1 MB/s ! > #) scp OLD to NEW via IPsec tunnel: 16.9 MB/s > > I did monitor "tcpdump -i ix0 -vv esp" at the OLD sever and do get many: > > 16:22:24.370486 IP (tos 0x8, ttl 64, id 17394, offset 0, flags [none], proto ESP (50), \ > length 140, bad cksum 0 (->b110)!) > "OLD" > "NEW": ESP(spi=0x0d83dae4,seq=0x3a8d9a), length 120 > > At the NEW server I do not observe those checksum errors at all. *BUT* I do see these error even after regaining full performance by modifying the MTU from 1500 to 1450 and back to 1500! > > Well, I do have to admit that I do not have enough knowledge about networking to find out by myself what to debug/modify next. > > Any help is highly appreciated. "bad cksum 0" is pretty normal for traffic going out via interface supporting hardware checksum offload, so kernel skips computing checksum before passing packets to the NIC. Your problem more likely is due to fragmented ESP packets. It's not uncommon when cloud IP stack or ISP infrastructure drop high percentage of fragmented ESP packets because they are not optimized for such packets, e.g. router has to process them in software instead of hardware like non-fragmented packets are processed. When you lower MTU of vtnet enough to make encapsulated packets (payload+overhead) <=1500 bytes, resulted ESP packets have not be fragmented and pass just fine. To verify if it's your case, you should run two tcpdump commands, one at sending side and another at receiving size and compare outputs to see if *every* outgoing packet reaches its destination or not.