From owner-freebsd-net@freebsd.org Sun Dec 10 16:55:22 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 043DBE90AD8 for ; Sun, 10 Dec 2017 16:55:22 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [IPv6:2001:41d0:d:3049:1:1:0:1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C6A9376FAA for ; Sun, 10 Dec 2017 16:55:21 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e67:d601:d060:873d:4a39:9eeb] (p2003008C2E67D601D060873D4A399EEB.dip0.t-ipconnect.de [IPv6:2003:8c:2e67:d601:d060:873d:4a39:9eeb]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3yvsdz1fgXz4W3 for ; Sun, 10 Dec 2017 17:55:19 +0100 (CET) From: Michael Grimm Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: [IPsec] Weird performance issue via IPsec/racoon tunnel Message-Id: <7A6EF712-920E-40BF-B155-113EE6C00AEA@ellael.org> Date: Sun, 10 Dec 2017 17:55:18 +0100 To: freebsd-net@FreeBSD.org X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Dec 2017 16:55:22 -0000 Hi I do run an IPsec/racoon tunnel between two servers (11.1-STABLE #0 = r326663). Some days ago I did migrate one of my servers from bare metal = to a public cloud instance. Now I do observe weird performance issues = from new to old server: ifconfig (OLD server, bare metal): ix0: flags=3D8843 metric = 0 mtu 1500 = options=3De407bb ifconfig (NEW server, cloud instance): vtnet0: flags=3D8843 = metric 0 mtu 1500 = options=3D6c07bb Immediately after booting of NEW (test file has 10 MB) I do observe the = following: #) scp OLD to NEW via ssh/internet: 16.7 MB/s #) scp NEW to OLD via ssh/internet: 17.4 MB/s #) scp NEW to OLD via IPsec tunnel: -> 65.8 KB/s ! #) scp OLD to NEW via IPsec tunnel: 16.5 MB/s Now I do a "ifconfig vtnet0 mtu 1500 up" and can observe very similar = performance. *BUT* if I do a "ifconfig vtnet0 mtu 1450 up ; ifconfig vtnet0 mtu 1500 = up" I do observe: #) scp NEW to OLD via IPsec tunnel: 17.1 MB/s ! #) scp OLD to NEW via IPsec tunnel: 16.9 MB/s I did monitor "tcpdump -i ix0 -vv esp" at the OLD sever and do get many: 16:22:24.370486 IP (tos 0x8, ttl 64, id 17394, offset 0, flags = [none], proto ESP (50), \ length 140, bad cksum 0 (->b110)!) "OLD" > "NEW": ESP(spi=3D0x0d83dae4,seq=3D0x3a8d9a), length = 120 At the NEW server I do not observe those checksum errors at all. *BUT* I = do see these error even after regaining full performance by modifying = the MTU from 1500 to 1450 and back to 1500! Well, I do have to admit that I do not have enough knowledge about = networking to find out by myself what to debug/modify next. Any help is highly appreciated. Thanks in advance, Michael