Date: Mon, 11 Dec 2017 00:34:55 +0700 From: Eugene Grosbein <eugen@grosbein.net> To: Michael Grimm <trashcan@ellael.org>, freebsd-net@FreeBSD.org Subject: Re: [IPsec] Weird performance issue via IPsec/racoon tunnel Message-ID: <5A2D703F.8040004@grosbein.net> In-Reply-To: <7A6EF712-920E-40BF-B155-113EE6C00AEA@ellael.org> References: <7A6EF712-920E-40BF-B155-113EE6C00AEA@ellael.org>
next in thread | previous in thread | raw e-mail | index | archive | help
10.12.2017 23:55, Michael Grimm wrote: > Hi > > I do run an IPsec/racoon tunnel between two servers (11.1-STABLE #0 r326663). Some days ago I did migrate one of my servers from bare metal to a public cloud instance. Now I do observe weird performance issues from new to old server: > > ifconfig (OLD server, bare metal): > ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=e407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,\ > TSO4,TSO6,LRO,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> > > ifconfig (NEW server, cloud instance): > vtnet0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 > options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,\ > TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> > > Immediately after booting of NEW (test file has 10 MB) I do observe the following: > > #) scp OLD to NEW via ssh/internet: 16.7 MB/s > #) scp NEW to OLD via ssh/internet: 17.4 MB/s > #) scp NEW to OLD via IPsec tunnel: -> 65.8 KB/s ! > #) scp OLD to NEW via IPsec tunnel: 16.5 MB/s > > Now I do a "ifconfig vtnet0 mtu 1500 up" and can observe very similar performance. > > *BUT* if I do a "ifconfig vtnet0 mtu 1450 up ; ifconfig vtnet0 mtu 1500 up" I do observe: > > #) scp NEW to OLD via IPsec tunnel: 17.1 MB/s ! > #) scp OLD to NEW via IPsec tunnel: 16.9 MB/s > > I did monitor "tcpdump -i ix0 -vv esp" at the OLD sever and do get many: > > 16:22:24.370486 IP (tos 0x8, ttl 64, id 17394, offset 0, flags [none], proto ESP (50), \ > length 140, bad cksum 0 (->b110)!) > "OLD" > "NEW": ESP(spi=0x0d83dae4,seq=0x3a8d9a), length 120 > > At the NEW server I do not observe those checksum errors at all. *BUT* I do see these error even after regaining full performance by modifying the MTU from 1500 to 1450 and back to 1500! > > Well, I do have to admit that I do not have enough knowledge about networking to find out by myself what to debug/modify next. > > Any help is highly appreciated. "bad cksum 0" is pretty normal for traffic going out via interface supporting hardware checksum offload, so kernel skips computing checksum before passing packets to the NIC. Your problem more likely is due to fragmented ESP packets. It's not uncommon when cloud IP stack or ISP infrastructure drop high percentage of fragmented ESP packets because they are not optimized for such packets, e.g. router has to process them in software instead of hardware like non-fragmented packets are processed. When you lower MTU of vtnet enough to make encapsulated packets (payload+overhead) <=1500 bytes, resulted ESP packets have not be fragmented and pass just fine. To verify if it's your case, you should run two tcpdump commands, one at sending side and another at receiving size and compare outputs to see if *every* outgoing packet reaches its destination or not.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5A2D703F.8040004>