Date: Thu, 21 Dec 2017 21:24:47 +0100 From: Michael Grimm <trashcan@ellael.org> To: freebsd-jail@FreeBSD.org, freebsd-net@freebsd.org Subject: performance issue within VNET jail Message-ID: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org>
next in thread | raw e-mail | index | archive | help
Hi
[ I did recently migrate my servers from bare metal to cloud instances =
(OpenStack at OVH) ]
[ FreeBSD 11.1-STABLE #0 r327055 =
]
My setup is as follows and didn't change for the last couple of years:
extIF0/pf/NAT <=E2=80=94> epairXa (bridge0) epairXb <-> jail
Downloading a file (by wget) at the host is around 30 MB/s, and an =
example tcpdump at extIF0 looks as follows:
19:32:10.711769 IP (tos 0x20, ttl 56, id 37539, offset 0, flags [DF], =
proto TCP (6), length 8680)
remote.http > myhost.14367: Flags [.], cksum 0x64ed (incorrect -> =
0x3223), seq 5753:14381, ack 146, win 235, options [nop,nop,TS val =
1007145732 ecr 3995852], length 8628: HTTP
19:32:10.713851 IP (tos 0x20, ttl 56, id 37545, offset 0, flags [DF], =
proto TCP (6), length 1490)
remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> =
0x8d1e), seq 14381:15819, ack 146, win 235, options [nop,nop,TS val =
1007145732 ecr 3995852], length 1438: HTTP
19:32:10.713899 IP (tos 0x20, ttl 56, id 37546, offset 0, flags [DF], =
proto TCP (6), length 1490)
remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> =
0x6ade), seq 15819:17257, ack 146, win 235, options [nop,nop,TS val =
1007145732 ecr 3995852], length 1438: HTTP
19:32:10.713934 IP (tos 0x20, ttl 56, id 37547, offset 0, flags [DF], =
proto TCP (6), length 1490)
remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> =
0x1173), seq 17257:18695, ack 146, win 235, options [nop,nop,TS val =
1007145732 ecr 3995852], length 1438: HTTP
19:32:10.713962 IP (tos 0x20, ttl 56, id 37548, offset 0, flags [DF], =
proto TCP (6), length 1490)
remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> =
0xcf7a), seq 18695:20133, ack 146, win 235, options [nop,nop,TS val =
1007145732 ecr 3995852], length 1438: HTTP
When downloading the very same file within a VIMAGE jail the performance =
drops to around 80 KB/s, quite a dramatic loss. An example tcpdump at =
exitIF0 looks as follows:
19:34:36.284175 IP (tos 0x0, ttl 56, id 28618, offset 0, flags [DF], =
proto TCP (6), length 2948)
remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> =
0x4478), seq 1449:4345, ack 146, win 235, options [nop,nop,TS val =
1007182125 ecr 4141429], length 2896: HTTP
19:34:36.481904 IP (tos 0x0, ttl 56, id 28620, offset 0, flags [DF], =
proto TCP (6), length 1500)
remote.http > myhost.63382: Flags [.], cksum 0xd11d (correct), seq =
1449:2897, ack 146, win 235, options [nop,nop,TS val 1007182175 ecr =
4141429], length 1448: HTTP
19:34:36.484109 IP (tos 0x0, ttl 56, id 28621, offset 0, flags [DF], =
proto TCP (6), length 2948)
remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> =
0x2e5b), seq 15929:18825, ack 146, win 235, options [nop,nop,TS val =
1007182175 ecr 4141629], length 2896: HTTP
19:34:36.682006 IP (tos 0x0, ttl 56, id 28623, offset 0, flags [DF], =
proto TCP (6), length 1500)
remote.http > myhost.63382: Flags [.], cksum 0x4ab6 (correct), seq =
2897:4345, ack 146, win 235, options [nop,nop,TS val 1007182225 ecr =
4141629], length 1448: HTTP
19:34:36.684159 IP (tos 0x0, ttl 56, id 28624, offset 0, flags [DF], =
proto TCP (6), length 2948)
remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> =
0xd7db), seq 18825:21721, ack 146, win 235, options [nop,nop,TS val =
1007182225 ecr 4141829], length 2896: HTTP
A tcpdump at epairXa looks comparable.
I did reduce all MTU settings at the involved interfaces from their =
initial settings (1490) to an experimental setting of 1400, just to be =
on the save side, to no avail. (FYI: I did have to reduce from 1500 to =
1490 to please IPSec after migration from bare metal to cloud =
infrastructure.)
Then, I did test the following settings found in the Net, to no avail =
either:
sysctl net.inet.tcp.tso=3D0
sysctl net.link.bridge.pfil_onlyip=3D0
sysctl net.link.bridge.pfil_bridge=3D0
sysctl net.link.bridge.pfil_member=3D0
sysctl net.add_addr_allfibs=3D0
I do have to admit that I am lost here, and that I cannot think about =
what is going wrong. The last download I did try at my old severs has =
been some weeks ago. Ever since I did upgrade FreeBSD 11.1-STABLE, and I =
did move my infrastructure from bare metal to cloud, thus I cannot test =
anymore if my old servers would have shown that performance issue in the =
meantime.
Thus any feedback is highly recommended!
Thanks in advance and regards,
Michael
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F5EE3F6-0163-4435-8726-56B0D4AE9FAF>