From owner-freebsd-net@freebsd.org Thu Dec 21 20:25:17 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 157F3E881FE; Thu, 21 Dec 2017 20:25:17 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [IPv6:2001:41d0:302:1100::7:9a96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B528F76B5A; Thu, 21 Dec 2017 20:25:16 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531] (p2003008C2E04E50140CCD10E17C00531.dip0.t-ipconnect.de [IPv6:2003:8c:2e04:e501:40cc:d10e:17c0:531]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z2jmc3pxtzDl2; Thu, 21 Dec 2017 21:24:48 +0100 (CET) From: Michael Grimm Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: performance issue within VNET jail Message-Id: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> Date: Thu, 21 Dec 2017 21:24:47 +0100 To: freebsd-jail@FreeBSD.org, freebsd-net@freebsd.org X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Dec 2017 20:25:17 -0000 Hi [ I did recently migrate my servers from bare metal to cloud instances = (OpenStack at OVH) ] [ FreeBSD 11.1-STABLE #0 r327055 = ] My setup is as follows and didn't change for the last couple of years: extIF0/pf/NAT <=E2=80=94> epairXa (bridge0) epairXb <-> jail Downloading a file (by wget) at the host is around 30 MB/s, and an = example tcpdump at extIF0 looks as follows: 19:32:10.711769 IP (tos 0x20, ttl 56, id 37539, offset 0, flags [DF], = proto TCP (6), length 8680) remote.http > myhost.14367: Flags [.], cksum 0x64ed (incorrect -> = 0x3223), seq 5753:14381, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 8628: HTTP 19:32:10.713851 IP (tos 0x20, ttl 56, id 37545, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0x8d1e), seq 14381:15819, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713899 IP (tos 0x20, ttl 56, id 37546, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0x6ade), seq 15819:17257, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713934 IP (tos 0x20, ttl 56, id 37547, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0x1173), seq 17257:18695, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP 19:32:10.713962 IP (tos 0x20, ttl 56, id 37548, offset 0, flags [DF], = proto TCP (6), length 1490) remote.http > myhost.14367: Flags [.], cksum 0x48d7 (incorrect -> = 0xcf7a), seq 18695:20133, ack 146, win 235, options [nop,nop,TS val = 1007145732 ecr 3995852], length 1438: HTTP When downloading the very same file within a VIMAGE jail the performance = drops to around 80 KB/s, quite a dramatic loss. An example tcpdump at = exitIF0 looks as follows: 19:34:36.284175 IP (tos 0x0, ttl 56, id 28618, offset 0, flags [DF], = proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> = 0x4478), seq 1449:4345, ack 146, win 235, options [nop,nop,TS val = 1007182125 ecr 4141429], length 2896: HTTP 19:34:36.481904 IP (tos 0x0, ttl 56, id 28620, offset 0, flags [DF], = proto TCP (6), length 1500) remote.http > myhost.63382: Flags [.], cksum 0xd11d (correct), seq = 1449:2897, ack 146, win 235, options [nop,nop,TS val 1007182175 ecr = 4141429], length 1448: HTTP 19:34:36.484109 IP (tos 0x0, ttl 56, id 28621, offset 0, flags [DF], = proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> = 0x2e5b), seq 15929:18825, ack 146, win 235, options [nop,nop,TS val = 1007182175 ecr 4141629], length 2896: HTTP 19:34:36.682006 IP (tos 0x0, ttl 56, id 28623, offset 0, flags [DF], = proto TCP (6), length 1500) remote.http > myhost.63382: Flags [.], cksum 0x4ab6 (correct), seq = 2897:4345, ack 146, win 235, options [nop,nop,TS val 1007182225 ecr = 4141629], length 1448: HTTP 19:34:36.684159 IP (tos 0x0, ttl 56, id 28624, offset 0, flags [DF], = proto TCP (6), length 2948) remote.http > myhost.63382: Flags [.], cksum 0x5df6 (incorrect -> = 0xd7db), seq 18825:21721, ack 146, win 235, options [nop,nop,TS val = 1007182225 ecr 4141829], length 2896: HTTP A tcpdump at epairXa looks comparable. I did reduce all MTU settings at the involved interfaces from their = initial settings (1490) to an experimental setting of 1400, just to be = on the save side, to no avail. (FYI: I did have to reduce from 1500 to = 1490 to please IPSec after migration from bare metal to cloud = infrastructure.) Then, I did test the following settings found in the Net, to no avail = either: sysctl net.inet.tcp.tso=3D0 sysctl net.link.bridge.pfil_onlyip=3D0 sysctl net.link.bridge.pfil_bridge=3D0 sysctl net.link.bridge.pfil_member=3D0 sysctl net.add_addr_allfibs=3D0 I do have to admit that I am lost here, and that I cannot think about = what is going wrong. The last download I did try at my old severs has = been some weeks ago. Ever since I did upgrade FreeBSD 11.1-STABLE, and I = did move my infrastructure from bare metal to cloud, thus I cannot test = anymore if my old servers would have shown that performance issue in the = meantime. Thus any feedback is highly recommended! Thanks in advance and regards, Michael