From owner-freebsd-net@freebsd.org Fri Dec 22 19:12:11 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D37EE8407E; Fri, 22 Dec 2017 19:12:11 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [91.121.41.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E81B63F3E; Fri, 22 Dec 2017 19:12:10 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e03:dc01:c8f8:8a2b:f09d:2a5a] (p2003008C2E03DC01C8F88A2BF09D2A5A.dip0.t-ipconnect.de [IPv6:2003:8c:2e03:dc01:c8f8:8a2b:f09d:2a5a]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z3J6B0y89z3DS; Fri, 22 Dec 2017 20:12:02 +0100 (CET) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: performance issue within VNET jail From: Michael Grimm In-Reply-To: Date: Fri, 22 Dec 2017 20:11:59 +0100 Cc: Kristof Provost , Eugene Grosbein Content-Transfer-Encoding: quoted-printable Message-Id: <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> To: freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Dec 2017 19:12:11 -0000 Kristof Provost wrote: > I run a very similar setup (although on CURRENT), and see no = performance issues from my jails. In utter despair I did upgrade one server to CURRENT (#327076) today, = but that hasn't been successful :-( Ok, right now I do know: (#) there is *no* performance loss (TCP) when: (-) fetching files from outside through PF/extIF to host (-) fetching files from partner server host via IPSEC tunnel = bound to extIF (ESP) to host (-) fetching files from partner server host via IPSEC tunnel = bound to extIF (ESP) to jail via bridge (-) fetching files from partner server jail via bridge and then = via IPSEC tunnel bound to extIF (ESP) to host (-) fetching files from partner server jail via bridge and then = via IPSEC tunnel bound to extIF (ESP) and then via bridge to jail (#) there is a *dramatic* performance loss (TCP) when: (-) fetching files from outside through PF/extIF via bridge to = jail (#) I did try to tweak the following settings *without* success: (-) sysctl net.inet.tcp.tso=3D0=20 (-) sysctl net.link.bridge.pfil_onlyip=3D0 (-) sysctl net.link.bridge.pfil_bridge=3D0 (-) sysctl net.link.bridge.pfil_member=3D0=20 (-) reducing mtu to 1400 (1490 before) on all interfaces extIF, = bridge, epairXs (-) deactivating "scrub in all" and "scrub out on $extIF all = random-id" in /etc/pf.conf (-) setting "set require-order yes" and "set require-order no" = in /etc/pf.conf [1] [1] I do see more a lot of out-of-order packages within a jail "netstat = -s -p tcp" after those slow downloads, but not after downloads via IPSEC = tunnel from partner host. That leads me to the conclusions: (#) the bridge is not to blame (#) it's either the PF/NATing or something else, right? Thanks for your suggestions so far, but I am lost here. Any ideas? Regards, Michael