From owner-freebsd-jail@freebsd.org Fri Dec 22 20:16:07 2017 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F2C7E871C5; Fri, 22 Dec 2017 20:16:07 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B57B650B2; Fri, 22 Dec 2017 20:16:06 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id vBMKFr99024657 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 22 Dec 2017 21:15:53 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: trashcan@ellael.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id vBMKFgPO076905 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Sat, 23 Dec 2017 03:15:42 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: performance issue within VNET jail To: Michael Grimm , freebsd-net@freebsd.org, freebsd-jail@FreeBSD.org References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> From: Eugene Grosbein Message-ID: <5A3D67EC.6010907@grosbein.net> Date: Sat, 23 Dec 2017 03:15:40 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Dec 2017 20:16:07 -0000 23.12.2017 2:11, Michael Grimm wrote: > Kristof Provost wrote: > >> I run a very similar setup (although on CURRENT), and see no performance issues from my jails. > > In utter despair I did upgrade one server to CURRENT (#327076) today, but that hasn't been successful :-( > > Ok, right now I do know: > > (#) there is *no* performance loss (TCP) when: > > (-) fetching files from outside through PF/extIF to host > (-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to host > (-) fetching files from partner server host via IPSEC tunnel bound to extIF (ESP) to jail via bridge > (-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) to host > (-) fetching files from partner server jail via bridge and then via IPSEC tunnel bound to extIF (ESP) and then via bridge to jail > > (#) there is a *dramatic* performance loss (TCP) when: > > (-) fetching files from outside through PF/extIF via bridge to jail > > (#) I did try to tweak the following settings *without* success: > > (-) sysctl net.inet.tcp.tso=0 > (-) sysctl net.link.bridge.pfil_onlyip=0 > (-) sysctl net.link.bridge.pfil_bridge=0 > (-) sysctl net.link.bridge.pfil_member=0 > (-) reducing mtu to 1400 (1490 before) on all interfaces extIF, bridge, epairXs > (-) deactivating "scrub in all" and "scrub out on $extIF all random-id" in /etc/pf.conf > (-) setting "set require-order yes" and "set require-order no" in /etc/pf.conf [1] > > [1] I do see more a lot of out-of-order packages within a jail "netstat -s -p tcp" after those slow downloads, but not after downloads via IPSEC tunnel from partner host. > > That leads me to the conclusions: > > (#) the bridge is not to blame > (#) it's either the PF/NATing or something else, right? > > Thanks for your suggestions so far, but I am lost here. Any ideas? It seems to me some kind of bug in the PF. I personally never tried it, I use ipfw and it works just fine. Maybe, you should try to switch to it too, at least for a test.