From owner-freebsd-net@freebsd.org Sat Dec 23 14:06:16 2017 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2E8A0EA67FC; Sat, 23 Dec 2017 14:06:16 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from mx1.enfer-du-nord.net (mx1.enfer-du-nord.net [IPv6:2001:41d0:302:1100::7:9a96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D59B9644D8; Sat, 23 Dec 2017 14:06:15 +0000 (UTC) (envelope-from trashcan@ellael.org) Received: from [IPv6:2003:8c:2e30:b801:3cc5:4d29:36db:5290] (p2003008C2E30B8013CC54D2936DB5290.dip0.t-ipconnect.de [IPv6:2003:8c:2e30:b801:3cc5:4d29:36db:5290]) by mx1.enfer-du-nord.net (Postfix) with ESMTPSA id 3z3nGr5lqNz3Cq; Sat, 23 Dec 2017 15:06:12 +0100 (CET) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: [SOLVED] performance issue within VNET jail From: Michael Grimm In-Reply-To: Date: Sat, 23 Dec 2017 15:06:10 +0100 Cc: freebsd-net@freebsd.org, freebsd-pf@FreeBSD.org Content-Transfer-Encoding: quoted-printable Message-Id: <5FD6CE98-601B-46B7-B598-83BE5A31200A@ellael.org> References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> <5A3D67EC.6010907@grosbein.net> <53687746-C487-4712-AA52-DE86CE70FDEF@ellael.org> To: "Bjoern A. Zeeb" X-Virus-Scanned: clamav-milter 0.99.2 at mail X-Virus-Status: Clean X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Dec 2017 14:06:16 -0000 Bjoern A. Zeeb wrote: >=20 > On 22 Dec 2017, at 20:30, Michael Grimm wrote: >> Hi =E2=80=94 >>=20 >> [ I am including freebsd-pf@FreeBSD.org now and removing = freebsd-jail@FreeBSD.org ] >> [ Thread starts at = https://lists.freebsd.org/pipermail/freebsd-net/2017-December/049470.html = ] >>>>=20 >>>> (#) there is a *dramatic* performance loss (TCP) when: >>>>=20 >>>> (-) fetching files from outside through PF/extIF via bridge to = jail > =E2=80=A6 >>>>=20 >>>> Thanks for your suggestions so far, but I am lost here. Any ideas? >>>=20 >>> It seems to me some kind of bug in the PF. >>> I personally never tried it, I use ipfw and it works just fine. >>=20 >> Before testing IPFW (which I have never used before) I'd like to ask = the experts in freebsd-pf@FreeBSD.org about possible tests/tweaks = regarding PF. >=20 > OK, too complicated setups; I am not getting it fully. ;-) > Can you please just describe the one case that doesn=E2=80=99t work = well in all detail and ignore all the others for a moment? >=20 > (a) what=E2=80=99s the external host interface? vtnet > (b) pf runs on the base system? yes > (c) you are bridging into a VNET-jail? How exactly? Are you bridging = to epairs? yes, I am bridging epairs > (d) where exactly are you NATing? I am NATing IPv4 and IPv6 at the host' PF ffirewall > (e) why are you bridging and NATing? That makes little sense to me. = Couldn=E2=80=99t you NAT and forward or just bridge? hmm, that has been developed by myself over the years. I do "consider" = my jails as jails with their own network stack, like isolated "VM".=20 > (f) what=E2=80=99s inside the VNET jail? Another pf or anything? no more firewall, my jails are merely service jails (dns, mail, web, = =E2=80=A6) > (g) out of curiosity, does dmesg on the base system indicate anything? No. > To understand your performance problem better: >=20 > (1) you are doing a fetch of a rather large file to test from within = the VNET jail? Or what are you fetching? Are you using fetch? yes, I do something like the following with the jail: wget = https://download.freebsd.org/ftp/releases/ISO-IMAGES/11.1/FreeBSD-11.1-REL= EASE-amd64-bootonly.iso -O /dev/null > (2) if you fetch from within the same VNET jail does that perform? > (3) if you fetch something to the VNET jail from the base system just = going through your internal setup but not leaving the machine, does that = still perform? > (4) if you fetch something to the VNET jail from the same LAN (if = possible to test) does that perform? > (5) if you fetch something to the VNET jail from a close by location = does that make a difference to something on the other side of the = planet? I will skip these questions for the time being, because I did solve my = issue 15 minutes before your mail ;-) And I feel sorry for all your now = "wasted" efforts in trying to help me. As I am using vtnet interface in a cloud environment (Public Cloud by = OVH) I did read the vtnet(4) man pages and stumbled about "LOADER = TUNABLES" like: hw.vtnet.lro_disable hw.vtnet.X.lro_disable This tunable disables LRO. The default value is 0. Well, without knowing and understanding the implications of those loader = tunables I did disabled them step by step, and bingo, setting =E2=80=A6 hw.vtnet.lro_disable=3D"1" =E2=80=A6 in /boot/loader.conf" and performance is back from KB/s to = MB/s. I really do not understand what I have done and why it is working and = whether that will have negative implications for my servers. Perhaps = someone of you experts could help me understand it. Because I am leaving in some hours for Xmas vacations, I won't be able = to come back to this issue for some days now.=20 I'd like to thank all of you for your patience and help, and: Merry Christmas and with kind regards, Michael