From owner-freebsd-net@freebsd.org Tue Jan 2 20:47:18 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2AEA1EB8A82; Tue, 2 Jan 2018 20:47:18 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from mail-lf0-x233.google.com (mail-lf0-x233.google.com [IPv6:2a00:1450:4010:c07::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9D62C72976; Tue, 2 Jan 2018 20:47:17 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by mail-lf0-x233.google.com with SMTP id o26so42668212lfc.10; Tue, 02 Jan 2018 12:47:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=afRxBTHA7+rHM5ITvQ0QIDpnRk263mQEpx7Yhb0aYW8=; b=Ei5ERtnqmcdLovigglQcmgiYnb8BWImVFLDfEbhI+916ZHTOVW22x52QtB0Q3vSGbZ s/GUh9+QTvAWnmq9JPrNEaJz1ZAiVpRyyvURQOOkcx+k9C7WC2P96yUDbkuVTAbQDBEw VjA71cVULZfdNOd8LgxDciG+uf7/8edtQ+WV8r6gWIEEKG1Ai6VVVlEM+mSn5wkRMRnj ae3NzvxpO7zYLEKVAbwEjVZgpaUt+H0pDNd826UluL/qiNJ/vkuy9ev23ygAKRvS8ZC8 V03/w+4SUU2xoWV4E0R+QIvhBNmvMFMbvO33t6wrbfdVoGHWdFDSKbRzpMZ+Xy91sc1u yoXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=afRxBTHA7+rHM5ITvQ0QIDpnRk263mQEpx7Yhb0aYW8=; b=ALU+gqcPlgZWpjOBzg8BTN8kHZO3YMBiAYK1a0GbRPl/giFL2w5AoFAv7ztMhAPqTj ZD+/MQsg4USY45yI+pAQroQTarqPJHbFZytRgqYxMd8D77nNZFyDZbaTDQ9LVZ07W5H4 YGIpBgKpZkdhHoMaJov9mxlP+g0h9nhZoTLqOWb1W4XWU6LREmKDJpbubD+D6oHalStr fliIqho+0LOlVFzCk4rTpESft//SDaraAhb9NQUvvByQT5OZ+lfQQc2QnbS5ziMlCVsu rsnoEG0sJdSoPIq3NSlKshra+3N/2hBliHVNBrogLUJF3mMdy13kX0HNbtfFRoRWBJw0 qPxg== X-Gm-Message-State: AKGB3mKrIw91xs80Du7XFOZOXOUI5hQlrBrrHPmFPF1lYvBLneeEK6+W WaR2dJIrItEO6zbWFASRAwDZc8IYr8gPmS48Og81FQ== X-Google-Smtp-Source: ACJfBosmqZdAk9kHB4yTXynKiUJ6sboTM29k70zCkaxPDbRjWZZopREkRI0fJLU3RAC9NkjS6ipNq4hnb3CvlsYnTUw= X-Received: by 10.25.193.197 with SMTP id r188mr18507815lff.43.1514926035619; Tue, 02 Jan 2018 12:47:15 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.163.207 with HTTP; Tue, 2 Jan 2018 12:47:14 -0800 (PST) Received: by 10.25.163.207 with HTTP; Tue, 2 Jan 2018 12:47:14 -0800 (PST) In-Reply-To: References: <4F5EE3F6-0163-4435-8726-56B0D4AE9FAF@ellael.org> <8102F5FD-DCFC-4EF8-A443-9E6C9EB1F467@ellael.org> <8C8A172B-4D4F-4066-8B94-EF5F59E2D345@ellael.org> <5A3D67EC.6010907@grosbein.net> <53687746-C487-4712-AA52-DE86CE70FDEF@ellael.org> <5FD6CE98-601B-46B7-B598-83BE5A31200A@ellael.org> From: Freddie Cash Date: Tue, 2 Jan 2018 12:47:14 -0800 Message-ID: Subject: Re: [SOLVED] performance issue within VNET jail To: Michael Grimm Cc: "Bjoern A. Zeeb" , freebsd-net , freebsd-pf@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jan 2018 20:47:18 -0000 On Dec 23, 2017 6:06 AM, "Michael Grimm" wrote: I will skip these questions for the time being, because I did solve my issue 15 minutes before your mail ;-) And I feel sorry for all your now "wasted" efforts in trying to help me. As I am using vtnet interface in a cloud environment (Public Cloud by OVH) I did read the vtnet(4) man pages and stumbled about "LOADER TUNABLES" like= : hw.vtnet.lro_disable hw.vtnet.X.lro_disable This tunable disables LRO. The default value is 0. Well, without knowing and understanding the implications of those loader tunables I did disabled them step by step, and bingo, setting =E2=80=A6 hw.vtnet.lro_disable=3D"1" =E2=80=A6 in /boot/loader.conf" and performance is back from KB/s to MB/s. I really do not understand what I have done and why it is working and whether that will have negative implications for my servers. Perhaps someone of you experts could help me understand it. I don't know all the specifics, but PF and IPFW really like to work on individual packets from NICs, especially when doing NAT. They don't play well with batched receives via LRO and batch transmits via TSO. Disabling those on the NICs being used for packet filtering greatly improves throughout. Not sure about PF man page, but pretty sure the IPFW man page mentions disabling those.