From owner-freebsd-net@FreeBSD.ORG Mon Jul 14 15:34:58 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AE066AB8 for ; Mon, 14 Jul 2014 15:34:58 +0000 (UTC) Received: from mail-ie0-x22c.google.com (mail-ie0-x22c.google.com [IPv6:2607:f8b0:4001:c03::22c]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7225023EF for ; Mon, 14 Jul 2014 15:34:58 +0000 (UTC) Received: by mail-ie0-f172.google.com with SMTP id lx4so1334154iec.3 for ; Mon, 14 Jul 2014 08:34:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=W6AWiEEe9ru0sxNMKHJMPMKjGyih7j+bJ+0EKJt1fCA=; b=QAFQfYu7RtX6vFk+CTQ5z3ijeWnux0TBwLvDsmmA+tU3dmLHjrET1a06GiJAb86Hdv BzsulP4KirVoYwuFW6ZqXQteBzw8DuZyMvLIShDUGaSptd57ou4BzOeqK3hslA40W8SB fBNaos1qQ4MBGbKD1UpdQ06rcIh/iDGK8SY3KIrCtnw3jeireMtbVLc/IUbpDJNzIirz 4H36SAZa2b0cgVhbNQnzo4RCo3F/eVZAwtb6czwDaxe+NFD90HfRqkF9WD+TIRWs0mgC xYADRtb4ZGk2IMTj+t9EkrJO6DINr075V0Y6mcrQuyLVOBflS/53Vc8+uAqHeEjPr5Ik WYMA== X-Received: by 10.50.36.106 with SMTP id p10mr26863746igj.9.1405352097815; Mon, 14 Jul 2014 08:34:57 -0700 (PDT) Received: from [10.1.68.187] (gs-sv-1-49-ac1.gsfc.nasa.gov. [198.119.56.43]) by mx.google.com with ESMTPSA id n1sm7130038igp.1.2014.07.14.08.34.55 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 14 Jul 2014 08:34:56 -0700 (PDT) Message-ID: <53C3F89D.1020503@gmail.com> Date: Mon, 14 Jul 2014 11:34:53 -0400 From: John Jasem User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: "Bjoern A. Zeeb" , =?windows-1252?Q?Olivier_Cochard-Labb=E9?= Subject: Re: tuning routing using cxgbe and T580-CR cards? References: <53C01EB5.6090701@gmail.com> <01AABF44-4801-45B5-9509-1CA7BAA3CB30@lists.zabbadoz.net> <68918930-4FEC-413B-AB5E-B544C936D54F@lists.zabbadoz.net> In-Reply-To: <68918930-4FEC-413B-AB5E-B544C936D54F@lists.zabbadoz.net> X-Enigmail-Version: 1.5.2 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Cc: FreeBSD Net , Navdeep Parhar X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 14 Jul 2014 15:34:58 -0000 Dropping lro on the interfaces decreased interrupt usage on the CPUs, as measured by top -CHIPSu, by 15-20%, at least from eyeballing it. It did not otherwise have an effect on packet rates. Thanks! On 07/12/2014 08:33 AM, Bjoern A. Zeeb wrote: > On 12 Jul 2014, at 12:17 , Olivier Cochard-Labbé wrote: > >> On Fri, Jul 11, 2014 at 8:03 PM, Bjoern A. Zeeb < >> bzeeb-lists@lists.zabbadoz.net> wrote: >> >>> If you are primarily forwarding packets (you say "routing" multiple times) >>> the first thing you should do is turn off LRO and TSO on all ports. >>> >> Hi Bjoern, >> >> I was not aware of disabling LRO+TSO for forwarding packet. >> If I read correctly the wikipedia page of LRO[1]: Disabling LRO is not a >> concern of performance but only of not breaking the end-to-end principle, >> right ? >> But regarding TSO[2]: It should improve performance only between the TCP >> and IP layer. But paquet forwarded didn't have to cross TCP<->IP layer, >> then disabling TSO should not impact performance, right ? > For forwarding it means that you are re-assembling a packet on receive, buffering multiple, etc, then hand them up the stack, only to find that you are sending it out again, and thus you break them into multiple packets again. In other words: you do a lot more work and add latency than you need/want. > > I seem to remember that we added the knob to automatically disable our soft-LRO when forwarding is turned on (but I haven’t checked if I really did). If we did, at least for soft-LRO you won’t notice a difference indeed. > > >> - Multi-flows (different UDP ports) of small packet (60B) at about 10Mpps >> … >> No difference proven at 95.0% confidence >> >> => There is not difference: Then I can disable LRO for respecting the >> end-to-end principle. But why disabling TSO ? > Try TCP flows. > > — > Bjoern A. Zeeb "Come on. Learn, goddamn it.", WarGames, 1983 >