From owner-freebsd-net@FreeBSD.ORG Sat Jul 12 12:33:51 2014 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 02693413 for ; Sat, 12 Jul 2014 12:33:51 +0000 (UTC) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:130:3ffc::401:25]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id AACED2ECD for ; Sat, 12 Jul 2014 12:33:50 +0000 (UTC) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id 7C47525D3A85; Sat, 12 Jul 2014 12:33:47 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id 83A5DC22BFD; Sat, 12 Jul 2014 12:33:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id REo1yqjB-j1T; Sat, 12 Jul 2014 12:33:44 +0000 (UTC) Received: from [IPv6:fde9:577b:c1a9:4410:80b3:3e87:431:f220] (unknown [IPv6:fde9:577b:c1a9:4410:80b3:3e87:431:f220]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id CB157C22BA9; Sat, 12 Jul 2014 12:33:43 +0000 (UTC) Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: tuning routing using cxgbe and T580-CR cards? From: "Bjoern A. Zeeb" In-Reply-To: Date: Sat, 12 Jul 2014 12:33:23 +0000 Content-Transfer-Encoding: quoted-printable Message-Id: <68918930-4FEC-413B-AB5E-B544C936D54F@lists.zabbadoz.net> References: <53C01EB5.6090701@gmail.com> <01AABF44-4801-45B5-9509-1CA7BAA3CB30@lists.zabbadoz.net> To: =?windows-1252?Q?Olivier_Cochard-Labb=E9?= X-Mailer: Apple Mail (2.1878.6) Cc: FreeBSD Net , John Jasem , Navdeep Parhar X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 12 Jul 2014 12:33:51 -0000 On 12 Jul 2014, at 12:17 , Olivier Cochard-Labb=E9 = wrote: > On Fri, Jul 11, 2014 at 8:03 PM, Bjoern A. Zeeb < > bzeeb-lists@lists.zabbadoz.net> wrote: >=20 >> If you are primarily forwarding packets (you say "routing" multiple = times) >> the first thing you should do is turn off LRO and TSO on all ports. >>=20 >=20 > Hi Bjoern, >=20 > I was not aware of disabling LRO+TSO for forwarding packet. > If I read correctly the wikipedia page of LRO[1]: Disabling LRO is not = a > concern of performance but only of not breaking the end-to-end = principle, > right ? > But regarding TSO[2]: It should improve performance only between the = TCP > and IP layer. But paquet forwarded didn't have to cross TCP<->IP = layer, > then disabling TSO should not impact performance, right ? For forwarding it means that you are re-assembling a packet on receive, = buffering multiple, etc, then hand them up the stack, only to find that = you are sending it out again, and thus you break them into multiple = packets again. In other words: you do a lot more work and add latency = than you need/want. I seem to remember that we added the knob to automatically disable our = soft-LRO when forwarding is turned on (but I haven=92t checked if I = really did). If we did, at least for soft-LRO you won=92t notice a = difference indeed. > - Multi-flows (different UDP ports) of small packet (60B) at about = 10Mpps > =85 > No difference proven at 95.0% confidence >=20 > =3D> There is not difference: Then I can disable LRO for respecting = the > end-to-end principle. But why disabling TSO ? Try TCP flows. =97=20 Bjoern A. Zeeb "Come on. Learn, goddamn it.", WarGames, 1983