From owner-freebsd-net@FreeBSD.ORG Wed Sep 4 12:50:12 2013 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id B7C3FA2C for ; Wed, 4 Sep 2013 12:50:12 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 7EE4220BF for ; Wed, 4 Sep 2013 12:50:12 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqIEAAwsJ1KDaFve/2dsb2JhbABaFoMmUYMovguBPHSCJAEBBAEjVgUWGAICDRkCWQYTCYdzBgyoE5FEgSmOGTQHgmmBNAOpW4M8IIFu X-IronPort-AV: E=Sophos;i="4.89,1021,1367985600"; d="scan'208";a="49418467" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-jnhn.mail.uoguelph.ca with ESMTP; 04 Sep 2013 08:50:05 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 9969079283; Wed, 4 Sep 2013 08:50:05 -0400 (EDT) Date: Wed, 4 Sep 2013 08:50:05 -0400 (EDT) From: Rick Macklem To: David Wolfskill Message-ID: <979862494.17918795.1378299005617.JavaMail.root@uoguelph.ca> In-Reply-To: <20130903192734.GA19406@albert.catwhisker.org> Subject: Re: TSO and FreeBSD vs Linux MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: FreeBSD Net X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Sep 2013 12:50:12 -0000 David Wolfskill wrote: > On Wed, Aug 21, 2013 at 07:12:38PM +0200, Andre Oppermann wrote: > > On 13.08.2013 19:29, Julian Elischer wrote: > > > I have been tracking down a performance embarrassment on AMAZON > > > EC2 and have found it I think. > > > Our OS cousins over at Linux land have implemented some > > > interesting behaviour when TSO is in use. > > > > There used to be a different problem with EC2 and FreeBSD TSO. The > > Xen hypervisor > > doesn't like large 64K TSO bursts we generate, the drivers drops > > the whole TSO chain, > > TCP gets upset and turns off TSO alltogether leaving the connection > > going at one > > packet a time as in the old days. > > ... > > My apologies for jumping in so late -- I'm not subscribed to -net@. > > At work, I received a new desktop machine a few months ago; here's a > recent history of what it has been running: > > FreeBSD 9.2-PRERELEASE #4 r254801M/254827:902501: Sun Aug 25 > 05:15:29 PDT 2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF > amd64 > FreeBSD 9.2-PRERELEASE #5 r255066M/255091:902503: Sat Aug 31 > 11:58:53 PDT 2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF > amd64 > FreeBSD 9.2-PRERELEASE #5 r255104M/255115:902503: Sun Sep 1 > 05:02:12 PDT 2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF > amd64 > > Now, I like to have a "private playground" for doing things with > machines, so I make use of both em(4) NICs on the machine: em0 > connects > to the rest of the work network; em1 is connected to a switch I > brought > in from home, and to which I connect "other things" (such as my > laptop). > And because I'm fairly comfortable with them, I use IPFW & natd. For > some folks here, none of that should come as a surprise. :-}) > > For reference, the em(4) devices in question are: > > em0@pci0:0:25:0: class=0x020000 card=0x060d15d9 > chip=0x10ef8086 rev=0x06 hdr=0x00 > vendor = 'Intel Corporation' > device = '82578DM Gigabit Network Connection' > > and > > em1@pci0:3:0:0: class=0x020000 card=0x060d15d9 chip=0x10d38086 > rev=0x00 hdr=0x00 > vendor = 'Intel Corporation' > device = '82574L Gigabit Network Connection' > > > > I noticed that when I tried to write files to NFS, I could write > small > files OK, but larger ones seemed to ... hang. > > Note: We don't use jumbo frames. (Work IT is convinced that they > don't help. I'm trying to better-understand their reasoning.) > > Further poking around showed that (under the above conditions): > * natd CPU% was climbing as more of the file was copied, up to 2^21 > bytes. (At that point, nothing further was saved on NFS.) > * dhcpd CPU% was also climbing. I tried killing that, but doing so > didn't affect the other results. (Killing natd made connectivity > cease, given the IPFW rules in effect.) > * Performing a tcpdump while trying to copy a file of length > 117709618 > showed lots of TCP retransmissions. In fact, I'd hazard that every > TCP > packet was getting retransmitted. > * "ifconfig -v em0" showed flags TSO4 & VLAN_HWTSO turned on. > * "sysctl net.inet.tcp.tso" showed "1" -- enabled. > > As soon as I issued "sudo net.inet.tcp.tso=0" ... the copy worked > without > a hitch or a whine. And I was able to copy all 117709618 bytes, not > just > 2097152 (2^21). > > Is the above expected? It came rather as a surprise to me. > Not surprising to me, I'm afraid. When there are serious NFS problems like this, it is often caused by a network fabric issue and broken TSO is at the top of the list w.r.t. cause. rick > Peace, > david > -- > David H. Wolfskill david@catwhisker.org > Taliban: Evil cowards with guns afraid of truth from a 14-year old > girl. > > See http://www.catwhisker.org/~david/publickey.gpg for my public key. >