From owner-freebsd-net@FreeBSD.ORG Tue Sep 3 22:37:10 2013 Return-Path: <owner-freebsd-net@FreeBSD.ORG> Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 39AF1354 for <net@freebsd.org>; Tue, 3 Sep 2013 22:37:10 +0000 (UTC) (envelope-from kudzu@tenebras.com) Received: from mail-oa0-f51.google.com (mail-oa0-f51.google.com [209.85.219.51]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 02A2720D4 for <net@freebsd.org>; Tue, 3 Sep 2013 22:37:09 +0000 (UTC) Received: by mail-oa0-f51.google.com with SMTP id h1so7555974oag.10 for <net@freebsd.org>; Tue, 03 Sep 2013 15:37:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=+/hlADK1MrP4qI/OjlQOKXqJSBXZQ774kBH0X+rmNqM=; b=cgiIxT6VW7UqE9tZfQ29dOAlENMQqWl/79JJAvJbOcMieOiaDNSe45URX7cGYHdcvG MRzDP+z7zZrJWTAUxRxTGFi5Mf+hNwQDfwP0SbfDNjmSJss9tMIiYEMfKo8eIhGX+Rth T8WFj2ECGrBpZC+jnUXaJu+qyFJveSfZZhs5EPLSlmxot2XZwAyPn4i6jHOOq3m2cNwh zGAEfnW9Sqr9+ZseRZ7vFTcSDNwSk3VX/YMu68gIiliIvojy1ijfPbTo3KJ895s3QPcK SFLkspsre3ACVvbpLfAwOZPDHL6kvTGmrpT1BAsUfOqsEGddN8z8IiWiULX03AOoVHLE i2ow== X-Gm-Message-State: ALoCoQlzMrNSxKalDHZ2I0G7SKzCjv7H1FJJ8xeG6AKzpnM5iOx2oPSGITJYzf9irA9XRtp9O/0+ MIME-Version: 1.0 X-Received: by 10.60.133.71 with SMTP id pa7mr10218093oeb.44.1378247823291; Tue, 03 Sep 2013 15:37:03 -0700 (PDT) Received: by 10.60.21.69 with HTTP; Tue, 3 Sep 2013 15:37:03 -0700 (PDT) In-Reply-To: <CAJ-Vmo=97-EAC8ELV=VhdHPXmMa+GE5FxHy2ik7kj-S0S7VKcg@mail.gmail.com> References: <520A6D07.5080106@freebsd.org> <5214F506.3070706@freebsd.org> <20130903192734.GA19406@albert.catwhisker.org> <CAJ-Vmo=97-EAC8ELV=VhdHPXmMa+GE5FxHy2ik7kj-S0S7VKcg@mail.gmail.com> Date: Tue, 3 Sep 2013 18:37:03 -0400 Message-ID: <CAHu1Y71FbStZp=HfQEY+0d4PCMtmsrmjsppt3zs35aZ4nXEO3Q@mail.gmail.com> Subject: Re: TSO and FreeBSD vs Linux From: Michael Sierchio <kudzu@tenebras.com> To: FreeBSD Net <net@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org> List-Unsubscribe: <http://lists.freebsd.org/mailman/options/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe> List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net> List-Post: <mailto:freebsd-net@freebsd.org> List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help> List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>, <mailto:freebsd-net-request@freebsd.org?subject=subscribe> X-List-Received-Date: Tue, 03 Sep 2013 22:37:10 -0000 I must have discovered this and forgotten - all my AMIs have net.inet.tcp.tso=0 dev.xn.0.enable_lro=0 or ifconfig xn0 -tso -lro - M On Tue, Sep 3, 2013 at 4:56 PM, Adrian Chadd <adrian@freebsd.org> wrote: > ... this is bad behaviour. So yes, it needs to be chased up and repaired. > > Thanks for finding it out! > > > On 3 September 2013 12:27, David Wolfskill <david@catwhisker.org> wrote: > > > On Wed, Aug 21, 2013 at 07:12:38PM +0200, Andre Oppermann wrote: > > > On 13.08.2013 19:29, Julian Elischer wrote: > > > > I have been tracking down a performance embarrassment on AMAZON EC2 > > and have found it I think. > > > > Our OS cousins over at Linux land have implemented some interesting > > behaviour when TSO is in use. > > > > > > There used to be a different problem with EC2 and FreeBSD TSO. The Xen > > hypervisor > > > doesn't like large 64K TSO bursts we generate, the drivers drops the > > whole TSO chain, > > > TCP gets upset and turns off TSO alltogether leaving the connection > > going at one > > > packet a time as in the old days. > > > ... > > > > My apologies for jumping in so late -- I'm not subscribed to -net@. > > > > At work, I received a new desktop machine a few months ago; here's a > > recent history of what it has been running: > > > > FreeBSD 9.2-PRERELEASE #4 r254801M/254827:902501: Sun Aug 25 05:15:29 > PDT > > 2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF amd64 > > FreeBSD 9.2-PRERELEASE #5 r255066M/255091:902503: Sat Aug 31 11:58:53 > PDT > > 2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF amd64 > > FreeBSD 9.2-PRERELEASE #5 r255104M/255115:902503: Sun Sep 1 05:02:12 > PDT > > 2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF amd64 > > > > Now, I like to have a "private playground" for doing things with > > machines, so I make use of both em(4) NICs on the machine: em0 connects > > to the rest of the work network; em1 is connected to a switch I brought > > in from home, and to which I connect "other things" (such as my laptop). > > And because I'm fairly comfortable with them, I use IPFW & natd. For > > some folks here, none of that should come as a surprise. :-}) > > > > For reference, the em(4) devices in question are: > > > > em0@pci0:0:25:0: class=0x020000 card=0x060d15d9 chip=0x10ef8086 > > rev=0x06 hdr=0x00 > > vendor = 'Intel Corporation' > > device = '82578DM Gigabit Network Connection' > > > > and > > > > em1@pci0:3:0:0: class=0x020000 card=0x060d15d9 chip=0x10d38086 rev=0x00 > > hdr=0x00 > > vendor = 'Intel Corporation' > > device = '82574L Gigabit Network Connection' > > > > > > > > I noticed that when I tried to write files to NFS, I could write small > > files OK, but larger ones seemed to ... hang. > > > > Note: We don't use jumbo frames. (Work IT is convinced that they > > don't help. I'm trying to better-understand their reasoning.) > > > > Further poking around showed that (under the above conditions): > > * natd CPU% was climbing as more of the file was copied, up to 2^21 > > bytes. (At that point, nothing further was saved on NFS.) > > * dhcpd CPU% was also climbing. I tried killing that, but doing so > > didn't affect the other results. (Killing natd made connectivity > > cease, given the IPFW rules in effect.) > > * Performing a tcpdump while trying to copy a file of length 117709618 > > showed lots of TCP retransmissions. In fact, I'd hazard that every TCP > > packet was getting retransmitted. > > * "ifconfig -v em0" showed flags TSO4 & VLAN_HWTSO turned on. > > * "sysctl net.inet.tcp.tso" showed "1" -- enabled. > > > > As soon as I issued "sudo net.inet.tcp.tso=0" ... the copy worked without > > a hitch or a whine. And I was able to copy all 117709618 bytes, not just > > 2097152 (2^21). > > > > Is the above expected? It came rather as a surprise to me. > > > > Peace, > > david > > -- > > David H. Wolfskill david@catwhisker.org > > Taliban: Evil cowards with guns afraid of truth from a 14-year old girl. > > > > See http://www.catwhisker.org/~david/publickey.gpg for my public key. > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >