Date: Tue, 3 Sep 2013 12:27:34 -0700 From: David Wolfskill <david@catwhisker.org> To: FreeBSD Net <net@FreeBSD.org> Subject: Re: TSO and FreeBSD vs Linux Message-ID: <20130903192734.GA19406@albert.catwhisker.org> In-Reply-To: <5214F506.3070706@freebsd.org> References: <520A6D07.5080106@freebsd.org> <5214F506.3070706@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--3V7upXqbjpZ4EhLz
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Wed, Aug 21, 2013 at 07:12:38PM +0200, Andre Oppermann wrote:
> On 13.08.2013 19:29, Julian Elischer wrote:
> > I have been tracking down a performance embarrassment on AMAZON EC2 and=
have found it I think.
> > Our OS cousins over at Linux land have implemented some interesting beh=
aviour when TSO is in use.
>=20
> There used to be a different problem with EC2 and FreeBSD TSO. The Xen h=
ypervisor
> doesn't like large 64K TSO bursts we generate, the drivers drops the whol=
e TSO chain,
> TCP gets upset and turns off TSO alltogether leaving the connection going=
at one
> packet a time as in the old days.
> ...
My apologies for jumping in so late -- I'm not subscribed to -net@.
At work, I received a new desktop machine a few months ago; here's a
recent history of what it has been running:
FreeBSD 9.2-PRERELEASE #4 r254801M/254827:902501: Sun Aug 25 05:15:29 PDT =
2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF amd64
FreeBSD 9.2-PRERELEASE #5 r255066M/255091:902503: Sat Aug 31 11:58:53 PDT =
2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF amd64
FreeBSD 9.2-PRERELEASE #5 r255104M/255115:902503: Sun Sep 1 05:02:12 PDT =
2013 root@dwolf-fbsd:/usr/obj/usr/src/sys/DWOLF amd64
Now, I like to have a "private playground" for doing things with
machines, so I make use of both em(4) NICs on the machine: em0 connects
to the rest of the work network; em1 is connected to a switch I brought
in from home, and to which I connect "other things" (such as my laptop).
And because I'm fairly comfortable with them, I use IPFW & natd. For
some folks here, none of that should come as a surprise. :-})
For reference, the em(4) devices in question are:
em0@pci0:0:25:0: class=3D0x020000 card=3D0x060d15d9 chip=3D0x10ef808=
6 rev=3D0x06 hdr=3D0x00
vendor =3D 'Intel Corporation'
device =3D '82578DM Gigabit Network Connection'
and
em1@pci0:3:0:0: class=3D0x020000 card=3D0x060d15d9 chip=3D0x10d38086 rev=3D=
0x00 hdr=3D0x00
vendor =3D 'Intel Corporation'
device =3D '82574L Gigabit Network Connection'
I noticed that when I tried to write files to NFS, I could write small
files OK, but larger ones seemed to ... hang.
Note: We don't use jumbo frames. (Work IT is convinced that they
don't help. I'm trying to better-understand their reasoning.)
Further poking around showed that (under the above conditions):
* natd CPU% was climbing as more of the file was copied, up to 2^21
bytes. (At that point, nothing further was saved on NFS.)
* dhcpd CPU% was also climbing. I tried killing that, but doing so
didn't affect the other results. (Killing natd made connectivity
cease, given the IPFW rules in effect.)
* Performing a tcpdump while trying to copy a file of length 117709618
showed lots of TCP retransmissions. In fact, I'd hazard that every TCP
packet was getting retransmitted.
* "ifconfig -v em0" showed flags TSO4 & VLAN_HWTSO turned on.
* "sysctl net.inet.tcp.tso" showed "1" -- enabled.
As soon as I issued "sudo net.inet.tcp.tso=3D0" ... the copy worked without
a hitch or a whine. And I was able to copy all 117709618 bytes, not just
2097152 (2^21).
Is the above expected? It came rather as a surprise to me.
Peace,
david
--=20
David H. Wolfskill david@catwhisker.org
Taliban: Evil cowards with guns afraid of truth from a 14-year old girl.
See http://www.catwhisker.org/~david/publickey.gpg for my public key.
--3V7upXqbjpZ4EhLz
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.21 (FreeBSD)
iEYEARECAAYFAlImOCUACgkQmprOCmdXAD1PlwCfRI35ikbwABwsF9L8S91GH+LM
0K4AnRYK8kglQA2RzY5R39gSnP/NmYKI
=/Nw8
-----END PGP SIGNATURE-----
--3V7upXqbjpZ4EhLz--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130903192734.GA19406>