Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 29 Apr 2011 11:12:59 -0400
From:      Adam Stylinski <kungfujesus06@gmail.com>
To:        Jack Vogel <jfvogel@gmail.com>
Cc:        freebsd-net@freebsd.org
Subject:   Re: em0 performance subpar
Message-ID:  <20110429151259.GA96064@ossumpossum.geop.uc.edu>
In-Reply-To: <BANLkTikt%2B4=DZ78Ky07psV%2Bin3FyP4Pavw@mail.gmail.com>
References:  <41EE7AB832F24194AC8730544E1C2EB7@multiplay.co.uk> <20110428152141.GA19362@ossumpossum.geop.uc.edu> <11659E32824B4B1E91B6B219BDEF1234@multiplay.co.uk> <20110428160919.GE19362@ossumpossum.geop.uc.edu> <BANLkTimRcAdO33AasTOMb%2BLSOjc1GwFe%2Bg@mail.gmail.com> <20110428180037.GA1889@zephyr.snd-wireless.uc.edu> <BANLkTikLb7k36mN3ktRGdm4mQyehHB-ONg@mail.gmail.com> <20110428224749.GA47010@freebsdbox.adamsnet> <BANLkTin6q2nC_Wwndj=bTbpuZxpBetbEZA@mail.gmail.com> <BANLkTikt%2B4=DZ78Ky07psV%2Bin3FyP4Pavw@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help

--45Z9DzgjV8m4Oswq
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Thu, Apr 28, 2011 at 09:41:01PM -0700, Jack Vogel wrote:
> We rarely test 32bit any more, the only time we would is because of a
> problem,
> so 99% is amd64, so that's not the problem.
>=20
> Running netperf without any special arguments, using TCP_STREAM and
> TCP_MAERTS tests what numbers are you seeing?
>=20
> Jack
>=20
>=20
> On Thu, Apr 28, 2011 at 5:34 PM, Adam Stylinski <kungfujesus06@gmail.com>=
wrote:
>=20
> > On Thu, Apr 28, 2011 at 6:47 PM, Adam Stylinski <kungfujesus06@gmail.co=
m>wrote:
> >
> >> On Thu, Apr 28, 2011 at 02:22:29PM -0700, Jack Vogel wrote:
> >> > My validation engineer set things up on an 8.2 REL system, testing t=
he
> >> > equivalent of
> >> > HEAD, and he reports performance is fine. This is without any tweaks
> >> from
> >> > what's
> >> > checked in.
> >> >
> >> > Increasing the descriptors to 4K is way overkill and might actually
> >> cause
> >> > problems,
> >> > go back to default.
> >> >
> >> > He has a Linux test client, what are you transmitting to?
> >> >
> >> > Jack
> >> >
> >> >
> >> > On Thu, Apr 28, 2011 at 11:00 AM, Adam Stylinski <
> >> kungfujesus06@gmail.com>wrote:
> >> >
> >> > > On Thu, Apr 28, 2011 at 09:52:14AM -0700, Jack Vogel wrote:
> >> > > > Adam,
> >> > > >
> >> > > > The TX ring for the legacy driver is small right now compared to=
 em,
> >> try
> >> > > > this experiment,
> >> > > > edit if_lem.c, search for "lem_txd" and change EM_DEFAULT_TXD to
> >> 1024,
> >> > > see
> >> > > > what
> >> > > > that does, then 2048.
> >> > > >
> >> > > > My real strategy with the legacy code was that it should stable,
> >> meaning
> >> > > not
> >> > > > getting
> >> > > > a lot of changes... that really hasn't worked out over time. I
> >> suppose
> >> > > I'll
> >> > > > have to try and
> >> > > > give it some tweaks and let you try it. The problem with this co=
de
> >> is it
> >> > > > technically supports
> >> > > > a huge range of old stuff we don't test any more, things I do mi=
ght
> >> cause
> >> > > > other regressions :(
> >> > > >
> >> > > > Oh well, let me know if increasing the TX descriptors helps.
> >> > > >
> >> > > > Jack
> >> > > Jack,
> >> > >
> >> > > Is this the same thing as adjusting these values?:
> >> > >
> >> > > hw.em.rxd=3D4096
> >> > > hw.em.txd=3D4096
> >> > >
> >> > > If so I've maxed this out and it's not helping.  I'll give it a sh=
ot
> >> on my
> >> > > 8-STABLE box as it has a kernel I can play with.
> >> > >
> >> > > Setting the MTU to 1500 gave lower throughput.
> >> > >
> >> > > --
> >> > > Adam Stylinski
> >> > > PGP Key: http://pohl.ececs.uc.edu/~adam/publickey.pub
> >> > > Blog: http://technicallyliving.blogspot.com
> >> > >
> >>
> >> I am transmitting to a linux client (kernel 2.6.38, 9000 byte MTU, PCI=
-Ex
> >> based card).  My sysctl's on the Linux client (apart from the default)=
 look
> >> like so:
> >>
> >> net.ipv4.ip_forward =3D 0
> >> # Enables source route verification
> >> net.ipv4.conf.default.rp_filter =3D 1
> >> # Enable reverse path
> >> net.ipv4.conf.all.rp_filter =3D 1
> >> net.core.rmem_max =3D 16777216
> >> net.core.wmem_max =3D 16777216
> >> net.ipv4.tcp_rmem =3D 4096 87380 16777216
> >> net.ipv4.tcp_wmem =3D 4096 87380 16777216
> >> net.core.wmem_default =3D 87380
> >> net.core.rmem_default =3D 87380
> >> net.ipv4.tcp_mem =3D 98304 131072 196608
> >> net.ipv4.tcp_no_metrics_save =3D 1
> >> net.ipv4.tcp_window_scaling =3D 1
> >> dev.rtc.max-user-freq =3D 1024
> >>
> >> The exact troublesome device (as reported by pciconf):
> >>
> >> em0@pci0:7:5:0: class=3D0x020000 card=3D0x13768086 chip=3D0x107c8086 r=
ev=3D0x05
> >> hdr=3D0x00
> >>    vendor     =3D 'Intel Corporation'
> >>     device     =3D 'Gigabit Ethernet Controller (Copper) rev 5 (82541P=
I)'
> >>    class      =3D network
> >>    subclass   =3D ethernet
> >> Apart from bus saturation (which I don't suspect is the problem) I'm n=
ot
> >> sure what the issue could be.  What should I try next?
> >>
> >> --
> >> Adam Stylinski
> >> PGP Key: http://pohl.ececs.uc.edu/~adam/publickey.pub
> >> Blog: http://technicallyliving.blogspot.com
> >>
> >
> > One detail that I didn't mention which may or may not matter is that the
> > issues I'm having this with are on amd64 distributions.  I have the same
> > card in an x86 system and iperf with nearly default settings managed to=
 do
> > 850ish Mbits.  Did your lab tests consist of amd64 machines?
> >

TCP_STREAM:
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.121 (192=
=2E168.0.121) port 0 AF_INET
Recv   Send    Send                         =20
Socket Socket  Message  Elapsed             =20
Size   Size    Size     Time     Throughput =20
bytes  bytes   bytes    secs.    10^6bits/sec =20

 87380  87380  87380    10.00     578.77 =20

TCP_MAERTS:

TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.121 (192=
=2E168.0.121) port 0 AF_INET
Recv   Send    Send                         =20
Socket Socket  Message  Elapsed             =20
Size   Size    Size     Time     Throughput =20
bytes  bytes   bytes    secs.    10^6bits/sec =20

 87380  87380  87380    10.00     836.91

interesting, so TCP_STREAM specifically has the issue.  I suppose I could r=
ead documentation on what the tests do but what does this tell?

--=20
Adam Stylinski
PGP Key: http://pohl.ececs.uc.edu/~adam/publickey.pub
Blog: http://technicallyliving.blogspot.com

--45Z9DzgjV8m4Oswq
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (FreeBSD)

iQIcBAEBAgAGBQJNutV7AAoJED6sRHE6Tvmne0EP+wblTPZyO6MOL7DYCdLbZ682
M2rnSztohrJkiGw4V7R5GB/fHLTbUgtO7/KwHVuc8ZeV/+RXd1iVGdxEK0MeV2a6
xEtosGqktyVTGAxhuEKRE69BckZ4uAMWBRycsZbT4DUrvv29b91ssAvr7uJGotbT
kQS/Ask2qyvSLGyrhn8JcTx+vkkcpnF87xQajzMkzFSQ9jyhdmwFoYjMOuNSTWTe
R5JOywe4+dAJ/c1ZTH5YlzllR3z4RTcxd4KGoP2yqowuLx1xL2mNgOCntBaLh3wQ
NRqNOTw4jORC3p2tPOjywvy4H26F5gngpVoxs8Yh/vfaOsH2hm99FD1iHR3xTwoE
AyEz3EkD6+ZEK9OvvOhPjbpdv5yKJl2iY9Hhf6YjpauEbnjLl/HOhTvTi1WhghJJ
iFXceiMBxZ5QRgfNt/4TbGjC6i/bx4P/3/5XhDayl4lhNmBGNFHxTvx9ilEldoK8
5OmIy+gyqSTw2ap0gW9+aEDhLUruZIe/S5/RTEZ+9VlWhUgyC9Ihwu8WAMMpOPZW
9p0COJIi5ilS1VTisI+/pBHPFkwNDlzYWT3lnCYiQQFbq3pkqFZpOo4rsgP/ZWTw
/NgIxqcMfNPz6STwIP0SS4YbijcP/OrMC3RnB/oyT94sCbpM25PtNcWwquN+Lztr
WW+jAuofYN4GG+WqeB/0
=92UI
-----END PGP SIGNATURE-----

--45Z9DzgjV8m4Oswq--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110429151259.GA96064>