From owner-freebsd-net@FreeBSD.ORG Fri Apr 29 15:12:44 2011 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63960106564A for ; Fri, 29 Apr 2011 15:12:44 +0000 (UTC) (envelope-from kungfujesus06@gmail.com) Received: from mail-iw0-f182.google.com (mail-iw0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id 1A0328FC15 for ; Fri, 29 Apr 2011 15:12:43 +0000 (UTC) Received: by iwn33 with SMTP id 33so4366937iwn.13 for ; Fri, 29 Apr 2011 08:12:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:date:from:to:cc:subject:message-id:references :mime-version:content-type:content-disposition:in-reply-to :user-agent; bh=vo4Qy+06+cACUNH/erPuJNUm/7xgimvkgPE7EP7vFUI=; b=X+kQ0ti+Lhj6hZqH0rECLI0x5xA/fyZRglLXen9SDkuyTah+0/DcqGRsbmPMfCauRA a4T/g9iqhv6qMSCwOGxyReV1OCvdMvU60+ws5QMkKmAaHg22yPRzFvm4FV0c60Ji7geV 2TvQybXu76c+CExDE6F9ld7GPET7F0Rs9pOmY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=vuPhAG1h9OCSYpoU0QPMI7dim2loW5fghr51r9mnlX2E64TDWKuXy/MHJuc0DOwjsL elAMTB+E+AigfISMmlayRiehM7FHcYLSG/8ncRad4Bd3zByHx26uGqh28yHriX6TAd4/ ZHGCyu+c1z8uyOMWSNI2r5aBSo/RzPA7lHkoU= Received: by 10.42.228.68 with SMTP id jd4mr6022487icb.142.1304089963209; Fri, 29 Apr 2011 08:12:43 -0700 (PDT) Received: from ossumpossum.geop.uc.edu ([129.137.163.68]) by mx.google.com with ESMTPS id e12sm666371ics.19.2011.04.29.08.12.40 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 29 Apr 2011 08:12:41 -0700 (PDT) Date: Fri, 29 Apr 2011 11:12:59 -0400 From: Adam Stylinski To: Jack Vogel Message-ID: <20110429151259.GA96064@ossumpossum.geop.uc.edu> References: <41EE7AB832F24194AC8730544E1C2EB7@multiplay.co.uk> <20110428152141.GA19362@ossumpossum.geop.uc.edu> <11659E32824B4B1E91B6B219BDEF1234@multiplay.co.uk> <20110428160919.GE19362@ossumpossum.geop.uc.edu> <20110428180037.GA1889@zephyr.snd-wireless.uc.edu> <20110428224749.GA47010@freebsdbox.adamsnet> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="45Z9DzgjV8m4Oswq" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@freebsd.org Subject: Re: em0 performance subpar X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 29 Apr 2011 15:12:44 -0000 --45Z9DzgjV8m4Oswq Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Apr 28, 2011 at 09:41:01PM -0700, Jack Vogel wrote: > We rarely test 32bit any more, the only time we would is because of a > problem, > so 99% is amd64, so that's not the problem. >=20 > Running netperf without any special arguments, using TCP_STREAM and > TCP_MAERTS tests what numbers are you seeing? >=20 > Jack >=20 >=20 > On Thu, Apr 28, 2011 at 5:34 PM, Adam Stylinski = wrote: >=20 > > On Thu, Apr 28, 2011 at 6:47 PM, Adam Stylinski wrote: > > > >> On Thu, Apr 28, 2011 at 02:22:29PM -0700, Jack Vogel wrote: > >> > My validation engineer set things up on an 8.2 REL system, testing t= he > >> > equivalent of > >> > HEAD, and he reports performance is fine. This is without any tweaks > >> from > >> > what's > >> > checked in. > >> > > >> > Increasing the descriptors to 4K is way overkill and might actually > >> cause > >> > problems, > >> > go back to default. > >> > > >> > He has a Linux test client, what are you transmitting to? > >> > > >> > Jack > >> > > >> > > >> > On Thu, Apr 28, 2011 at 11:00 AM, Adam Stylinski < > >> kungfujesus06@gmail.com>wrote: > >> > > >> > > On Thu, Apr 28, 2011 at 09:52:14AM -0700, Jack Vogel wrote: > >> > > > Adam, > >> > > > > >> > > > The TX ring for the legacy driver is small right now compared to= em, > >> try > >> > > > this experiment, > >> > > > edit if_lem.c, search for "lem_txd" and change EM_DEFAULT_TXD to > >> 1024, > >> > > see > >> > > > what > >> > > > that does, then 2048. > >> > > > > >> > > > My real strategy with the legacy code was that it should stable, > >> meaning > >> > > not > >> > > > getting > >> > > > a lot of changes... that really hasn't worked out over time. I > >> suppose > >> > > I'll > >> > > > have to try and > >> > > > give it some tweaks and let you try it. The problem with this co= de > >> is it > >> > > > technically supports > >> > > > a huge range of old stuff we don't test any more, things I do mi= ght > >> cause > >> > > > other regressions :( > >> > > > > >> > > > Oh well, let me know if increasing the TX descriptors helps. > >> > > > > >> > > > Jack > >> > > Jack, > >> > > > >> > > Is this the same thing as adjusting these values?: > >> > > > >> > > hw.em.rxd=3D4096 > >> > > hw.em.txd=3D4096 > >> > > > >> > > If so I've maxed this out and it's not helping. I'll give it a sh= ot > >> on my > >> > > 8-STABLE box as it has a kernel I can play with. > >> > > > >> > > Setting the MTU to 1500 gave lower throughput. > >> > > > >> > > -- > >> > > Adam Stylinski > >> > > PGP Key: http://pohl.ececs.uc.edu/~adam/publickey.pub > >> > > Blog: http://technicallyliving.blogspot.com > >> > > > >> > >> I am transmitting to a linux client (kernel 2.6.38, 9000 byte MTU, PCI= -Ex > >> based card). My sysctl's on the Linux client (apart from the default)= look > >> like so: > >> > >> net.ipv4.ip_forward =3D 0 > >> # Enables source route verification > >> net.ipv4.conf.default.rp_filter =3D 1 > >> # Enable reverse path > >> net.ipv4.conf.all.rp_filter =3D 1 > >> net.core.rmem_max =3D 16777216 > >> net.core.wmem_max =3D 16777216 > >> net.ipv4.tcp_rmem =3D 4096 87380 16777216 > >> net.ipv4.tcp_wmem =3D 4096 87380 16777216 > >> net.core.wmem_default =3D 87380 > >> net.core.rmem_default =3D 87380 > >> net.ipv4.tcp_mem =3D 98304 131072 196608 > >> net.ipv4.tcp_no_metrics_save =3D 1 > >> net.ipv4.tcp_window_scaling =3D 1 > >> dev.rtc.max-user-freq =3D 1024 > >> > >> The exact troublesome device (as reported by pciconf): > >> > >> em0@pci0:7:5:0: class=3D0x020000 card=3D0x13768086 chip=3D0x107c8086 r= ev=3D0x05 > >> hdr=3D0x00 > >> vendor =3D 'Intel Corporation' > >> device =3D 'Gigabit Ethernet Controller (Copper) rev 5 (82541P= I)' > >> class =3D network > >> subclass =3D ethernet > >> Apart from bus saturation (which I don't suspect is the problem) I'm n= ot > >> sure what the issue could be. What should I try next? > >> > >> -- > >> Adam Stylinski > >> PGP Key: http://pohl.ececs.uc.edu/~adam/publickey.pub > >> Blog: http://technicallyliving.blogspot.com > >> > > > > One detail that I didn't mention which may or may not matter is that the > > issues I'm having this with are on amd64 distributions. I have the same > > card in an x86 system and iperf with nearly default settings managed to= do > > 850ish Mbits. Did your lab tests consist of amd64 machines? > > TCP_STREAM: TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.121 (192= =2E168.0.121) port 0 AF_INET Recv Send Send =20 Socket Socket Message Elapsed =20 Size Size Size Time Throughput =20 bytes bytes bytes secs. 10^6bits/sec =20 87380 87380 87380 10.00 578.77 =20 TCP_MAERTS: TCP MAERTS TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.0.121 (192= =2E168.0.121) port 0 AF_INET Recv Send Send =20 Socket Socket Message Elapsed =20 Size Size Size Time Throughput =20 bytes bytes bytes secs. 10^6bits/sec =20 87380 87380 87380 10.00 836.91 interesting, so TCP_STREAM specifically has the issue. I suppose I could r= ead documentation on what the tests do but what does this tell? --=20 Adam Stylinski PGP Key: http://pohl.ececs.uc.edu/~adam/publickey.pub Blog: http://technicallyliving.blogspot.com --45Z9DzgjV8m4Oswq Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) iQIcBAEBAgAGBQJNutV7AAoJED6sRHE6Tvmne0EP+wblTPZyO6MOL7DYCdLbZ682 M2rnSztohrJkiGw4V7R5GB/fHLTbUgtO7/KwHVuc8ZeV/+RXd1iVGdxEK0MeV2a6 xEtosGqktyVTGAxhuEKRE69BckZ4uAMWBRycsZbT4DUrvv29b91ssAvr7uJGotbT kQS/Ask2qyvSLGyrhn8JcTx+vkkcpnF87xQajzMkzFSQ9jyhdmwFoYjMOuNSTWTe R5JOywe4+dAJ/c1ZTH5YlzllR3z4RTcxd4KGoP2yqowuLx1xL2mNgOCntBaLh3wQ NRqNOTw4jORC3p2tPOjywvy4H26F5gngpVoxs8Yh/vfaOsH2hm99FD1iHR3xTwoE AyEz3EkD6+ZEK9OvvOhPjbpdv5yKJl2iY9Hhf6YjpauEbnjLl/HOhTvTi1WhghJJ iFXceiMBxZ5QRgfNt/4TbGjC6i/bx4P/3/5XhDayl4lhNmBGNFHxTvx9ilEldoK8 5OmIy+gyqSTw2ap0gW9+aEDhLUruZIe/S5/RTEZ+9VlWhUgyC9Ihwu8WAMMpOPZW 9p0COJIi5ilS1VTisI+/pBHPFkwNDlzYWT3lnCYiQQFbq3pkqFZpOo4rsgP/ZWTw /NgIxqcMfNPz6STwIP0SS4YbijcP/OrMC3RnB/oyT94sCbpM25PtNcWwquN+Lztr WW+jAuofYN4GG+WqeB/0 =92UI -----END PGP SIGNATURE----- --45Z9DzgjV8m4Oswq--