Date: Thu, 28 Apr 2011 11:51:24 -0400 From: Adam Stylinski <kungfujesus06@gmail.com> To: Pierre Lamy <pierre@userid.org> Cc: freebsd-net@freebsd.org Subject: Re: em0 performance subpar Message-ID: <20110428155124.GD19362@ossumpossum.geop.uc.edu> In-Reply-To: <4DB994E7.9060805@userid.org> References: <20110428072946.GA11391@zephyr.adamsnet> <4DB961EA.4080407@userid.org> <20110428132513.GB2800@ossumpossum.geop.uc.edu> <4DB994E7.9060805@userid.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--FFoLq8A0u+X9iRU8 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Apr 28, 2011 at 11:25:11AM -0500, Pierre Lamy wrote: > Someone mentioned on freebsd-current: >=20 > With the 7.2.2 driver you also will use different mbuf pools depending on > > the MTU you are using. If you use jumbo frames it will use 4K clusters, > > if you go to 9K jumbos it will use 9K mbuf clusters. *The number of the= se > > allocated by default is small (like 6400 small :) .* > > > > I would use 'netstat -m' to see what the pools look like. >=20 > Hope this helps. Check the perf with 1500 byte frames. >=20 > Adam Stylinski wrote: > > On Thu, Apr 28, 2011 at 08:47:38AM -0400, Pierre Lamy wrote: > > =20 > >> Try using netblast on FreeBSD instead of iperf, there have been a lot = of=20 > >> discussions about this on this list. > >> > >> Is it possible you're maxing out the system's PCI-xxx bus? Did you tun= e=20 > >> up the system buffers? Data doesn't just get queued up on the NIC=20 > >> driver, it also queues within the system's kernel buffers. Try these, = I=20 > >> have no idea if it will help: > >> > >> sysctl -w net.inet.tcp.sendspace=3D373760 > >> sysctl -w net.inet.tcp.recvspace=3D373760 > >> sysctl -w net.local.stream.sendspace=3D82320 > >> sysctl -w net.local.stream.recvspace=3D82320 > >> sysctl -w net.local.stream.recvspace=3D373760 > >> sysctl -w net.local.stream.sendspace=3D373760 > >> sysctl -w net.raw.recvspace=3D373760 > >> sysctl -w net.raw.sendspace=3D373760 > >> sysctl -w net.inet.tcp.local_slowstart_flightsize=3D10 > >> sysctl -a net.inet.tcp.delayed_ack=3D0 > >> sysctl -w kern.maxvnodes=3D600000 > >> sysctl -w net.local.dgram.recvspace=3D8192 > >> sysctl -w net.local.dgram.maxdgram=3D8192 > >> sysctl -w net.inet.tcp.slowstart_flightsize=3D10 > >> sysctl -w net.inet.tcp.path_mtu_discovery=3D0 > >> > >> They're all tunable while system is running. > >> > >> -Pierre > >> > >> On 4/28/2011 3:29 AM, Adam Stylinski wrote: > >> =20 > >>> Hello, > >>> > >>> I have an intel gigabit network adapter (the 1000 GT w/chipset 82541P= I) which performs poorly in Freebsd compared to the same card in Linux. I'= ve tried this card in two different freebsd boxes and for whatever reason I= get poor transmit performance. I've done all of the tweaking specified in= just about every guide out there (the usual TCP window scaling, larger nmb= clusters, delayed acks, etc) and still I get only around 600mbps. I'm usin= g jumbo frames, with an MTU of 9000. I'm testing this with iperf. While I= realize that this may not be the most realistic test, linux hosts with the= same card can achieve 995Mbit/s to another host running this. When the Fr= eebsd box is the server, Linux hosts can transmit to it at around 800 somet= hing Mbit/s. I've increased the transmit descriptors as specified in the i= f_em man page, and while that gave me 20 or 30 more mbit/s, my transmit per= formance is still below normal. > >>> > >>> sysctl stats report that the card is trigger a lot of tx_desc_fail2: > >>> dev.em.0.tx_desc_fail2: 3431 > >>> > >>> Looking at a comment in the source code this indicates that the card = was not able to obtain enough transmit descriptors (but I've given the card= the maximum of 4096 in my loader.conf tunable). Is this a bug or a perfor= mance regression of some kind? Does anybody have a fix for this? I tried = another card with the same chip in a different box on 8-STABLE to no avail = (the box I'm trying to improve performance on is on 8.2-RELEASE-p1). > >>> > >>> Anybody manage to make this card push above 600mbps in ideal network = benchmarks? Any help would be gladly appreciated. > >>> =20 > >> > >> =20 > > > > I doubt I'm saturating the PCI bus, the only other thing on the bus is = a really really crappy PCI video card. The same card on lesser powerful ma= chines with Linux (Pentium 4) are able to achieve much more throughput, so = it's not likely a bus limitation. > > > > I adjusted the listed sysctl live tunables which I hadn't already compe= nsated for and it didn't seem to have an effect. > > > > =20 >=20 >=20 >=20 I tweaked that, doesn't seem to help. I'd rather not go down to a 1500 byt= e MTU just yet. I may try this later, but most of my network is configured= for jumbo frames. [adam@nasbox ~]$ sysctl kern.ipc.nmbjumbo9 kern.ipc.nmbjumbo9: 12800 --=20 Adam Stylinski PGP Key: http://pohl.ececs.uc.edu/~adam/publickey.pub Blog: http://technicallyliving.blogspot.com --FFoLq8A0u+X9iRU8 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (FreeBSD) iQIcBAEBAgAGBQJNuYz7AAoJED6sRHE6TvmnoZUP/02iM1WELL0XtWFBu++CHH42 NCjLH3LUVsQW/ief/XoqXt3zvfaPIemRZEi50jAKPbXDcq9t+PCG8vbvE2d+Q2i+ h1zmwbdIgT/x0lqm9U9aaTNpcIXjFUqVS1Qs36HcA31DOMcP4bSakJt+iSZQp9tl TfwLpuH8NojyLDMvZkkEpzX8Rgvc3NmmjSysCLfhnSortjQhiyxCBP0u5IyDHC38 KTB3PPkTeFm3jvXzCRtwRWqeLG/Wzp8I9PdpmoU1cXuVlMqUukEqWjd0NL85dbHL T3QFQChe4KsZ7s/WLF654WZR9QDXDgDmi06i3mD5kWfJf/hhUn05aAbFy8etcYwf gizKccgip4TegIzuIVbkk5Kb/KCI0WsSFQkGBQeQv+YMVXww6N0nnK9GDgxIUtJu cbWNwrOt+RWMCBahZbFvAqB5ffDnD0MLjuRTlWYVlCkjsi6gL+a1sdyYtHqfB+J0 wF1eGmImVHrQqmQo8QTBXn6smaSktGrnByhkwS1TeYigT5RXYtk12tFUCZ5Z6hCV gDSUWsZG38DlXldocRvHv4OEpIA5eGXGr7X7V9n+52qRO6YG6QLumpzOPxpofmhy mImjUaGyzdRk7eHXM8V9DaG5J3DZQkEpMBjRbjftyw0AdlsXByodyDF6tCnrETzI +3QUGoIXfP5Pw0qUq016 =+gSG -----END PGP SIGNATURE----- --FFoLq8A0u+X9iRU8--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110428155124.GD19362>