Date: Thu, 22 Apr 2010 09:39:25 -0700 From: Jack Vogel <jfvogel@gmail.com> To: Stephen Sanders <ssanders@softhammer.net> Cc: Brandon Gooch <jamesbrandongooch@gmail.com>, freebsd-performance@freebsd.org Subject: Re: FreeBSD 8.0 ixgbe Poor Performance Message-ID: <h2x2a41acea1004220939k23e34403hcd57480d92f4a0f1@mail.gmail.com> In-Reply-To: <4BD06E28.3060609@softhammer.net> References: <4BCF0C9A.10005@softhammer.net> <y2y179b97fb1004210804s6ca12944qf194f3a6d8c33cfe@mail.gmail.com> <x2j2a41acea1004211113kf8e4de95s9ff5c1669156b82c@mail.gmail.com> <4BCF5783.9050007@softhammer.net> <v2t2a41acea1004211353nbfc4e68cy6dfaae6f47f86034@mail.gmail.com> <4BD06E28.3060609@softhammer.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Couple more things that come to mind: make sure you increase mbuf pool, nmbclusters up to at least 262144, and the driver uses 4K clusters if you go to jumbo frames (nmbjumbop). some workloads will benefit from increeasing the various sendspace and recvspace parameters, maxsockets and maxfiles are other candidates. Another item: look in /var/log/messages to see if you are getting any Interrupt storm messages, if you are that can throttle the irq and reduce performance, there is an intr_storm_threshold that you can increase to keep that from happening. Finally, it is sometimes not possible to fully utilize the hardware from a single process, you can get limited by the socket layer, stack, scheduler, whatever, so you might want to use multiple test processes. I believe iperf has a builtin way to do this. Run more threads and look at your cumulative. Good luck, Jack On Thu, Apr 22, 2010 at 8:41 AM, Stephen Sanders <ssanders@softhammer.net>wrote: > I believe that "pciconf -lvc" showed that the cards were in the correct > slot. I'm not sure as to what all of the output means but I'm guessing that > " cap 10[a0] = PCI-Express 2 endpoint max data 128(256) link x8(x8)" means > that the card is an 8 lane card and is using all 8 lanes. > > Setting kern.ipc.maxsockbuf to16777216 got a better result with ipref TCP > testing. The rate when from ~2.5Gpbs to ~5.5Gbps. > > Running iperf in UDP test mode is still yielding ~2.5Gbps. Running > tcpreplay tests is also yielding ~2.5Gbps as well. > > Command lines for iperf testing are: > > ipref -t 10 -w 2.5m -l 2.5m -c 169.1.0.2 > iperf -s -w 2.5m -B 169.1.0.2 > > iperf -t 10 -w 2.5m -c 169.1.0.2 -u > iperf -s -w 2.5m -B 169.1.0.2 -u > > For the tcpdump test, I'm sending output to /dev/null and using the cache > flag on tcpreplay in order to avoid limiting my network interface throughput > to the disk speed. > Commands lines for this test are: > > tcpdump -i ix1 -w /dev/null > tcpreplay -i ix1 -t -l 0 -K ./rate.pcap > > Please forgive my lack of kernel building prowess but I'm guessing that the > latest driver needs to be built in a FreeBSD STABLE tree. I ran into an > undefined symbol "drbr_needs_enqueue" in the ixgbe code I downloaded. > > Thanks for all the help. > > On 4/21/2010 4:53 PM, Jack Vogel wrote: > > Use my new driver and it will tell you when it comes up with the slot speed > is, > and if its substandard it will SQUAWK loudly at you :) > > I think the S5000PAL only has Gen1 PCIE slots which is going to limit you > somewhat. Would recommend a current generation (x58 or 5520 chipset) > system if you want the full benefit of 10G. > > BTW, you dont way what adapter, 82598 or 82599, you are using? > > Jack > > > On Wed, Apr 21, 2010 at 12:52 PM, Stephen Sanders <ssanders@softhammer.net > > wrote: > >> I'd be most pleased to get near 9k. >> >> I'm running FreeBSD 8.0 amd64 on both of the the test hosts. I've reset >> the configurations to system default as I was getting no where with >> sysctl and loader.conf settings. >> >> The motherboards have been configured to do MSI interrupts. The >> S5000PAL has a MSI to old style interrupt BIOS setting that confuses the >> driver interrupt setup. >> >> The 10Gbps cards should be plugged into the 8x PCI-E slots on both >> hosts. I'm double checking that claim right now and will get back later. >> >> Thanks >> >> >> On 4/21/2010 2:13 PM, Jack Vogel wrote: >> > When you get into the 10G world your performance will only be as good >> > as your weakest link, what I mean is if you connect to something that >> has >> > less than stellar bus and/or memory performance it is going to throttle >> > everything. >> > >> > Running back to back with two good systems you should be able to get >> > near line rate (9K range). Things that can effect that: 64 bit kernel, >> > TSO, LRO, how many queues come to mind. The default driver config >> > should get you there, so tell me more about your hardware/os config?? >> > >> > Jack >> > >> > >> > >> > On Wed, Apr 21, 2010 at 8:04 AM, Brandon Gooch >> > <jamesbrandongooch@gmail.com>wrote: >> > >> > >> >> On Wed, Apr 21, 2010 at 9:32 AM, Stephen Sanders >> >> <ssanders@softhammer.net> wrote: >> >> >> >>> I am running speed tests on a pair of systems equipped with Intel >> 10Gbps >> >>> cards and am getting poor performance. >> >>> >> >>> iperf and tcpdump testing indicates that the card is running at >> roughly >> >>> 2.5Gbps max transmit/receive. >> >>> >> >>> My attempts at turning fiddling with netisr, polling, and varying the >> >>> buffer sizes has been fruitless. I'm sure there is something that I'm >> >>> missing so I'm hoping for suggestions. >> >>> >> >>> There are two systems that are connected head to head via cross over >> >>> cable. The two systems have the same hardware configuration. The >> >>> hardware is as follows: >> >>> >> >>> 2 Intel E5430 (Quad core) @ 2.66 Ghz >> >>> Intel S5000PAL Motherboard >> >>> 16GB Memory >> >>> >> >>> My iperf command line for the client is: >> >>> >> >>> iperf -t 10 -c 169.0.0.1 -w 2.5M -l 2.5M >> >>> >> >>> My TCP dump test command lines are: >> >>> >> >>> tcpdump -i ix0 -w/dev/null >> >>> tcpreplay -i ix0 -t -l 0 -K ./test.pcap >> >>> >> >> If you're running 8.0-RELEASE, you might try updating to 8-STABLE. >> >> Jack Vogel recently committed updated Intel NIC driver code: >> >> >> >> http://svn.freebsd.org/viewvc/base/stable/8/sys/dev/ixgbe/ >> >> >> >> -Brandon >> >> _______________________________________________ >> >> freebsd-performance@freebsd.org mailing list >> >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> >> To unsubscribe, send any mail to " >> >> freebsd-performance-unsubscribe@freebsd.org" >> >> >> >> >> > _______________________________________________ >> > freebsd-performance@freebsd.org mailing list >> > http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> > To unsubscribe, send any mail to " >> freebsd-performance-unsubscribe@freebsd.org" >> > >> > >> >> > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?h2x2a41acea1004220939k23e34403hcd57480d92f4a0f1>