Date: Tue, 23 Oct 2012 16:45:34 -0700 From: Garrett Cooper <yanegomi@gmail.com> To: Andre Oppermann <oppermann@networx.ch> Cc: Jack F Vogel <jfv@freebsd.org>, freebsd-net@freebsd.org Subject: Re: ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1 Message-ID: <CAGH67wT6sTUkNaUBnd7iN-k1tEOZzXaYJiq16L24--_6%2BgAp%2BA@mail.gmail.com> In-Reply-To: <50872919.90701@networx.ch> References: <CAGH67wTjVw8O7i9OCvHVJjVjCAXsNHNhs0LobDi4G9iukgf%2BeQ@mail.gmail.com> <CAGH67wRs5XB2FqnPz%2BNY6i28UNa4gYBx5=5ny_ScFPVKh3zYdQ@mail.gmail.com> <50872919.90701@networx.ch>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Oct 23, 2012 at 4:32 PM, Andre Oppermann <oppermann@networx.ch> wrote:
...
>>> Doing some poking around at the ixgb driver with a card I have at
>>> $work using netperf and two machines hooked up over crossover, I
>>> discovered that while ixgb's throughput performance was fantastic on
>>> 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1
>>> by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO
>>> performance on the other hand is fantastic and doesn't degrade with
>>> the card across FreeBSD versions. Performance remains constant with
>>> ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage.
>>>
>>> More details:
>>>
>>> The machines are hooked up in the following configuration:
>>>
>>> -----------------------
>>> --------------------
>>> | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 |
>>> -----------------------
>>> ---------------------
>>>
>>> Machine Configuration:
>>>
>>> The card in Machine 2 is an 82599EB card according to pciconf -lv.
>>>
>>> /boot/loader.conf tunables (most of these are set according to 9.x
>>> defaults in order to establish a sane baseline):
>>>
>>> kern.ipc.nmbjumbo9=262144
>>> kern.ipc.nmbjumbo16=262144
>>> kern.ipc.nmbclusters=262144
>>> kern.ipc.nmbjumbop=262144
>>> kern.ipc.maxsockbuf=2097152
>>>
>>> /etc/sysctl.conf tunables:
>>>
>>> net.inet.tcp.recvspace=65536
>>> net.inet.tcp.recvspace_inc=16384
>>> net.inet.tcp.recvspace_max=2097152
>>> net.inet.tcp.sendspace=32768
>>> net.inet.tcp.sendbuf_max=2097152
>>> net.inet.tcp.sendbuf_inc=8192
>>>
>>> Kernel Config:
>>>
>>> Machine 1 is running a custom version of FreeBSD. The version has been
>>> constant over the course of my testing. Can give vague details on the
>>> config, but can't give some specific details.
>>> Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel.
>>>
>>> Networking configuration:
>>>
>>> - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured.
>>> The interface mtu is 1500.
>>> - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured.
>>> The interface mtu is 1500.
>>>
>>> Netperf configuration:
>>>
>>> - netserver is run on both machines; I don't add any additional
>>> arguments to the netserver invocation so it just goes off and forks.
>>> - netperf is run like: netperf -cCjt TCP_STREAM -H <IP-ADDRESS>
>>>
>>> I was wondering if this was a known issue and/or others had seen
>>> similar problems with this card. I haven't gone into profiling the
>>> kernel yet with DTrace, but if no one gets back to me before sometime
>>> later on this week/next week that will be my next course of action for
>>> tracking down the source of the performance problem.
>>
>>
>> A couple more notes:
>>
>> 1. We're not using Intel SFP modules -- they're Finisar based
>> (shouldn't matter, but I know that some Intel NICs are incredibly
>> picky when it comes to SFP modules).
>> 2. The performance on 8.2 is actually worse than on 9.x: ~5700Mbps.
>
> There have been a very large number of changes to our network
> stack between 7.4 and today. That makes it very difficult to
> pinpoint a specific change.
>
> First of all please test 10-current as well but make sure that
> you turn off WITNESS and INVARIANTS in your kernel config.
>
> Second there may be an issue with TSO and ixgb where TSO can
> cause packets that are a bit larger than 64K by the ethernet
> header size. ixgb DMA is set up for 64K all inclusive. While
> TSO did generate a IP valid packet it is arguably not very well
> digestable for the driver considering that 64K is a very magic
> number. TSO should reduce its largest packet size by max_linkhdr.
> A fix for that is in the works and will be available shortly.
Ok. I'll give it a shot sometime this upcoming week to see if
things improve (of course without INVARIANTS and WITNESS ;)..); I saw
there were some changes in the works on CURRENT -- just haven't gotten
around to testing that yet (melifaro and a few other folks have been
working on performance and removing unnecessary locking too it seems).
Thanks for the feedback!
-Garrett
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGH67wT6sTUkNaUBnd7iN-k1tEOZzXaYJiq16L24--_6%2BgAp%2BA>