Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 23 Oct 2012 16:13:48 -0700
From:      Garrett Cooper <yanegomi@gmail.com>
To:        freebsd-net@freebsd.org
Cc:        Jack F Vogel <jfv@freebsd.org>
Subject:   Re: ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1
Message-ID:  <CAGH67wRs5XB2FqnPz%2BNY6i28UNa4gYBx5=5ny_ScFPVKh3zYdQ@mail.gmail.com>
In-Reply-To: <CAGH67wTjVw8O7i9OCvHVJjVjCAXsNHNhs0LobDi4G9iukgf%2BeQ@mail.gmail.com>
References:  <CAGH67wTjVw8O7i9OCvHVJjVjCAXsNHNhs0LobDi4G9iukgf%2BeQ@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Oct 23, 2012 at 3:57 PM, Garrett Cooper <yanegomi@gmail.com> wrote:
> Hi,
>
> Doing some poking around at the ixgb driver with a card I have at
> $work using netperf and two machines hooked up over crossover, I
> discovered that while ixgb's throughput performance was fantastic on
> 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1
> by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO
> performance on the other hand is fantastic and doesn't degrade with
> the card across FreeBSD versions. Performance remains constant with
> ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage.
>
> More details:
>
> The machines are hooked up in the following configuration:
>
>  -----------------------                                 --------------------
> | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 |
>  -----------------------                                ---------------------
>
> Machine Configuration:
>
> The card in Machine 2 is an 82599EB card according to pciconf -lv.
>
> /boot/loader.conf tunables (most of these are set according to 9.x
> defaults in order to establish a sane baseline):
>
> kern.ipc.nmbjumbo9=262144
> kern.ipc.nmbjumbo16=262144
> kern.ipc.nmbclusters=262144
> kern.ipc.nmbjumbop=262144
> kern.ipc.maxsockbuf=2097152
>
> /etc/sysctl.conf tunables:
>
> net.inet.tcp.recvspace=65536
> net.inet.tcp.recvspace_inc=16384
> net.inet.tcp.recvspace_max=2097152
> net.inet.tcp.sendspace=32768
> net.inet.tcp.sendbuf_max=2097152
> net.inet.tcp.sendbuf_inc=8192
>
> Kernel Config:
>
> Machine 1 is running a custom version of FreeBSD. The version has been
> constant over the course of my testing. Can give vague details on the
> config, but can't give some specific details.
> Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel.
>
> Networking configuration:
>
> - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured.
> The interface mtu is 1500.
> - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured.
> The interface mtu is 1500.
>
> Netperf configuration:
>
> - netserver is run on both machines; I don't add any additional
> arguments to the netserver invocation so it just goes off and forks.
> - netperf is run like: netperf -cCjt TCP_STREAM -H <IP-ADDRESS>
>
> I was wondering if this was a known issue and/or others had seen
> similar problems with this card. I haven't gone into profiling the
> kernel yet with DTrace, but if no one gets back to me before sometime
> later on this week/next week that will be my next course of action for
> tracking down the source of the performance problem.

A couple more notes:

1. We're not using Intel SFP modules -- they're Finisar based
(shouldn't matter, but I know that some Intel NICs are incredibly
picky when it comes to SFP modules).
2. The performance on 8.2 is actually worse than on 9.x: ~5700Mbps.

Thanks!
-Garrett



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGH67wRs5XB2FqnPz%2BNY6i28UNa4gYBx5=5ny_ScFPVKh3zYdQ>