Date: Wed, 24 Oct 2012 01:32:41 +0200 From: Andre Oppermann <oppermann@networx.ch> To: Garrett Cooper <yanegomi@gmail.com> Cc: Jack F Vogel <jfv@freebsd.org>, freebsd-net@freebsd.org Subject: Re: ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1 Message-ID: <50872919.90701@networx.ch> In-Reply-To: <CAGH67wRs5XB2FqnPz%2BNY6i28UNa4gYBx5=5ny_ScFPVKh3zYdQ@mail.gmail.com> References: <CAGH67wTjVw8O7i9OCvHVJjVjCAXsNHNhs0LobDi4G9iukgf%2BeQ@mail.gmail.com> <CAGH67wRs5XB2FqnPz%2BNY6i28UNa4gYBx5=5ny_ScFPVKh3zYdQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 24.10.2012 01:13, Garrett Cooper wrote: > On Tue, Oct 23, 2012 at 3:57 PM, Garrett Cooper <yanegomi@gmail.com> wrote: >> Hi, >> >> Doing some poking around at the ixgb driver with a card I have at >> $work using netperf and two machines hooked up over crossover, I >> discovered that while ixgb's throughput performance was fantastic on >> 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 >> by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO >> performance on the other hand is fantastic and doesn't degrade with >> the card across FreeBSD versions. Performance remains constant with >> ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. >> >> More details: >> >> The machines are hooked up in the following configuration: >> >> ----------------------- -------------------- >> | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | >> ----------------------- --------------------- >> >> Machine Configuration: >> >> The card in Machine 2 is an 82599EB card according to pciconf -lv. >> >> /boot/loader.conf tunables (most of these are set according to 9.x >> defaults in order to establish a sane baseline): >> >> kern.ipc.nmbjumbo9=262144 >> kern.ipc.nmbjumbo16=262144 >> kern.ipc.nmbclusters=262144 >> kern.ipc.nmbjumbop=262144 >> kern.ipc.maxsockbuf=2097152 >> >> /etc/sysctl.conf tunables: >> >> net.inet.tcp.recvspace=65536 >> net.inet.tcp.recvspace_inc=16384 >> net.inet.tcp.recvspace_max=2097152 >> net.inet.tcp.sendspace=32768 >> net.inet.tcp.sendbuf_max=2097152 >> net.inet.tcp.sendbuf_inc=8192 >> >> Kernel Config: >> >> Machine 1 is running a custom version of FreeBSD. The version has been >> constant over the course of my testing. Can give vague details on the >> config, but can't give some specific details. >> Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. >> >> Networking configuration: >> >> - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. >> The interface mtu is 1500. >> - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. >> The interface mtu is 1500. >> >> Netperf configuration: >> >> - netserver is run on both machines; I don't add any additional >> arguments to the netserver invocation so it just goes off and forks. >> - netperf is run like: netperf -cCjt TCP_STREAM -H <IP-ADDRESS> >> >> I was wondering if this was a known issue and/or others had seen >> similar problems with this card. I haven't gone into profiling the >> kernel yet with DTrace, but if no one gets back to me before sometime >> later on this week/next week that will be my next course of action for >> tracking down the source of the performance problem. > > A couple more notes: > > 1. We're not using Intel SFP modules -- they're Finisar based > (shouldn't matter, but I know that some Intel NICs are incredibly > picky when it comes to SFP modules). > 2. The performance on 8.2 is actually worse than on 9.x: ~5700Mbps. There have been a very large number of changes to our network stack between 7.4 and today. That makes it very difficult to pinpoint a specific change. First of all please test 10-current as well but make sure that you turn off WITNESS and INVARIANTS in your kernel config. Second there may be an issue with TSO and ixgb where TSO can cause packets that are a bit larger than 64K by the ethernet header size. ixgb DMA is set up for 64K all inclusive. While TSO did generate a IP valid packet it is arguably not very well digestable for the driver considering that 64K is a very magic number. TSO should reduce its largest packet size by max_linkhdr. A fix for that is in the works and will be available shortly. -- Andre
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?50872919.90701>