From owner-freebsd-net@FreeBSD.ORG Tue Oct 23 23:32:54 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 15F745B7 for ; Tue, 23 Oct 2012 23:32:54 +0000 (UTC) (envelope-from oppermann@networx.ch) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 6F48B8FC14 for ; Tue, 23 Oct 2012 23:32:53 +0000 (UTC) Received: (qmail 90046 invoked from network); 24 Oct 2012 01:10:48 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 24 Oct 2012 01:10:48 -0000 Message-ID: <50872919.90701@networx.ch> Date: Wed, 24 Oct 2012 01:32:41 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:16.0) Gecko/20121010 Thunderbird/16.0.1 MIME-Version: 1.0 To: Garrett Cooper Subject: Re: ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1 References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Jack F Vogel , freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 23:32:54 -0000 On 24.10.2012 01:13, Garrett Cooper wrote: > On Tue, Oct 23, 2012 at 3:57 PM, Garrett Cooper wrote: >> Hi, >> >> Doing some poking around at the ixgb driver with a card I have at >> $work using netperf and two machines hooked up over crossover, I >> discovered that while ixgb's throughput performance was fantastic on >> 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 >> by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO >> performance on the other hand is fantastic and doesn't degrade with >> the card across FreeBSD versions. Performance remains constant with >> ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. >> >> More details: >> >> The machines are hooked up in the following configuration: >> >> ----------------------- -------------------- >> | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | >> ----------------------- --------------------- >> >> Machine Configuration: >> >> The card in Machine 2 is an 82599EB card according to pciconf -lv. >> >> /boot/loader.conf tunables (most of these are set according to 9.x >> defaults in order to establish a sane baseline): >> >> kern.ipc.nmbjumbo9=262144 >> kern.ipc.nmbjumbo16=262144 >> kern.ipc.nmbclusters=262144 >> kern.ipc.nmbjumbop=262144 >> kern.ipc.maxsockbuf=2097152 >> >> /etc/sysctl.conf tunables: >> >> net.inet.tcp.recvspace=65536 >> net.inet.tcp.recvspace_inc=16384 >> net.inet.tcp.recvspace_max=2097152 >> net.inet.tcp.sendspace=32768 >> net.inet.tcp.sendbuf_max=2097152 >> net.inet.tcp.sendbuf_inc=8192 >> >> Kernel Config: >> >> Machine 1 is running a custom version of FreeBSD. The version has been >> constant over the course of my testing. Can give vague details on the >> config, but can't give some specific details. >> Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. >> >> Networking configuration: >> >> - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. >> The interface mtu is 1500. >> - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. >> The interface mtu is 1500. >> >> Netperf configuration: >> >> - netserver is run on both machines; I don't add any additional >> arguments to the netserver invocation so it just goes off and forks. >> - netperf is run like: netperf -cCjt TCP_STREAM -H >> >> I was wondering if this was a known issue and/or others had seen >> similar problems with this card. I haven't gone into profiling the >> kernel yet with DTrace, but if no one gets back to me before sometime >> later on this week/next week that will be my next course of action for >> tracking down the source of the performance problem. > > A couple more notes: > > 1. We're not using Intel SFP modules -- they're Finisar based > (shouldn't matter, but I know that some Intel NICs are incredibly > picky when it comes to SFP modules). > 2. The performance on 8.2 is actually worse than on 9.x: ~5700Mbps. There have been a very large number of changes to our network stack between 7.4 and today. That makes it very difficult to pinpoint a specific change. First of all please test 10-current as well but make sure that you turn off WITNESS and INVARIANTS in your kernel config. Second there may be an issue with TSO and ixgb where TSO can cause packets that are a bit larger than 64K by the ethernet header size. ixgb DMA is set up for 64K all inclusive. While TSO did generate a IP valid packet it is arguably not very well digestable for the driver considering that 64K is a very magic number. TSO should reduce its largest packet size by max_linkhdr. A fix for that is in the works and will be available shortly. -- Andre