From owner-freebsd-net@FreeBSD.ORG Tue Oct 23 23:45:36 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A953EB04; Tue, 23 Oct 2012 23:45:36 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id E7F0A8FC14; Tue, 23 Oct 2012 23:45:35 +0000 (UTC) Received: by mail-lb0-f182.google.com with SMTP id b5so929892lbd.13 for ; Tue, 23 Oct 2012 16:45:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=ZmYJpjCi/HIcH13ORgtZbxl7BwDqXDOc3rlQXCXtwho=; b=eL9Y6wZKJ6wG4CL0NsFZVWSK1eQ+RTa834FWj2Ms9TeCpSXHi+VWLgmZTOJwhVya/k yRTswmmORBOQm4UtA/8cqOYLpBZrcIUNTWVySG7FtZkJqj+aYntwlGgikFNLBZI6I6pD CGIV5l+FEA+vRoTqhx767loWoP2zJeCWAbwobcVCn/9+W4nhLafz/u/J4dZmP0KIuOyP etENN0XRLV8LH1lsGxfagfn0RklMXXy9X+pCPfgMt2hxPF7RkAlgrzz5ysJxD3Rk98mD k5AAJUd9/MgY+sqcp65pMwr0Df3Syn8kjhBmkqC32hooTarDwpvEcvrFUZnzfeqc3uX/ sq+Q== MIME-Version: 1.0 Received: by 10.152.133.140 with SMTP id pc12mr12658596lab.53.1351035934573; Tue, 23 Oct 2012 16:45:34 -0700 (PDT) Received: by 10.112.43.232 with HTTP; Tue, 23 Oct 2012 16:45:34 -0700 (PDT) In-Reply-To: <50872919.90701@networx.ch> References: <50872919.90701@networx.ch> Date: Tue, 23 Oct 2012 16:45:34 -0700 Message-ID: Subject: Re: ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1 From: Garrett Cooper To: Andre Oppermann Content-Type: text/plain; charset=ISO-8859-1 Cc: Jack F Vogel , freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 23:45:36 -0000 On Tue, Oct 23, 2012 at 4:32 PM, Andre Oppermann wrote: ... >>> Doing some poking around at the ixgb driver with a card I have at >>> $work using netperf and two machines hooked up over crossover, I >>> discovered that while ixgb's throughput performance was fantastic on >>> 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 >>> by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO >>> performance on the other hand is fantastic and doesn't degrade with >>> the card across FreeBSD versions. Performance remains constant with >>> ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. >>> >>> More details: >>> >>> The machines are hooked up in the following configuration: >>> >>> ----------------------- >>> -------------------- >>> | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | >>> ----------------------- >>> --------------------- >>> >>> Machine Configuration: >>> >>> The card in Machine 2 is an 82599EB card according to pciconf -lv. >>> >>> /boot/loader.conf tunables (most of these are set according to 9.x >>> defaults in order to establish a sane baseline): >>> >>> kern.ipc.nmbjumbo9=262144 >>> kern.ipc.nmbjumbo16=262144 >>> kern.ipc.nmbclusters=262144 >>> kern.ipc.nmbjumbop=262144 >>> kern.ipc.maxsockbuf=2097152 >>> >>> /etc/sysctl.conf tunables: >>> >>> net.inet.tcp.recvspace=65536 >>> net.inet.tcp.recvspace_inc=16384 >>> net.inet.tcp.recvspace_max=2097152 >>> net.inet.tcp.sendspace=32768 >>> net.inet.tcp.sendbuf_max=2097152 >>> net.inet.tcp.sendbuf_inc=8192 >>> >>> Kernel Config: >>> >>> Machine 1 is running a custom version of FreeBSD. The version has been >>> constant over the course of my testing. Can give vague details on the >>> config, but can't give some specific details. >>> Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. >>> >>> Networking configuration: >>> >>> - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. >>> The interface mtu is 1500. >>> - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. >>> The interface mtu is 1500. >>> >>> Netperf configuration: >>> >>> - netserver is run on both machines; I don't add any additional >>> arguments to the netserver invocation so it just goes off and forks. >>> - netperf is run like: netperf -cCjt TCP_STREAM -H >>> >>> I was wondering if this was a known issue and/or others had seen >>> similar problems with this card. I haven't gone into profiling the >>> kernel yet with DTrace, but if no one gets back to me before sometime >>> later on this week/next week that will be my next course of action for >>> tracking down the source of the performance problem. >> >> >> A couple more notes: >> >> 1. We're not using Intel SFP modules -- they're Finisar based >> (shouldn't matter, but I know that some Intel NICs are incredibly >> picky when it comes to SFP modules). >> 2. The performance on 8.2 is actually worse than on 9.x: ~5700Mbps. > > There have been a very large number of changes to our network > stack between 7.4 and today. That makes it very difficult to > pinpoint a specific change. > > First of all please test 10-current as well but make sure that > you turn off WITNESS and INVARIANTS in your kernel config. > > Second there may be an issue with TSO and ixgb where TSO can > cause packets that are a bit larger than 64K by the ethernet > header size. ixgb DMA is set up for 64K all inclusive. While > TSO did generate a IP valid packet it is arguably not very well > digestable for the driver considering that 64K is a very magic > number. TSO should reduce its largest packet size by max_linkhdr. > A fix for that is in the works and will be available shortly. Ok. I'll give it a shot sometime this upcoming week to see if things improve (of course without INVARIANTS and WITNESS ;)..); I saw there were some changes in the works on CURRENT -- just haven't gotten around to testing that yet (melifaro and a few other folks have been working on performance and removing unnecessary locking too it seems). Thanks for the feedback! -Garrett