From owner-freebsd-net@FreeBSD.ORG Tue Oct 23 23:58:20 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6C64BDD2; Tue, 23 Oct 2012 23:58:20 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from mail-vc0-f182.google.com (mail-vc0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id EB87F8FC16; Tue, 23 Oct 2012 23:58:19 +0000 (UTC) Received: by mail-vc0-f182.google.com with SMTP id fw7so6215802vcb.13 for ; Tue, 23 Oct 2012 16:58:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=rV5ATEA3Jkt1tvRgD73XFJnreTyXJaaDQRguXBmXaYY=; b=UG3N69mdtJHYWORbj91BrkSggRW+8n5alolGY7W7cncMl5SSCoRsrE4Fnn4VMfh0eQ WvsBv2RrBt3AULlju47fiCybfOcVr+M+u8M82BrC3vlQaX8g+dzXluNsrjmg2aN3qr75 Wn8jGY3MoDdHns4LVZNndpOQxVGFBjXZfUKtr6HaYQk9ZDBTeCGxRkksdD4G/0xSnGG2 wdNDGAWfjwU4X2vOAQ6ogjcH+51PumzX+DEGFu6d68nVOPBUvQnkZCtQk+KxpKhnvdoj FsrF8ZMrx+WvXK+LiZZHzEgJfPFwDVb1acTYLewIo4pA4ZgIqAz+YbkvXhLS5GfFdR3D dn5A== MIME-Version: 1.0 Received: by 10.220.226.67 with SMTP id iv3mr5181967vcb.57.1351036698910; Tue, 23 Oct 2012 16:58:18 -0700 (PDT) Received: by 10.58.68.8 with HTTP; Tue, 23 Oct 2012 16:58:18 -0700 (PDT) In-Reply-To: References: Date: Tue, 23 Oct 2012 16:58:18 -0700 Message-ID: Subject: Re: ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1 From: Jack Vogel To: Garrett Cooper Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Jack F Vogel , freebsd-net@freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 23:58:20 -0000 It you mean the ixgbe driver please call it that, the IS an ixgb driver which is for VERY old PCI cards, from the context I assume you mean the newer hardware :) Jack On Tue, Oct 23, 2012 at 3:57 PM, Garrett Cooper wrote: > Hi, > > Doing some poking around at the ixgb driver with a card I have at > $work using netperf and two machines hooked up over crossover, I > discovered that while ixgb's throughput performance was fantastic on > 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 > by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO > performance on the other hand is fantastic and doesn't degrade with > the card across FreeBSD versions. Performance remains constant with > ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. > > More details: > > The machines are hooked up in the following configuration: > > ----------------------- > -------------------- > | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | > ----------------------- > --------------------- > > Machine Configuration: > > The card in Machine 2 is an 82599EB card according to pciconf -lv. > > /boot/loader.conf tunables (most of these are set according to 9.x > defaults in order to establish a sane baseline): > > kern.ipc.nmbjumbo9=262144 > kern.ipc.nmbjumbo16=262144 > kern.ipc.nmbclusters=262144 > kern.ipc.nmbjumbop=262144 > kern.ipc.maxsockbuf=2097152 > > /etc/sysctl.conf tunables: > > net.inet.tcp.recvspace=65536 > net.inet.tcp.recvspace_inc=16384 > net.inet.tcp.recvspace_max=2097152 > net.inet.tcp.sendspace=32768 > net.inet.tcp.sendbuf_max=2097152 > net.inet.tcp.sendbuf_inc=8192 > > Kernel Config: > > Machine 1 is running a custom version of FreeBSD. The version has been > constant over the course of my testing. Can give vague details on the > config, but can't give some specific details. > Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. > > Networking configuration: > > - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. > The interface mtu is 1500. > - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. > The interface mtu is 1500. > > Netperf configuration: > > - netserver is run on both machines; I don't add any additional > arguments to the netserver invocation so it just goes off and forks. > - netperf is run like: netperf -cCjt TCP_STREAM -H > > I was wondering if this was a known issue and/or others had seen > similar problems with this card. I haven't gone into profiling the > kernel yet with DTrace, but if no one gets back to me before sometime > later on this week/next week that will be my next course of action for > tracking down the source of the performance problem. > > Thanks! > -Garrett >