From owner-freebsd-net@FreeBSD.ORG Tue Oct 23 22:57:03 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5133D9D7; Tue, 23 Oct 2012 22:57:03 +0000 (UTC) (envelope-from yanegomi@gmail.com) Received: from mail-la0-f54.google.com (mail-la0-f54.google.com [209.85.215.54]) by mx1.freebsd.org (Postfix) with ESMTP id 97B438FC0C; Tue, 23 Oct 2012 22:57:02 +0000 (UTC) Received: by mail-la0-f54.google.com with SMTP id e12so3445409lag.13 for ; Tue, 23 Oct 2012 15:57:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:cc:content-type; bh=dBVxftSDWviWS7AWvyiCnHX9kNivgUSTmwH7+yztROM=; b=OPQBs+1nIvyZ0xlQ7+oLheaK+iNzDll+/zS64USGpml4L/HzWREN35yZgCeZnWw88m W4fRnEE7GsEUdHwg0TCxTGmznHxyctGCFkaCimYmONtFbzoU/JbrhJibVdIUNAo1Sirb iitnbxV4R1QwLRLUbcdCvGX9C3AUZ9ZjDksHLYV6YJ58TG5FkaVTIMdXgcPhe+mb10Ro Z6Jq7pr+oJA8zFzjAiX0w/vC2CBuRIsEqz5AvZh/dImXoLapfsio7Yq7gCFcIyFFMlxz ZtNHAxaTKis91Rm23pHLw2+5dTXrHfNwXNHp+YfZSap7i4is1HMCLgJCZcInlkdToX3L z/xg== MIME-Version: 1.0 Received: by 10.152.124.83 with SMTP id mg19mr12685574lab.6.1351033021343; Tue, 23 Oct 2012 15:57:01 -0700 (PDT) Received: by 10.112.43.232 with HTTP; Tue, 23 Oct 2012 15:57:01 -0700 (PDT) Date: Tue, 23 Oct 2012 15:57:01 -0700 Message-ID: Subject: ixgb TSO performance degrades by ~30% between 7.4 and 8.2/9.0/9.1 From: Garrett Cooper To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Cc: Jack F Vogel X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Oct 2012 22:57:03 -0000 Hi, Doing some poking around at the ixgb driver with a card I have at $work using netperf and two machines hooked up over crossover, I discovered that while ixgb's throughput performance was fantastic on 7.3/7.4, thoughput performance of the card is degraded on 8.2/9.0/9.1 by ~30% (9400Mbps on 7.4 -> 6294Mbps on 9.0 for example). LRO performance on the other hand is fantastic and doesn't degrade with the card across FreeBSD versions. Performance remains constant with ixgb across 8.2/9.0/9.1. I didn't observe the CPU usage. More details: The machines are hooked up in the following configuration: ----------------------- -------------------- | Machine 1 | cxgb | <- 10Gbit fibre -> | ix1 | Machine 2 | ----------------------- --------------------- Machine Configuration: The card in Machine 2 is an 82599EB card according to pciconf -lv. /boot/loader.conf tunables (most of these are set according to 9.x defaults in order to establish a sane baseline): kern.ipc.nmbjumbo9=262144 kern.ipc.nmbjumbo16=262144 kern.ipc.nmbclusters=262144 kern.ipc.nmbjumbop=262144 kern.ipc.maxsockbuf=2097152 /etc/sysctl.conf tunables: net.inet.tcp.recvspace=65536 net.inet.tcp.recvspace_inc=16384 net.inet.tcp.recvspace_max=2097152 net.inet.tcp.sendspace=32768 net.inet.tcp.sendbuf_max=2097152 net.inet.tcp.sendbuf_inc=8192 Kernel Config: Machine 1 is running a custom version of FreeBSD. The version has been constant over the course of my testing. Can give vague details on the config, but can't give some specific details. Machine 2 is running 7.4/8.2/9.0/9.1 with a GENERIC kernel. Networking configuration: - Machine 1 has an IPv4 address of 10.10.10.1; IPv6 is not configured. The interface mtu is 1500. - Machine 2 has an IPv4 address of 10.10.10.2; IPv6 is not configured. The interface mtu is 1500. Netperf configuration: - netserver is run on both machines; I don't add any additional arguments to the netserver invocation so it just goes off and forks. - netperf is run like: netperf -cCjt TCP_STREAM -H I was wondering if this was a known issue and/or others had seen similar problems with this card. I haven't gone into profiling the kernel yet with DTrace, but if no one gets back to me before sometime later on this week/next week that will be my next course of action for tracking down the source of the performance problem. Thanks! -Garrett