From owner-freebsd-net@FreeBSD.ORG  Wed Dec 21 15:37:20 2011
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 399B61065673
	for <freebsd-net@freebsd.org>; Wed, 21 Dec 2011 15:37:20 +0000 (UTC)
	(envelope-from luigi@onelab2.iet.unipi.it)
Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238])
	by mx1.freebsd.org (Postfix) with ESMTP id E91088FC16
	for <freebsd-net@freebsd.org>; Wed, 21 Dec 2011 15:37:19 +0000 (UTC)
Received: by onelab2.iet.unipi.it (Postfix, from userid 275)
	id 910E27300A; Wed, 21 Dec 2011 16:34:56 +0100 (CET)
Date: Wed, 21 Dec 2011 16:34:56 +0100
From: Luigi Rizzo <rizzo@iet.unipi.it>
To: Clint <clint@servedomains.com>
Message-ID: <20111221153456.GA30629@onelab2.iet.unipi.it>
References: <2f23773b5165887ac51d19a612bf4c93@servedomains.com>
	<20111220224814.GA7715@michelle.cdnetworks.com>
	<5b246a7cf0bceedf39ff5718336b0753@servedomains.com>
	<CAFOYbcn+raVVFThgnZKX6NgEmKPktzeysW60N+KOUWNiXO+6-g@mail.gmail.com>
	<91610121f172c87d5095c8ac03ea9837@servedomains.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <91610121f172c87d5095c8ac03ea9837@servedomains.com>
User-Agent: Mutt/1.4.2.3i
Cc: freebsd-net@freebsd.org
Subject: Re: igb performance/load udp issue
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Wed, 21 Dec 2011 15:37:20 -0000

On Wed, Dec 21, 2011 at 08:04:54AM -0700, Clint wrote:
> On Tue, 20 Dec 2011 22:46:46 -0800, Jack Vogel wrote:
> >I was fighting with UDP issues before the latest checkin, so you 
> >should
> >look at THAT version, 2.3.1 in HEAD please.
> >
> >Jack
> >
> 
> I'm actually running the HEAD version now, but my problems still 
> persist.  I did notice that the excessive load is now spread across 
> multiple cores where before it was isolated to one core at a time.  This 
> version also doesn't suffer from the horrible performance I experienced 
> with v2.2.5.

as someone mentioned, it may have to do with the sender
using multiple queues thus causing reordering in the frames,
which then the receiver has to sort (and often, the receive
queue is a list so O(N) complexity per packet.)

Can you run the igb driver with just a single queue ?
In the bootloader, or before loading the module using "kenv", you
should set

	hw.igb.num_queues = 1

This should prevent gratuitous reordering and perhaps fix
your performance.

cheers
luigi