From owner-freebsd-net@FreeBSD.ORG  Tue Apr  7 12:54:26 2009
Return-Path: <owner-freebsd-net@FreeBSD.ORG>
Delivered-To: freebsd-net@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 432621065740;
	Tue,  7 Apr 2009 12:54:26 +0000 (UTC)
	(envelope-from rwatson@FreeBSD.org)
Received: from cyrus.watson.org (cyrus.watson.org [65.122.17.42])
	by mx1.freebsd.org (Postfix) with ESMTP id 1D1778FC24;
	Tue,  7 Apr 2009 12:54:26 +0000 (UTC)
	(envelope-from rwatson@FreeBSD.org)
Received: from fledge.watson.org (fledge.watson.org [65.122.17.41])
	by cyrus.watson.org (Postfix) with ESMTPS id C830846BC8;
	Tue,  7 Apr 2009 08:54:25 -0400 (EDT)
Date: Tue, 7 Apr 2009 13:54:25 +0100 (BST)
From: Robert Watson <rwatson@FreeBSD.org>
X-X-Sender: robert@fledge.watson.org
To: Sepherosa Ziehau <sepherosa@gmail.com>
In-Reply-To: <ea7b9c170904062157u1c457f27md565f9a95a51a705@mail.gmail.com>
Message-ID: <alpine.BSF.2.00.0904071350520.45341@fledge.watson.org>
References: <gra7mq$ei8$1@ger.gmane.org>
	<alpine.BSF.2.00.0904051422280.12639@fledge.watson.org>
	<grac1s$p56$1@ger.gmane.org>
	<ea7b9c170904062157u1c457f27md565f9a95a51a705@mail.gmail.com>
User-Agent: Alpine 2.00 (BSF 1167 2008-08-23)
MIME-Version: 1.0
Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed
Cc: freebsd-net@freebsd.org, Ivan Voras <ivoras@freebsd.org>
Subject: Re: Advice on a multithreaded netisr patch?
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-net>,
	<mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 07 Apr 2009 12:54:27 -0000


On Tue, 7 Apr 2009, Sepherosa Ziehau wrote:

> On Sun, Apr 5, 2009 at 9:34 PM, Ivan Voras <ivoras@freebsd.org> wrote:
>> Robert Watson wrote:
>>>
>>> On Sun, 5 Apr 2009, Ivan Voras wrote:
>>>
>>>> I thought this has something to deal with NIC moderation (em) but
>>>> can't really explain it. The bad performance part (not the jump) is
>>>> also visible over the loopback interface.
>>>
>>> FYI, if you want high performance, you really want a card supporting 
>>> multiple input queues -- igb, cxgb, mxge, etc.  if_em-only cards are
>
> PCI-E em(4) supports 2 RX queues.  82571/82572 support 2 TX queues. I have 
> not tested multi-TX queues, but em(4) multi-RX queues work well in dfly 
> (tested with 82573 and 82571)

You may not have seen, but in FreeBSD 7.x and higher, we have a new if_igb 
driver to support more recent Intel gigabit devices, which now probes a few of 
the devices historically associated with if_em.  For example, on one of the 
boxes I use:

igb0: <Intel(R) PRO/1000 Network Connection version - 1.3.0> port 
0x3000-0x301f mem 
0xd8220000-0xd823ffff,0xd8200000-0xd821ffff,0xd8280000-0xd8283fff irq 32 at 
device 0.0 on pci8
igb0: Using MSIX interrupts with 3 vectors
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: [ITHREAD]
igb0: Ethernet address: 00:30:48:d2:ca:c2
igb1: <Intel(R) PRO/1000 Network Connection version - 1.3.0> port 
0x3020-0x303f mem 
0xd8260000-0xd827ffff,0xd8240000-0xd825ffff,0xd8284000-0xd8287fff irq 46 at 
device 0.1 on pci8
igb1: Using MSIX interrupts with 3 vectors
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: [ITHREAD]
igb1: Ethernet address: 00:30:48:d2:ca:c3
igb0: RX LRO Initialized
igb1: RX LRO Initialized

Robert N M Watson
Computer Laboratory
University of Cambridge