From owner-freebsd-current@FreeBSD.ORG  Tue Dec  6 20:50:20 2011
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: current@freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34])
	by hub.freebsd.org (Postfix) with ESMTP id 799CC1065670
	for <current@freebsd.org>; Tue,  6 Dec 2011 20:50:20 +0000 (UTC)
	(envelope-from luigi@onelab2.iet.unipi.it)
Received: from onelab2.iet.unipi.it (onelab2.iet.unipi.it [131.114.59.238])
	by mx1.freebsd.org (Postfix) with ESMTP id 36CB88FC17
	for <current@freebsd.org>; Tue,  6 Dec 2011 20:50:19 +0000 (UTC)
Received: by onelab2.iet.unipi.it (Postfix, from userid 275)
	id B75367300A; Tue,  6 Dec 2011 22:06:25 +0100 (CET)
Date: Tue, 6 Dec 2011 22:06:25 +0100
From: Luigi Rizzo <rizzo@iet.unipi.it>
To: Daniel Kalchev <daniel@digsys.bg>
Message-ID: <20111206210625.GB62605@onelab2.iet.unipi.it>
References: <20111205192703.GA49118@onelab2.iet.unipi.it>
	<2D87D847-A2B7-4E77-B6C1-61D73C9F582F@digsys.bg>
	<20111205222834.GA50285@onelab2.iet.unipi.it>
	<4EDDF9F4.9070508@digsys.bg> <4EDE259B.4010502@digsys.bg>
	<CAFOYbcmVR_K0iZU_Z4TxDVzPzx6-GZuzfCxUZbf6KQn4siF2UA@mail.gmail.com>
	<F5BCA7E9-6A61-4492-9F18-423178E9C9B4@digsys.bg>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <F5BCA7E9-6A61-4492-9F18-423178E9C9B4@digsys.bg>
User-Agent: Mutt/1.4.2.3i
Cc: Jack Vogel <jfvogel@gmail.com>, current@freebsd.org
Subject: Re: datapoints on 10G throughput with TCP ?
X-BeenThere: freebsd-current@freebsd.org
X-Mailman-Version: 2.1.5
Precedence: list
List-Id: Discussions about the use of FreeBSD-current
	<freebsd-current.freebsd.org>
List-Unsubscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>, 
	<mailto:freebsd-current-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-current>
List-Post: <mailto:freebsd-current@freebsd.org>
List-Help: <mailto:freebsd-current-request@freebsd.org?subject=help>
List-Subscribe: <http://lists.freebsd.org/mailman/listinfo/freebsd-current>,
	<mailto:freebsd-current-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 06 Dec 2011 20:50:20 -0000

On Tue, Dec 06, 2011 at 07:40:21PM +0200, Daniel Kalchev wrote:
> I see significant difference between number of interrupts on the Intel and the AMD blades. When performing a test between the Intel and AMD blades, the Intel blade generates 20,000-35,000 interrupts, while the AMD blade generates under 1,000 interrupts.
> 

Even in my experiments there is a lot of instability in the results.
I don't know exactly where the problem is, but the high number of
read syscalls, and the huge impact of setting interrupt_rate=0
(defaults at 16us on the ixgbe) makes me think that there is something
that needs investigation in the protocol stack.

Of course we don't want to optimize specifically for the one-flow-at-10G
case, but devising something that makes the system less affected
by short timing variations, and can pass upstream interrupt mitigation
delays would help.

I don't have a solution yet..

cheers
luigi