From owner-freebsd-current@FreeBSD.ORG Wed Dec 7 11:26:23 2011 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D1B0106566B for ; Wed, 7 Dec 2011 11:26:23 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 63F358FC1E for ; Wed, 7 Dec 2011 11:26:22 +0000 (UTC) Received: (qmail 66075 invoked from network); 7 Dec 2011 09:30:27 -0000 Received: from unknown (HELO [62.48.0.94]) ([62.48.0.94]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 7 Dec 2011 09:30:27 -0000 Message-ID: <4EDF471F.1030202@freebsd.org> Date: Wed, 07 Dec 2011 11:59:43 +0100 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20111105 Thunderbird/8.0 MIME-Version: 1.0 To: Luigi Rizzo References: <20111205192703.GA49118@onelab2.iet.unipi.it> <2D87D847-A2B7-4E77-B6C1-61D73C9F582F@digsys.bg> <20111205222834.GA50285@onelab2.iet.unipi.it> <4EDDF9F4.9070508@digsys.bg> <4EDE259B.4010502@digsys.bg> <20111206210625.GB62605@onelab2.iet.unipi.it> In-Reply-To: <20111206210625.GB62605@onelab2.iet.unipi.it> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: current@freebsd.org, Jack Vogel , Daniel Kalchev Subject: Re: datapoints on 10G throughput with TCP ? X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Dec 2011 11:26:23 -0000 On 06.12.2011 22:06, Luigi Rizzo wrote: > On Tue, Dec 06, 2011 at 07:40:21PM +0200, Daniel Kalchev wrote: >> I see significant difference between number of interrupts on the Intel and the AMD blades. When performing a test between the Intel and AMD blades, the Intel blade generates 20,000-35,000 interrupts, while the AMD blade generates under 1,000 interrupts. >> > > Even in my experiments there is a lot of instability in the results. > I don't know exactly where the problem is, but the high number of > read syscalls, and the huge impact of setting interrupt_rate=0 > (defaults at 16us on the ixgbe) makes me think that there is something > that needs investigation in the protocol stack. > > Of course we don't want to optimize specifically for the one-flow-at-10G > case, but devising something that makes the system less affected > by short timing variations, and can pass upstream interrupt mitigation > delays would help. I'm not sure the variance is only coming from the network card and driver side of things. The TCP processing and interactions with scheduler and locking probably play a big role as well. There have been many changes to TCP recently and maybe an inefficiency that affects high-speed single sessions throughput has crept in. That's difficult to debug though. -- Andre