From owner-freebsd-net@FreeBSD.ORG Wed Aug 12 21:36:29 2009 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 47E09106566B for ; Wed, 12 Aug 2009 21:36:29 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from mail-yx0-f181.google.com (mail-yx0-f181.google.com [209.85.210.181]) by mx1.freebsd.org (Postfix) with ESMTP id ED57A8FC47 for ; Wed, 12 Aug 2009 21:36:28 +0000 (UTC) Received: by mail-yx0-f181.google.com with SMTP id 11so462269yxe.3 for ; Wed, 12 Aug 2009 14:36:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:from:date:to:cc :subject:message-id:reply-to:references:mime-version:content-type :content-disposition:in-reply-to:user-agent; bh=0GoSbJZbIhxGje95SEey4zKxcivexqjB1TxP5QIW/0s=; b=LrvzRF/N+Wc+BIDURxbp0Tqykx7HquDYbvR5UpOR9LFphgNTVgrzgJrYeUBtLobaU7 NXJfb1ixYCWHq0lrhfu7yqIL/yiP3nXkf3RMBgytrlz+9QysBEjMLjaFGfS/JzxmaW3m Oz58KKiIS6r1HC4hd5klhm0fWsPb42uDRNXCI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:date:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=cpd0daCIdm8aN6+fM5e/aJOTM0Nb0bD5W3A/wkmpcYzoFffCdQGbulihVQ4YjKE94m 0d0K+Q293fRXc4ANv0N4AVn2A98ktSkbj7bwtqXfP241mh21Fa/QWH0/izWDweppY59I Ucdoo2YY5Z+LGniusQ2nES5GVAVaNag9sxu08= Received: by 10.101.165.2 with SMTP id s2mr475064ano.16.1250112985642; Wed, 12 Aug 2009 14:36:25 -0700 (PDT) Received: from pyunyh@gmail.com ([174.35.1.224]) by mx.google.com with ESMTPS id c14sm532519ana.19.2009.08.12.14.36.23 (version=TLSv1/SSLv3 cipher=RC4-MD5); Wed, 12 Aug 2009 14:36:24 -0700 (PDT) Received: by pyunyh@gmail.com (sSMTP sendmail emulation); Wed, 12 Aug 2009 14:35:21 -0700 From: Pyun YongHyeon Date: Wed, 12 Aug 2009 14:35:21 -0700 To: Peter Steele Message-ID: <20090812213521.GG55129@michelle.cdnetworks.com> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i Cc: freebsd-net@freebsd.org Subject: Re: nfe taskq performance issues X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 12 Aug 2009 21:36:29 -0000 On Thu, Jul 23, 2009 at 08:58:07AM -0700, Peter Steele wrote: > We've been hitting serious nfe taskq performance issues during stress > tests and in doing some research on the problem we came across this old > email: > > > > From: Ivan Voras > Date: April 28, 2009 3:53:14 AM PDT > To: freebsd-threads@freebsd.org > Cc: freebsd-net@freebsd.org, freebsd-performance@freebsd.org > Subject: Re: FreeBSD 7.1 taskq em performance > > > > I have been hitting some barrier with FreeBSD 7.1 network performance. > I > > have written an application which contains two kernel threads that > takes > > mbufs directly from a network interface and forwards to another > network > > interface. This idea is to simulate different network environment. > > > > I have been using FreeBSD 6.4 amd64 and tested with an Ixia box > > (specialised hardware firing very high packet rate). The PC was a > Core2 2.6 > > GHz with dual ports Intel PCIE Gigabit network card. It can manage up > to 1.2 > > million pps. > > > > I have a higher spec PC with FreeBSD 7.1 amd64 and Quadcore 2.3 GHz > and > > PCIE Gigabit network card. The performance can only achieve up to 600k > pps. > > I notice the 'taskq em0' and 'taskq em1' is solid 100% CPU but it is > not in > > FreeBSD 6.4. > > > > In our case we are running FreeBSD 7.0, but we are seeing our boxes > experience serious thread starvation issues as the nfe0 cpu percentage > climbs steadily while cpu idle time drops at times to 0 percent. This > email thread mentioned a patch for the em driver here: > > > > http://people.yandex-team.ru/~wawa/ > > > > > Does anyone know if this patch will work with the nfe driver? > That's for em(4). AFAIK all nfe(4) controllers lacks intelligent interrupts moderation so driver should be prepared to handle excessive interrupt loads. I'm not sure whether NVIDIA ethernet controllers really lacks efficient interrupt mitigation mechanism but it seems Linux also faces the same hardware problem. As you might know there is no publicly available data sheet for NVIDIA controllers so setting it right looks very hard to me.