From owner-freebsd-stable@FreeBSD.ORG Tue Jan 22 00:34:19 2008 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F35616A474 for ; Tue, 22 Jan 2008 00:34:19 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from wa-out-1112.google.com (wa-out-1112.google.com [209.85.146.177]) by mx1.freebsd.org (Postfix) with ESMTP id 456AF13C447 for ; Tue, 22 Jan 2008 00:34:19 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by wa-out-1112.google.com with SMTP id k17so4049116waf.3 for ; Mon, 21 Jan 2008 16:34:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:received:date:from:to:cc:subject:message-id:reply-to:references:mime-version:content-type:content-disposition:in-reply-to:user-agent; bh=LyGUWNp0FqBqE0w2asChZFPrL1oI7/+Zj3xm+iNPKoY=; b=vfB1QFGJRIo+72nIHooRmIqZWKGwNovwmdDzlPHTUDiRnHDlTV8nx5HhBGFqRxL5UT4EFv1CCo1jPFXmK4PYpBVyFdKwc0dJgUn1rdODTFA+iORkZF45FhyMIvnqpHdzbA//51gYKXWhsQwHRTGheoMaZdnLSw9sBKF/wWKSJCA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:reply-to:references:mime-version:content-type:content-disposition:in-reply-to:user-agent; b=VAlDhY37Y+/pXfcVCLmKzxWwm+kx5W1idOAUX6nzC5UmvfSsdif+pWmnVxw6ukJwkPNKCD4egMWcoaimnKWwHHKt2jJAsfOHnT6nMmsk6sjfWNb6rRhcb03NrAV+b5F0pdRB1zfygFmDuCXUl8x71q9E4DomI/EsC3qzSxVcFe0= Received: by 10.114.27.20 with SMTP id a20mr6046032waa.101.1200962058442; Mon, 21 Jan 2008 16:34:18 -0800 (PST) Received: from michelle.cdnetworks.co.kr ( [211.53.35.84]) by mx.google.com with ESMTPS id n38sm16435079wag.2.2008.01.21.16.34.15 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 21 Jan 2008 16:34:17 -0800 (PST) Received: from michelle.cdnetworks.co.kr (localhost.cdnetworks.co.kr [127.0.0.1]) by michelle.cdnetworks.co.kr (8.13.5/8.13.5) with ESMTP id m0M0YCpY010131 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 22 Jan 2008 09:34:12 +0900 (KST) (envelope-from pyunyh@gmail.com) Received: (from yongari@localhost) by michelle.cdnetworks.co.kr (8.13.5/8.13.5/Submit) id m0M0YBj3010130; Tue, 22 Jan 2008 09:34:11 +0900 (KST) (envelope-from pyunyh@gmail.com) Date: Tue, 22 Jan 2008 09:34:11 +0900 From: Pyun YongHyeon To: The Wolf Message-ID: <20080122003411.GA9894@cdnetworks.co.kr> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i Cc: shigeaki@f.csce.kyushu-u.ac.jp, freebsd-stable@freebsd.org Subject: Re: Huge number of interrupts on nfe driver X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jan 2008 00:34:19 -0000 On Mon, Jan 21, 2008 at 02:37:55PM -0500, The Wolf wrote: > Hello everyone. > > A webserver that I've been working on seems to be having some issues of some > sort with the nfe-20071124 driver. We have it running on two gigabit ports, > one facing inwards to the database server, and the other facing outwards. > The outward facing port is the only one which I currently have bandwidth > information on, and it seems to be pushing around 35-40 Mb/s. > > Unfortunately, there are literally billions of interrupts generated in only > a short period of time. In the last 40 days, there are 4,502,689,887 > interrupts on the outward facing port, and another 653,327,087 on the inward > facing port. It is literally slowing down the machine to a crawl - on a > comparatively powerful server that previously performed the same task, load > hovered around 0.2 -> 0.4. On the current machine, the load average hovers > between 14 and 16. > > /var/log/messages is flooded with messages such as: kernel: nfe0: watchdog > timeout (missed Tx interrupts) -- recovering. > I think this was fixed by scott's commit. If you still see this issue it would indicate other bug in nfe(4). What FreeBSD version do you use? > Attached I am including portions of the output of lspci, in case there is > relevant data there. > > If there is anything else I can provide you with, please don't hesitate to > ask. > > Sincerely, > tsawolf According to the output of lscpi, I think nfe(4) should use MSI/MSIX on your system. Would you show me the output of nfe related verbosed boot message and vmstat -i output? Btw, nfe(4)'s interrupt moderation does not seem to work at all. I failed to find a way to make interrupt moderation work on nfe(4). I guess Linux also have the same issue too. However I don't think NVIDIA ethernet controllers lack the feature, we just still do not know what regergister is related with interrupt moderation control and what magic value should be used to activate it. Maybe I should see disassembled NDIS driver as a reference but the code size is too big to analyze it. :-( -- Regards, Pyun YongHyeon